# How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp Ryan Stevens, Ramp [Talk page](https://ai.engineer/talks/WJTQBENv0KI-ramp-tests-its-ai-bookkeeping-agent-before) · [Video](https://www.youtube.com/watch?v=WJTQBENv0KI) ## How Ramp Tests Its AI Bookkeeping Agent Before Customers Do Ramp and Mercor built synthetic businesses with full books so Stack could try the same accounting work again and again. Those back-tests exposed the gap between an almost-correct answer and a finished reconciliation. They also gave Ramp a way to test which skills were worth keeping and what the agent could carry into the next close. ### What does it take for the books to tie? [0:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1s) A company keeps a ledger of the bills it owes in its enterprise resource planning system, or ERP. It can also use Ramp to pay those bills. At close, the accountant needs the two systems to agree. That gave Ryan Stevens, who leads applied science at Ramp, a concrete starting point for evaluating Stack, its bookkeeping agent. The team sat down with bookkeepers and accounting firms to learn what a person actually does when the numbers differ. Which records need to agree? The accountant brings the accounts payable ledger from the ERP into Excel row by row, then brings in the records from the system that holds the bills. If the numbers differ, the accountant works out why and resolves the discrepancy. That last step matters for an end-to-end test. An agent can correctly flag a mismatch and still leave the accountant with the work of reconciling it. Reconciliation became one of four benchmark categories. Ramp also tested data entry, variance analysis (explaining why numbers moved), and schedules and accruals. Breaking the close into tasks at this level let the team define what a successful result looked like for each one. ### Build a business whose books can be replayed [3:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=200s) Before Stack had enough users to supply a representative set of tasks, data was the bottleneck. The agent would call tools that queried a database. What should those tools find when it went looking for a bill? Ramp built the benchmark in partnership with Mercor. Synthetic businesses let them supply the records on both sides of a reconciliation. One business was a direct-to-consumer lifestyle company that earned revenue through Shopify and Amazon, followed a calendar year, and closed its books monthly. Alongside the company description, the team built its ERP data, customer and vendor lists, and accounts payable records. A reconciliation could now send Stack into the books of a particular retailer, with its own counterparties and bills. Keeping those books fixed made the evaluation a back-test. Git gives a coding agent a convenient way to revisit an earlier version of a repository. Replaying a bookkeeping task requires the corresponding state of the business. With a synthetic company, Ramp controlled that state from the start. It could give a changed agent yesterday’s task against yesterday’s records and compare the results. A snapshot of production data could serve the same purpose. ### Grade the finished task, then count complete passes [7:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=427s) What does the grader inspect once Stack finishes a reconciliation? Ramp paired the task with a fixed business environment, borrowing Mercor’s term *world*. It let the agent run, then checked the final output against task-specific criteria. A typical task had 10 to 20. The agent could earn full credit regardless of which tools it called, which skills it invoked, or which route it took. Ramp cared about cost and latency too, both of which it could assess without prescribing a sequence of tool calls. Building the evaluation took four to six months before Stack launched. In Stevens’s terms, the criteria were labels. Once the team had specified them, the same labels could answer two different questions about performance. Take the example with six criteria. Across three attempts at the same task, the agent satisfied four criteria, then five, then all six. - **Accuracy** asks how much of the work was right on a given run. The first attempt scored 4/6, or about 67%. The second scored 5/6 even though one part still needed attention. - **Pass@k** asks whether at least one of *k* attempts satisfied every criterion. In this sequence, pass@1 was 0, pass@2 was still 0, and pass@3 was 1. Only the third attempt produced a complete answer. An engineer can learn something from the difference between four and five. An accountant handed the second result still has to find the remaining error. Where a single mistake can undermine trust in the whole reconciliation, the chance of getting a complete answer on the first attempt deserves its own measure. ### The models found the files and still got the accounting wrong [10:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=622s) How often did high accuracy translate into a complete answer? In the model comparison Ramp published when Stack launched, the **Opus** models performed best. Yet criterion accuracy was high while first-attempt pass rates were low. Even the leaders left work unfinished often enough for Ramp to measure improvements. Changing the software around the model gave the team more complete passes. To decide what to change, the team examined failed runs. Had the agent gathered the wrong information? Had it failed to follow an instruction? Sometimes it had all the data and drew the wrong conclusion, which Ramp classified as a reasoning failure. A fourth category, reflection failure, served largely as a catch-all. Reasoning was the main failure on these accounting tasks. Think back to accounts payable. Once the agent understands that it needs records of bills the company owes, identifying the relevant files is relatively straightforward. The runs showed that an agent could gather those files successfully and still reason incorrectly about their contents. Better retrieval alone would leave that failure intact. ### Auto-tuning stalled. Which skills were worth their context? [12:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=737s) With a repeatable score, Ramp tried automatically tuning the text of prompts and skills. The idea was to try a rewrite, evaluate the resulting agent, and use the score to search for better wording. That search failed to deliver a gain for Stack. Stevens’s objection came down to direction. When tuning a model, a gradient can indicate which way to adjust its parameters. An evaluation can tell you that one rewrite did better than another and still give you little guidance about which words to change next. Skill ablation posed a narrower question. A skill takes up space in the agent’s context. How much performance does its presence buy? Ramp borrowed the logic of feature ablation in machine learning, where removing a feature or adding it back helps isolate its contribution. A no-skill baseline measured what the model could do with its training alone as guidance. Runs with a skill present and absent tested the value of supplying that extra guidance on the same work. This experiment varied skills. Stevens suggested tools as a possible follow-up. Reading an individual result depends on which condition is subtracted from which. Stevens flagged a possible reversal in his reading of the chart signs, so the direction of a particular skill’s effect is uncertain from his explanation. The comparisons did show that changing the skill set changed performance. They gave Ramp a shortlist of skills to examine for deletion or compression, along with a way to test whether a shorter configuration still did the job. ### Can Stack use what it learned at the next close? [15:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=904s) Writing a summary to a memory store and reading it back demonstrates that storage works. Ramp wanted to know whether the contents helped Stack do the next period’s work. Its *roll-forward worlds* gave the same business the same kind of task in two successive periods. A memory-writing prompt compressed the first run into something the agent could use during the second. How could a record-matching rule learned in one period reach the next reconciliation? Customers can have conventions for matching records that they never fully spell out. Ramp built an example where satisfying the criteria required inferring those conventions. After the first reconciliation, the experiment could put its memory directly into the next run’s context or let the agent search a set of memories. Direct injection tests whether the information helps once supplied. Search also asks whether the agent can find it when it needs it. Changing the memory-writing prompt changed what the agent preserved about the task. It also changed how well the agent performed in the following period. Stevens described this as one of Ramp’s largest gains from the evaluations. The second close gave the team a way to judge a summary by what another run could accomplish with it. How does a memory reach the next reconciliation? Both conditions begin with the first period’s work and end with a fresh reconciliation. The search condition adds a retrieval decision between writing the memory and using it. - First reconciliation — Match records under implicit customer rules. - Write a memory — The memory prompt compresses information from the run. - Direct injection — Supply the memory to the next run. - Memory search — Let the agent look for relevant information among stored memories. - Next reconciliation — Check how successfully the agent matches the next period’s records under the customer’s rules. - First reconciliation → Write a memory (First run supplies the experience to compress) - Write a memory → Direct injection (Use the newly written memory directly) - Write a memory → Memory search (Make the memory available in the searchable set) - Direct injection → Next reconciliation (Agent receives the selected memory) - Memory search → Next reconciliation (Agent can use what its search retrieves) ### Run regression tests in the background [16:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1018s) A repeatable world still contains a nondeterministic agent. Stevens had seen tests pass 70% or 80% of the time. With that much variation, a single failure makes a poor reason to block a code change in continuous integration. Ramp put its evaluations on an internal platform where people could add cases, tag them into groups, and schedule them to run in the background daily or weekly. After a scheduled run, a background agent examines the results for failures and regressions. When it spots a possible decline, a Slack message flags the issue and suggests what might have broken. Ramp now uses the benchmark to catch regressions in familiar bookkeeping work, with a concrete lead for the engineer who investigates. ## Key takeaways - Replaying a bookkeeping task requires the starting books as well as the instructions. A synthetic business or a saved production snapshot can supply that state. - Criterion accuracy measures partial progress. Pass@1 measures how often the accountant gets a fully correct result on the first attempt. Ramp’s model comparison showed a substantial gap between the two. - A second accounting period makes memory quality testable. Direct injection lets the team assess the written memory’s usefulness. Search also tests whether the agent can retrieve it. ## Slides ### Building Better Agents through End-to-End Evals Ryan Stevens uses Stack, Ramp’s AI agent, as a case study in how to evaluate an agent from start to finish. [0:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=12s) [Ramp’s title slide introduces a case study of Stack and names Ryan Stevens, Senior Director of Applied Sciences at Ramp, as the speaker.](https://assets.ai.engineer/sites/aiecode2025/sha256/7196903b5e7cb9be55e60142315932f53a89128ffbb2d32ce6c2ae356790e373.webp) Building Better Agents through End-to-End Evals Case Study of Stack Ryan Stevens Senior Director, Applied Sciences at Ramp ramp Confidential. Not for unauthorized distribution. ### Some Introductions: Ramp and me Ramp serves 70,000+ businesses. Ryan Stevens leads its Applied Sciences team, which tackles credit and fraud risk, processes documents, and builds agents to automate financial workflows. [0:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=21s) [Bullets describe Ramp’s financial products, its Applied Sciences team and the speaker’s background. A Ramp logo and speaker portrait appear on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe519b484fcebc4071aeb4a67ed6f829cd952d6f47405ed32757f4cc21d1f781.webp) Some Introductions: Ramp and me AIRFLOW @ RAMP What is Ramp • Financial technology platform that serves 70,000+ businesses • Product suite with cards, expense management, AP, banking, travel and more • Typical problems my team solves are traditional Risk Management (Credit and Fraud), Document Collection and Processing, and Workflow Automation (Agents) Who am I • Lead our Applied Sciences team; mixture of AI Engineers, Machine Learning Engineers, and Product Data Scientists • Build machine learning models and agents, derive insights and metrics to help the company make better decisions • PhD in Economics from NYU, previously at Meta 2 ### What is your agent’s goal? The bold first item sets the order for Ramp’s bookkeeping agent: define what it should accomplish, then design evaluations around that goal and put the results to use. [0:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=46s) [A numbered agenda emphasizes the agent’s goal in bold, followed by designing evals and getting value from them.](https://assets.ai.engineer/sites/aiecode2025/sha256/e01f578cb39b4661dbc96e35cb4afb24320611931cb76e5a5739a04e9e62d099.webp) 01 What is your agent’s goal? 02 Designing your evals 03 Getting value from your evals ### Understand the process before building an agent A dog at a keyboard admits it has no idea how to do accounting, yet it has already built an accounting agent. Stevens would start by breaking the job into tasks and spelling out what success looks like. [0:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=57s) [A dog sits at a keyboard. The caption jokes that it built an accounting agent despite having no idea how to do accounting.](https://assets.ai.engineer/sites/aiecode2025/sha256/eab3cf8b8899e0353c444cf5c9d924926db657b9802a28bd3f768580a24397ff.webp) Biggest mistake people make is agentifying a process they don’t understand AIRFLOW @ RAMP I HAVE NO IDEA HOW TO DO ACCOUNTING BUT I BUILT AN AGENT FOR IT memebuilder.ai 5 ### How an accountant reconciles bills in Excel An accountant compares Ramp bills with the QuickBooks ledger row by row in Excel. Two bills match. Bill 103 shows $500 in Ramp and has no entry in QuickBooks. Ramp uses that manual process to define the agent’s task. [1:40](https://www.youtube.com/watch?v=WJTQBENv0KI&t=100s) [Arrows lead from the QuickBooks ledger and Ramp bills into an Excel worksheet. Two bills match. A third appears only in Ramp, leaving a $500 gap.](https://assets.ai.engineer/sites/aiecode2025/sha256/b8bd84f5b57c81585b22bde3e337ffa9ee5024f4ed4bd254813717f9ddbce633.webp) We map the exact process a human follows, then translate it to an agent AIRFLOW @ RAMP QuickBooks AP general ledger Ramp Bills X Excel Current reconciliation worksheet Bill QuickBooks Ramp Status 101 $1,000 $1,000 ✓ Match 102 $2,000 $2,000 ✓ Match 103 Missing $500 ! $500 gap ● 2 matched ● 1 difference Today: accountant Manually matches bills and adds notes 8 ### Four categories of accounting benchmark tasks Reconciliation is the largest category in Ramp’s benchmark. In 61 of 160 tasks, the agent must match two sources and explain or correct discrepancies. The rest cover schedules and accruals (45), variance analysis (28), and data entry (26). [2:42](https://www.youtube.com/watch?v=WJTQBENv0KI&t=162s) [The table divides 160 tasks into 61 for reconciliation, 26 for data entry, 28 for variance analysis and 45 for schedules and accruals.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebfdbefc04330491c0ee82ee9daca27389660248b4fba51629034052aaf04ac0.webp) We wanted to build a benchmark, so we designed four task categories AIRFLOW @ RAMP Category Definition Tasks Reconciliation Tie two sources together, identify differences, and explain or correct the breaks. 61 Data Entry Post or update transactions, journal entries, vendor bills, and invoices. 26 Variance Analysis Compare actuals against budget, prior periods, or expectations and explain the drivers. 28 Schedules & Accruals Build or update supporting schedules, calculate accruals, and carry the right balances into the close. 45 Overall — 160 9 ### Evaluation data before and after a product has users The bookkeeping agent uses tools to access a database in both stages. The bottleneck is filling it with clean records. Synthetic bills, payments, and ledger entries stand in until enough users supply representative customer data. [3:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=198s) [Before users, synthetic business data supplies a simulated database. With users, real customer data supplies the customer database. Both connect to an agent through tools. A red outline highlights the data and database in both rows.](https://assets.ai.engineer/sites/aiecode2025/sha256/38666b26bd869591bffba03c35f57c73c9116f77fd478a7c2304857f300c3b38.webp) Building your evals (benchmark) is different depending on the product life cycle AIRFLOW @ RAMP Data is the bottleneck to evals Before users Synthetic business data Bills · payments · ledger → Database Simulated state ↔ Tools ↔ Agent With users Real customer data Bills · payments · ledger → Database Customer state ↔ Tools ↔ Agent 11 ### Accounting records for a synthetic business Ramp built QuickBooks records for a fictional retailer to test its bookkeeping agent offline. The December 2024 close draws on 12 months of wholesale invoices and vendor bills, including a few bills coded to the wrong sales channel. [4:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=286s) [A profile describes a fictional home goods business and its month-end close. An adjacent table lists its chart of accounts, customer and vendor lists, invoices and bills.](https://assets.ai.engineer/sites/aiecode2025/sha256/c89ccd158d6c57da5ccc57afc885355440ac4aa54102a9e3d3183885c67c6da1.webp) We built synthetic businesses with payments, bills, and ERP data AIRFLOW @ RAMP Company Profile Velo Goods Co. is a DTC lifestyle and home goods brand selling through three channels: Shopify (direct-to-consumer storefront), Amazon FBA, and wholesale to specialty retailers. ~$4M annual revenue, 12 employees across operations, marketing, and G&A. Single-entity LLC on QuickBooks Online with Gusto for payroll. Uses a third-party logistics provider (3PL) for warehousing and fulfillment. Sources products from overseas manufacturers with landed cost inventory (product + freight + duties). Fiscal year: Calendar year (Jan-Dec) | Period in focus: December 2024 month-end close | Accounting basis: US GAAP (accrual) Key complexity drivers: Multi-channel revenue recognition with different fee structures per platform, landed cost inventory with overseas sourcing, payment processor reconciliation across Shopify Payments/Amazon/PayPal, multi-state sales tax nexus, returns and refund reserves by channel, and seasonal Q4 holiday variance. Document | Content | Medium Inside QBO Chart of Accounts | ~60 accounts. Revenue split by channel (Shopify DTC, Amazon FBA, Wholesale). COGS includes product cost, freight, duties, and fulfillment. OpEx categories for marketing (by channel), payroll, G&A, shipping. Classes for Shopify, Amazon, Wholesale. | QBO Customer list | 8-10 wholesale accounts (boutiques, specialty retailers). DTC customers are anonymous through Shopify and Amazon, not individually tracked in QBO. | QBO Vendor list | 25-30 vendors. Product manufacturers (overseas), 3PL warehouse (fulfillment), Shopify (platform fees), Amazon (seller fees), freight forwarders, marketing agencies, packaging suppliers. Some vendor names have inconsistent formatting. | QBO Invoices (AR) | 12 months of wholesale invoices. Net 30 terms. A handful of short-pays and disputed invoices. No AR for DTC since Shopify and Amazon settle directly. | QBO Bills (AP) | 12 months of vendor bills. Product purchases (large POs with landed cost components), 3PL monthly invoices, marketing agency retainer, Amazon seller fees, packaging supplies. A few bills coded to the wrong channel class. | QBO ### Terms used to backtest agents and measure success Four Spider-Man figures labeled evals, benchmarks, labels and verifiable outputs point at one another. For Stevens, all four terms get at the same question: when you rerun the agent on the same data, how do you tell whether it’s improving? [5:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=337s) [Four Spider-Man figures point at one another. Their labels read “Evals,” “Benchmarks,” “Labels” and “Verifiable outputs.”](https://assets.ai.engineer/sites/aiecode2025/sha256/2ad0ce01256593156e7b4c6ad55d85275cc05f1eca89040e6f1474e4cb1d8ba9.webp) Lots of words for same concept: we want to backtest agents and measure success AIRFLOW @ RAMP EVALS BENCHMARKS LABELS VERIFIABLE OUTPUTS NYPD imgflip.com 14 ### Scoring an accounts payable reconciliation report Ramp’s agent reconciles synthetic records for Q4 2025. An LLM scores only the final report. It meets 4 of 6 criteria (67%): it gets the balance right but misses duplicates and leaves timing gaps unexplained. [7:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=425s) [Four panels show the task, synthetic records, reconciliation steps and final report score. The report meets four of six criteria (67%). It misses the criteria for flagging duplicates and explaining timing gaps.](https://assets.ai.engineer/sites/aiecode2025/sha256/a85c8f0982c4e2efaa82926d0faecce8cf2fac7c1dd8ce471e67eeafb1086a10.webp) AIRFLOW @ RAMP For each business, we designed a series of accounting tasks, rubric and LLM as a judge Task “Reconcile our AP payables and bills on Ramp for Q4 2025.” World Synthetic ERP Data AP ledger AP aging Credits / adjustments Synthetic Business Data Ramp bills Ramp payments Vendor invoices Trajectory 1 Read bills + AP data 2 Match bills + payments 3 Investigate differences FINAL OUTPUT AP reconciliation report Score FINAL OUTPUT ONLY 67% 4 of 6 criteria met ✓ Q4 AP balance correct ✓ Bills matched ✓ Payments matched × Duplicates flagged × Timing gaps explained ✓ Exceptions listed ### Two complementary ways to measure performance The agent meets 4 of 6 criteria on its first attempt, earning 67% accuracy. Pass@k stays at 0 through the second attempt, even at 5/6. The third attempt meets all six criteria, so Pass@3 = 1. [9:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=552s) [Four of six correct criteria earn 67% accuracy. Three attempts score 4/6, 5/6 and 6/6, so Pass@1 and Pass@2 are zero and Pass@3 is one.](https://assets.ai.engineer/sites/aiecode2025/sha256/577fa9569ce62607cbc4e1ef3e00cbcd5a842662c7f1adb1eb97b79fa1b709f8.webp) AIRFLOW @ RAMP Two complementary ways to measure performance Accuracy How many criteria are correct? ✓ ✓ ✓ ✓ × × 67% 4 of 6 criteria correct Partial credit counts Pass@k Did any attempt get everything right? Attempt 1 ✓ ✓ ✓ ✓ × × 4/6 Incomplete Pass@1 = 0 Attempt 2 ✓ ✓ ✓ ✓ ✓ × 5/6 Incomplete Pass@2 = 0 Attempt 3 ✓ ✓ ✓ ✓ ✓ ✓ 6/6 Fully correct Pass@3 = 1 One fully correct attempt is enough (k = 3) ### Getting value from your evals Ramp spent four to six months building evals for its bookkeeping agent. The third agenda item marks the shift from how the team designed those evals to how it uses the results. [10:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=620s) [A three-part agenda lists the agent’s goal, designing evals and getting value from evals. The third item is bold.](https://assets.ai.engineer/sites/aiecode2025/sha256/10d3619ad22ace9fbfd18e28a7f1691d45ff4aaeeb00228f1d1900648da9ae4b.webp) 01 What is your agent’s goal? 02 Designing your evals 03 Getting value from your evals ### Using evals to choose a model When Ramp Stack launched, it had the highest accuracy in the comparison, about 66%, yet it passed only about one in five tasks on the first try. All six raw models also scored much lower on first-try success than on accuracy. [10:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=633s) [A grouped bar chart with confidence intervals compares Ramp Stack and six models on Accuracy, Pass@1, Pass@5 and Pass^5. Every series scores higher on Accuracy than on Pass@1.](https://assets.ai.engineer/sites/aiecode2025/sha256/c39551a92589cf66c547d65eb69d91ba66615ed3bf3d17e47fbb49b1ce3b7a03.webp) AIRFLOW @ RAMP First obvious use case is to decide which models to use Model comparison by metric 95% CIs from world-bootstrap · raw models run without Stack skills or memory Y-axis ticks: 0%, 15%, 30%, 45%, 60%, 75% X-axis: Accuracy, Pass@1, Pass@5, Pass^5 Legend: Ramp Stack, Opus 4.7, Opus 4.6, GPT 5.5, GPT 5.4, Gemini 3.5 Flash, Gemini 3.1 Pro ### Model failures by cause All three models most often missed accounting optimizations when they had the data but drew the wrong conclusion. Ramp counted 13 to 21 reasoning failures per model, compared with 2 to 5 information-gathering failures. [11:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=677s) [Reasoning failures lead for all three models, with counts of 13, 19 and 21. Information gathering and instruction following each account for five or fewer failures per model. Planning and reflection account for one per model.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2139de614e43d1257e956928abb3728f6c223de56c178af8c17e091512731ae.webp) We dig deeper into why models are failing to spot optimizations AIRFLOW @ RAMP Legend: Muse-Spark-1.1 (xHigh) | Claude-Fable-5 (Max) | GPT-5.6-Sol (Max+ Pro) Y-axis: Failures Y-axis ticks: 0, 5, 10, 15, 20 Bar labels in legend order: Reasoning failure: 13 | 19 | 21 Information-gathering failure: 5 | 3 | 2 Instruction-following and prompt understanding failure: 3 | 1 | 4 Planning and reflection failure: 1 | 1 | 1 ### What didn’t work in the experiments GEPA automatically edited skills for Ramp’s bookkeeping agent. Scores improved during training, but the gains didn’t generalize. Stevens says that result may be specific to Ramp’s agent. [12:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=755s) [A red outline highlights the GEPA row in a five-experiment table. Automated skill edits produced training gains that failed to generalize. An arrow connects the row to the presenters’ comment that they have yet to see convincing evidence for this approach.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a3dd9257bd6f0b359979979b5a16210b48f84e03e2624218fd1e0d8e4c107c4.webp) We ran a lot of experiments, here’s what didn’t work AIRFLOW @ RAMP Experiment | Change tested | Result Model selection | Gemini, GPT-5.4, GPT-5.5; Claude Opus vs. Sonnet | Benchmark quantified model performance gaps Auto-tuned skills (GEPA) | Automated skill edits with autoresearch | Training gains did not generalize High reasoning effort | High vs. low effort across model combinations | Same accuracy; higher cost Minimal skill bundle | PDF + Excel only vs. SLIM 4-skill baseline | Below the 4-skill baseline Conservative rewrites | Shorter, more cautious finance skills (xsafe) | No gain over standard skills The underlying theory of GEPA or “autoresearch” on skills / prompts is very shoddy, we’ve yet to see good evidence it works ### Testing agent memory with Roll Forward Worlds Ramp’s agent learns a customer’s bookkeeping rules in one month, such as how a restaurant group matches sales to bank deposits. On the next month’s fresh records, accuracy rose from 56% with memory off to 61.5% with memory on. [15:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=914s) [Construction and restaurant examples show an agent learning business rules from one month’s records and applying them to fresh records the next month. Accuracy rises from 56% with memory off to 61.5% with memory on.](https://assets.ai.engineer/sites/aiecode2025/sha256/d045f1c90d83dce36d3183dd4256dd2e902fdda68c072567ef35094d6a1246f2.webp) People don’t test memory systems, we designed a unique way to do that → Roll Forward Worlds AIRFLOW @ RAMP This month Construction company Match records ✓ Differences explained Learn how this business works Restaurant group Match records ✓ Differences explained Learn how this business works What the agent remembers CUSTOMER RULES • How to count income on unfinished jobs • Which job each cost belongs to Needed to solve the next period CUSTOMER RULES • How sales match bank deposits • Which store each sale belongs to Needed to solve the next period Next month Construction company New month. Same business. Use its rules on new records Calculate the new month’s numbers Restaurant group New month. Same business. Use its rules on new records Calculate the new month’s numbers Remember how the business works. Use fresh numbers. Last month’s amounts are not this month’s answers. Memory on vs. off Y-axis: Accuracy (%) Y-axis ticks: 50%, 52%, 54%, 56%, 58%, 60%, 62%, 64% Memory off: 56% Memory on: 61.5% +5.5pp ### Scheduled evaluations for production regressions Ramp reruns the same bookkeeping cases from the same starting state on a schedule, then compares accuracy and Pass@k with a baseline. A background agent reviews failures and flags suspected regressions in Slack for the team to investigate. [16:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1017s) [Two interface panels show how to choose cases, trial count and model, then set a recurring schedule. A diagram shows the benchmark running on the same cases and starting worlds, followed by baseline comparison and review of failed tasks and traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d5f92a5b4414bfac9a57a9f57b592142126c178319386011ad1ae18715b955e.webp) AIRFLOW @ RAMP Once it’s in production, internal eval platform runs on a schedule to catch regressions 1 Choose cases, trials, and model Case tags Matches any selected tag. ☐ Select all tags (20) Trials per case: 3 Each selected case runs 3 times. Run 206 matching cases Model: Stack default Reasoning effort: Model default Subagent model: Stack default Subagent reasoning effort: Model default Case | Tags | Run abk_excel_organization | excel, canonical_benchmark | ▷ A payroll sheet that subtotals in place: a row with no name is a subtotal of the detail rows beneath it, names ... abk_pdf_bank_statement_to_excel | excel, pdf, canonical_benchmark | ▷ A scanned-style PDF bank statement the agent has to read and hand back as a workbook. The statement c... 2 Set the recurring schedule Schedule ID | Schedule (New York time) | Case tags stack-scheduled-evals | At 02:00 AM, only on Monday and Thursday | reconciliation, excel, mercor_world_2_2 0 2 * * MON,THU 3 Catch regressions automatically Run benchmark → Compare with baseline → Review regressions Same cases + starting worlds → Accuracy + Pass@k → Inspect failed tasks + traces Interface text: Cases | Runs | Schedules ### Takeaways for offline agent evaluation Break the bookkeeping job into tasks and define success for each one. Test those tasks offline with realistic data, then make the results part of how the team routinely judges the agent’s work. [18:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1089s) [Four bullets recommend breaking a job into tasks with clear success criteria, building realistic environments and tasks for offline evaluation, getting more value from those evaluations, and using them consistently to judge agent performance.](https://assets.ai.engineer/sites/aiecode2025/sha256/deef0b6a5e4098e6627f66ec13348b9ea74a67fbdf904bdaeb838432fdfc8f1e.webp) AIRFLOW @ RAMP What are the takeaways • Breakdown a job into the discrete jobs to be done with clear success criteria • Build realistic data environments and tasks for offline evaluation • Focus on extracting as much value from these offline efforts as possible • Figure out how to make it so that offline evaluations are consistently used to judge agent performance ## Full transcript [0:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1s) [music] [0:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=12s) >> Today I'm going to talk about building [0:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=14s) better agents through end-to-end evals. [0:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=16s) I'm going to talk about one of Ramp's [0:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=17s) products, Stacks, a a stack R A I agent. [0:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=22s) Uh so [0:23](https://www.youtube.com/watch?v=WJTQBENv0KI&t=23s) this is a short talk briefly about me, [0:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=25s) about Ramp. Ramp, 70,000 plus [0:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=27s) businesses. [0:29](https://www.youtube.com/watch?v=WJTQBENv0KI&t=29s) Um we have card, travel, AP, AR. We're [0:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=32s) an expense management platform. I lead [0:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=34s) our applied sciences team. These are AI [0:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=36s) engineers, MLEs, product data [0:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=38s) scientists, formerly of Meta, uh and I [0:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=41s) have a PhD in economics. I care a lot [0:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=43s) about measurement and evaluation, which [0:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=45s) is what we're going to talk about today. [0:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=47s) So, [0:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=48s) three things. [0:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=50s) What's your agent's goal? [0:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=51s) Design those evals, and then how did we [0:54](https://www.youtube.com/watch?v=WJTQBENv0KI&t=54s) get uh value from the evals that we [0:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=56s) designed? [0:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=58s) Uh [1:00](https://www.youtube.com/watch?v=WJTQBENv0KI&t=60s) so I'm particularly opinionated about [1:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=62s) this, that people try to identify a [1:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=64s) process that they don't really [1:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=65s) understand. [1:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=67s) And the first task in uh building an [1:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=70s) agent for a process is breaking down the [1:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=72s) job, the tasks, and what success looks [1:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=74s) like. [1:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=76s) So, this is what a bookkeeper's job [1:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=78s) tends to look like. [1:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=80s) So, they want to close the books. We're [1:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=81s) not going to do a ton of accounting [1:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=82s) today, but generally they have a set of [1:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=85s) tasks that they're trying to accomplish. [1:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=87s) When we're building our benchmark, uh we [1:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=90s) created a set of categories. These are [1:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=92s) three of them. There's going to be a [1:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=93s) fourth that you're going to see a little [1:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=94s) later. Um and then we thought about what [1:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=97s) success looks like. Um once we knew the [1:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=99s) tasks, once we knew success, [1:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=101s) we sat down with bookkeepers, we sat [1:44](https://www.youtube.com/watch?v=WJTQBENv0KI&t=104s) down with accounting firms, and then we [1:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=106s) wanted to understand exactly the full [1:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=109s) end-to-end process to solve a specific [1:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=111s) task. This is an example of a [1:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=113s) reconciliation. [1:54](https://www.youtube.com/watch?v=WJTQBENv0KI&t=114s) A reconciliation is actually a pretty [1:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=116s) simple exercise. You have on the [1:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=118s) left-hand side, you have an ERP uh that [2:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=121s) represents the accounts payable ledger. [2:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=125s) Those are bills that you have to pay. If [2:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=127s) you don't know the term accounts [2:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=128s) payable, I didn't until a few years ago [2:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=129s) either. [2:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=131s) And then on the right, you have the [2:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=132s) system that has the bills. In this case, [2:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=134s) you can pay your bills with Ramp. You [2:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=135s) need these two systems to have the same [2:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=137s) numbers. It's a pretty simple exercise. [2:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=140s) What does an accountant do? Well, they [2:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=142s) go into Excel. [2:23](https://www.youtube.com/watch?v=WJTQBENv0KI&t=143s) They bring in the data for the ERP row [2:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=146s) by row. They bring in the data from the [2:29](https://www.youtube.com/watch?v=WJTQBENv0KI&t=149s) system that is accounts payable. [2:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=151s) Do they reconcile? Do they tie? Yes, no. [2:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=154s) And then they figure out why they don't, [2:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=156s) and then they make them tie. [2:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=158s) Pretty easy to identify this process if [2:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=161s) you understand exactly what it is. [2:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=163s) So, we want to build a benchmark. [2:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=166s) Uh I'm going to interchangeably use [2:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=169s) terms like benchmarks, evals, [2:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=171s) verifiability, [2:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=173s) because frankly, I think it's a soup of [2:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=176s) language [2:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=177s) that are all trying to do the same [2:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=178s) thing. These are the four sets of [3:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=181s) categories of things [3:03](https://www.youtube.com/watch?v=WJTQBENv0KI&t=183s) that we wanted to eval. There's [3:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=186s) reconciliation, what I just described, [3:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=188s) data entry, variance analysis, it's why [3:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=191s) do numbers move, [3:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=192s) schedules and accruals, not going to go [3:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=194s) deep into there, [3:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=195s) but it is a complex thing that an [3:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=197s) accountant is doing. [3:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=200s) So, this is just like a very high-level [3:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=204s) schematized version [3:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=206s) of how to think about the process of [3:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=208s) building evaluations. [3:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=210s) You can think about the state before you [3:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=212s) have users or at least enough users that [3:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=214s) you think you have a good distribution [3:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=215s) of data. And then with users. So, before [3:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=219s) you have users, remember all agents are [3:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=221s) doing is they're calling tools that hit [3:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=223s) a database. That's it. You don't have [3:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=225s) users. You need simulation. You need [3:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=229s) synthetic data. [3:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=231s) I would highly recommend anytime you're [3:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=233s) building an agent for an area where you [3:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=235s) don't have enough users, you should [3:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=238s) build synthetic data. Um [4:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=241s) we built this benchmark in partnership [4:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=244s) with a company called Mercore. [4:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=246s) Uh, they were a great partner for this. [4:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=248s) Um, you don't have to build it with a [4:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=250s) company, um, but simulation is pretty [4:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=253s) important. Once you have users, things [4:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=255s) get a little easier, if not harder. [4:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=258s) We're not going to talk about that phase [4:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=259s) in this talk. I only have 18 minutes. [4:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=261s) That would be longer than an 18-minute [4:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=264s) talk. Um, so data is the bottleneck to [4:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=267s) evals. [4:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=268s) I think that's an obvious statement, but [4:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=270s) I want that to be like very clear that [4:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=272s) to evaluate, you have to have clean data [4:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=273s) to do things on. [4:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=276s) So, [4:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=277s) how do we design the evals? Like, [4:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=279s) how did we think about making a data [4:42](https://www.youtube.com/watch?v=WJTQBENv0KI&t=282s) set, a set of task criteria that we [4:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=286s) would get value out of? [4:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=288s) Uh, so we built synthetic businesses. [4:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=291s) Uh, this is an example of one of our [4:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=293s) synthetic businesses. [4:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=295s) Uh, they're a D2C lifestyle company. Uh, [4:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=298s) they use Amazon. Uh, there's some [5:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=301s) complexity to how they do their [5:03](https://www.youtube.com/watch?v=WJTQBENv0KI&t=303s) accounting. They have a calendar year. [5:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=306s) They want to do just close at the end of [5:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=308s) the month. They're multi-channel. They [5:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=310s) get revenue from Shopify, Amazon, a [5:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=313s) bunch of different stuff. Uh, the [5:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=315s) details of this are not that important [5:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=318s) of who this company is, but what we also [5:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=320s) did is we built a set of documents or [5:23](https://www.youtube.com/watch?v=WJTQBENv0KI&t=323s) data for this company. We built their [5:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=325s) ERP. We built their customer list, their [5:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=328s) vendor list, the accounts payable, [5:29](https://www.youtube.com/watch?v=WJTQBENv0KI&t=329s) everything [5:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=331s) that would be needed for a synthetic [5:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=333s) business, [5:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=334s) uh, in order to do good offline [5:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=336s) evaluation. [5:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=338s) Uh, [5:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=339s) all we're doing here, by the way, is [5:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=341s) just backtesting agents and measuring [5:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=343s) success. Like I said, um, [5:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=347s) people use these terms interchangeably. [5:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=350s) My training is as an MLE. Uh, before I [5:54](https://www.youtube.com/watch?v=WJTQBENv0KI&t=354s) came at Ramp, I was the first MLE at [5:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=356s) Ramp. [5:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=357s) Um, and I think one of the problems on [6:00](https://www.youtube.com/watch?v=WJTQBENv0KI&t=360s) the discourse on evaluations, [6:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=362s) verifiability, and benchmarks, [6:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=365s) uh, is that they're just trying to find [6:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=367s) labels. They're just trying to find ways [6:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=370s) to measure success for things. [6:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=373s) Um and they're all sort of saying the [6:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=375s) same thing. [6:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=376s) Uh but really you're just looking to run [6:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=379s) back tests. You're looking to run your [6:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=381s) agents against the same set of data [6:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=385s) day over day over day to find ways to [6:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=388s) optimize it. I sort of have a theory [6:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=390s) that the industry's been a little um [6:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=394s) uh I the term is not hurt, but I think [6:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=397s) we had it good when it's just coding [6:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=399s) agents cuz Git by nature allows you to [6:42](https://www.youtube.com/watch?v=WJTQBENv0KI&t=402s) replay things in the past. State is [6:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=405s) always captured. When you start to think [6:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=407s) about production agents, [6:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=410s) how do you capture state over time so [6:52](https://www.youtube.com/watch?v=WJTQBENv0KI&t=412s) you can replay? [6:54](https://www.youtube.com/watch?v=WJTQBENv0KI&t=414s) Uh we're in a nice place when we have no [6:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=417s) users. We build that state and we keep [6:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=419s) that state the same over time. [7:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=422s) But that's really just our goal. We're [7:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=424s) running a back test. We're measuring [7:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=426s) success. [7:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=427s) And so this is is sort of what the [7:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=430s) trajectory of the process looks like. We [7:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=433s) build a task, in this case a [7:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=435s) reconciliation. [7:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=437s) We call these things worlds. It's a term [7:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=439s) that Mercor [7:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=441s) um came up with and I quite like it. Uh [7:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=444s) a world could also be your production [7:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=446s) data, a snapshot of it. There's [7:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=450s) no reason to swap that in. It's the same [7:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=452s) thing. [7:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=453s) We have the agent run and then we just [7:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=456s) evaluate on the final output. You can [7:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=458s) see the criteria that we have. For each [7:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=461s) task we tended to have anywhere from 10 [7:44](https://www.youtube.com/watch?v=WJTQBENv0KI&t=464s) to 20 criteria. We didn't judge the [7:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=467s) trajectories um themselves like were the [7:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=470s) right tools called, were the right [7:52](https://www.youtube.com/watch?v=WJTQBENv0KI&t=472s) skills invoked? Did it take an efficient [7:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=476s) path? We do care about latency, cost, [7:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=479s) and like success. Um [8:03](https://www.youtube.com/watch?v=WJTQBENv0KI&t=483s) there are different reasons you might [8:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=485s) care about trajectories, but when you're [8:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=487s) initially building your agent, you just [8:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=488s) want it to be right, to be cheap, and to [8:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=491s) be fast. How it gets there is a little [8:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=494s) bit less important. [8:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=496s) Uh so, that's not what we focused on. [8:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=499s) And like I had said my mini rant before, [8:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=502s) in this as example, we're looking for a [8:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=505s) few things that define success. If I [8:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=508s) were just in MLE, I would say these are [8:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=510s) the labels. Um the thing I want to say [8:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=513s) is the process to build this whole thing [8:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=515s) was [8:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=517s) 4 to 6 months before launching this [8:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=519s) agent. And I think that level of time [8:42](https://www.youtube.com/watch?v=WJTQBENv0KI&t=522s) investment in evals uh is typical in [8:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=526s) companies that are trying to build [8:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=529s) agents for [8:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=531s) areas that are not fully optimized uh by [8:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=535s) the frontier models. You're throwing [8:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=536s) something as a coding agent, sure, it'll [8:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=539s) probably work great out of the box. If [9:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=542s) you're in a front an a frontier area, an [9:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=544s) area that's [9:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=546s) slightly orthogonal to what they're [9:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=548s) optimized against, um you need to spend [9:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=551s) some time building these things. [9:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=554s) So, [9:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=555s) how do we measure performance? I'm sure [9:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=556s) there's a lot of people in this room [9:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=558s) that have seen these metrics. Um one is [9:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=560s) accuracy, that's pretty simple. So, like [9:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=562s) I said, this given task had six [9:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=565s) criteria, four of those six are right, [9:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=568s) 67%, so partial credit counts. Uh the [9:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=572s) other way um that you'll see reported a [9:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=574s) lot is pass at K. Um this is an example [9:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=577s) of three runs on this given agent. Uh in [9:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=581s) the first run, it got four right. Second [9:44](https://www.youtube.com/watch?v=WJTQBENv0KI&t=584s) run, it got five right. Final attempt, [9:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=587s) it got six right, so it got all of them [9:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=589s) right. Pass at K is an all or nothing [9:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=591s) metric. Um you could think for [9:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=595s) areas where the cost of error is very [9:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=598s) high, maybe like accounting, uh pass at [10:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=601s) K might be the thing you're optimizing [10:03](https://www.youtube.com/watch?v=WJTQBENv0KI&t=603s) for. You need everything to be right. If [10:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=605s) an accountant can't trust that one of [10:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=607s) the steps is wrong, [10:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=609s) then they may not use your agent. Um so, [10:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=613s) this is pass at K. You can see pass at [10:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=615s) one is zero, pass at two is zero, and [10:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=618s) then pass at three is one because it [10:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=619s) finally solved it. [10:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=622s) Okay. So, now I'm going to talk about [10:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=624s) that's how we built the evals. That's [10:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=625s) what our evals were. [10:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=627s) I'm going to talk about all the things [10:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=628s) we now do with these evals now that we [10:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=630s) spent that four to six months, now that [10:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=632s) we spent all the money on it and time. [10:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=634s) I mean, the first obvious case, right, [10:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=635s) is you're going to like pick a frontier [10:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=637s) model. You're going to be like, which [10:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=639s) model is best at this? Um this is a [10:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=641s) little old. This is from a blog post we [10:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=643s) wrote when Stack launched. Um as you [10:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=646s) could sort of see at the time, [10:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=648s) uh the Opus models uh were the highest [10:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=651s) performing. That's still what we see [10:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=653s) when we rerun these benchmarks. 55 is [10:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=656s) the best model. What you'll also see is [10:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=658s) accuracy is very high. Pass at one is [11:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=661s) very low. [11:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=662s) These aren't saturated yet, which is [11:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=664s) great. Um it's great for us, too, [11:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=667s) because we find harness optimizations [11:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=669s) allow us to get extra pass at. Um so, it [11:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=673s) means you can't just throw a frontier [11:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=675s) model at this problem and have it [11:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=677s) solved. [11:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=678s) We dug deeper into why the models are [11:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=682s) failing to spot areas for optimization. [11:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=685s) Uh so, we looked at things like [11:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=686s) reasoning failure. Does it have all the [11:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=688s) data? It's just reasoning through it [11:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=690s) incorrectly? [11:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=691s) Uh does it not gather the right [11:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=693s) information? Does it not follow [11:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=694s) instructions? [11:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=696s) Um [11:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=697s) and then like reflection failure, which [11:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=698s) is I would actually call it other. Um [11:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=701s) interesting result that we see is that [11:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=703s) for accounting, it's reasoning failure. [11:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=705s) What that means is that these harnesses [11:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=707s) are pretty good at actually gathering [11:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=708s) the right amount of data. Accounting is [11:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=710s) not that difficult of an [11:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=713s) gathering exercise. Accounts payable is [11:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=715s) like something I could explain to like a [11:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=718s) five-year-old, so you could sort of get [12:00](https://www.youtube.com/watch?v=WJTQBENv0KI&t=720s) a well, sophisticated five-year-old. Um [12:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=722s) so, you could sort of get uh what files [12:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=725s) you would need. Uh and so, what this [12:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=727s) sort of says is like, you know, the [12:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=728s) models, they just ain't that smart. [12:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=731s) Uh and we've got some room to run here. [12:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=734s) Uh, what was interesting was Music Spark [12:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=735s) actually did really well in this. [12:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=737s) Um, so we ran a ton of experiments. [12:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=740s) These are all the things that failed. [12:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=742s) Um, I sort of like this because like [12:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=744s) you'll see papers of like these awesome [12:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=748s) things that they did to optimize and [12:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=751s) tune their agents and they don't show [12:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=753s) you the 30 things that failed. Actually [12:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=755s) want to highlight one. I'm going to go [12:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=757s) on like a mini side quest for a moment [12:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=759s) here. [12:40](https://www.youtube.com/watch?v=WJTQBENv0KI&t=760s) Uh, Jeepa, auto tuning, auto researching [12:44](https://www.youtube.com/watch?v=WJTQBENv0KI&t=764s) skills. [12:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=765s) No good. Uh, [12:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=768s) it might be our agent, [12:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=770s) um, but I think the idea of hill [12:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=773s) climbing with evals on text space [12:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=777s) is not the first thing I would do here. [13:00](https://www.youtube.com/watch?v=WJTQBENv0KI&t=780s) Uh, intuitively it's difficult to [13:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=782s) understand why it would work, i.e. how [13:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=785s) you can get direction on a gradient. Um, [13:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=788s) we did not see success here. We were [13:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=790s) still not seeing success. I just want to [13:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=792s) call this out because I think there's a [13:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=793s) lot of noise on auto research. Um, the [13:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=797s) Karpathy famous tweet is about tuning ML [13:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=801s) models, which obviously it has a [13:23](https://www.youtube.com/watch?v=WJTQBENv0KI&t=803s) gradient, it has a direction. It's also [13:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=805s) a finite space it's searching over. [13:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=807s) Great. Um, so one lesson to take away [13:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=810s) from here is maybe don't think that auto [13:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=811s) research is going to solve your [13:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=814s) bad agent problem. [13:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=816s) So here's some experiments that did [13:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=817s) work. [13:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=818s) Um, I'm going to talk about two. [13:40](https://www.youtube.com/watch?v=WJTQBENv0KI&t=820s) I'm going to talk about skill ablation [13:42](https://www.youtube.com/watch?v=WJTQBENv0KI&t=822s) and trajectory distilled memories. So [13:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=826s) I'm an MLE by training. [13:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=828s) Uh, usually what you do is you build a [13:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=830s) bunch of features, [13:52](https://www.youtube.com/watch?v=WJTQBENv0KI&t=832s) uh, throw them in a model, [13:54](https://www.youtube.com/watch?v=WJTQBENv0KI&t=834s) and then you might do like a leave one [13:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=836s) out feature, or you might add features [13:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=839s) back in. It's all just trying to assess [14:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=842s) the incrementality [14:03](https://www.youtube.com/watch?v=WJTQBENv0KI&t=843s) of a given feature on model performance. [14:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=846s) Uh, agents with many skills, many tools. [14:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=849s) We didn't do with tools, we did with [14:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=850s) skills. Tools are a little bit more [14:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=852s) difficult to deal with, but you could do [14:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=853s) a similar thing. [14:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=855s) Uh, we just tried to trim the number of [14:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=858s) skills saving context. That's your goal. [14:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=861s) Um, and what this basically shows is if [14:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=864s) you look at no skill baseline, basically [14:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=866s) this is you could just think it's like [14:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=867s) the raw model, whatever's in the [14:29](https://www.youtube.com/watch?v=WJTQBENv0KI&t=869s) weights, [14:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=870s) um, versus [14:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=872s) uh, [14:33](https://www.youtube.com/watch?v=WJTQBENv0KI&t=873s) removing uh, uh, [14:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=875s) versus like adding the skill. I I think [14:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=877s) I've reversed these actually. I think [14:38](https://www.youtube.com/watch?v=WJTQBENv0KI&t=878s) the negatives are like it's good with [14:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=881s) the skill and the positives is bad with [14:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=883s) the skill. Regardless, um, what you can [14:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=885s) see is there's definitely differences in [14:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=887s) skills being there versus not being [14:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=889s) there. And this created a candidate set [14:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=891s) for us to then be like maybe we should [14:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=893s) delete these skills, compress these [14:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=895s) skills, [14:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=897s) basically make things more efficient. [14:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=899s) So, skill ablation, you should do it. [15:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=901s) Tool ablation, you can also do. [15:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=904s) Trajectory distilled memory. So, [15:06](https://www.youtube.com/watch?v=WJTQBENv0KI&t=906s) a lot of people talk about memories. Uh, [15:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=908s) I think the rise of personal agents, [15:10](https://www.youtube.com/watch?v=WJTQBENv0KI&t=910s) people talk about the memories and how [15:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=912s) good they are, [15:13](https://www.youtube.com/watch?v=WJTQBENv0KI&t=913s) blah, blah, blah. [15:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=915s) Um, [15:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=916s) but people don't really test their [15:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=917s) memory systems. Um, or at least I don't [15:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=920s) tend to see people testing their memory [15:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=922s) systems. They'll have a prompt, they'll [15:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=924s) compress some information, they'll put [15:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=925s) it into some store, uh, and then they'll [15:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=927s) read it back. Uh, so, one little [15:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=930s) innovation we did here is what we call [15:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=932s) roll forward worlds. So, it's a little [15:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=935s) complex, so I'm going to walk you [15:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=936s) through that chart on the left. Uh, but [15:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=939s) basically, uh, you do the same task over [15:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=943s) two time periods. [15:45](https://www.youtube.com/watch?v=WJTQBENv0KI&t=945s) You write a memory after the first task, [15:49](https://www.youtube.com/watch?v=WJTQBENv0KI&t=949s) uh, and then you either inject the [15:51](https://www.youtube.com/watch?v=WJTQBENv0KI&t=951s) memory in if you want to cheat a little [15:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=953s) bit, or you let the agent search over a [15:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=957s) set of memories, [15:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=958s) um, for the same task in the following [16:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=961s) period. So, [16:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=964s) in the example I have here is like uh, [16:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=967s) we want to match some records, do a [16:08](https://www.youtube.com/watch?v=WJTQBENv0KI&t=968s) reconciliation. There's some implicit [16:11](https://www.youtube.com/watch?v=WJTQBENv0KI&t=971s) customer rules and how they like to do [16:14](https://www.youtube.com/watch?v=WJTQBENv0KI&t=974s) that. Uh, [16:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=976s) those will be not fully encoded, but you [16:19](https://www.youtube.com/watch?v=WJTQBENv0KI&t=979s) can infer them from the success criteria [16:22](https://www.youtube.com/watch?v=WJTQBENv0KI&t=982s) like you have to learn these rules in [16:25](https://www.youtube.com/watch?v=WJTQBENv0KI&t=985s) order to be successful at this task. We [16:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=988s) have a memory prompt that then [16:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=990s) compresses, writes, and then we rerun [16:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=992s) it, and it reads again. And what we [16:34](https://www.youtube.com/watch?v=WJTQBENv0KI&t=994s) actually saw that there is a ton of [16:36](https://www.youtube.com/watch?v=WJTQBENv0KI&t=996s) alpha here. Like the types of memories [16:40](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1000s) that were written for tasks really [16:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1003s) depended on the prompts, and then [16:47](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1007s) success was pretty in like also [16:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1010s) dependent on those prompts. We saw a [16:52](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1012s) very large gain. This was actually one [16:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1013s) of the biggest gains we saw from having [16:56](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1016s) uh, these types of evals. [16:58](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1018s) So, [16:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1019s) uh, you have a great data set uh, that [17:04](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1024s) you can run evals on. The thing that I [17:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1027s) think people don't do enough of is what [17:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1029s) I call regression testing. I'm not a big [17:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1032s) believer in CI gating based off of [17:16](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1036s) agentic evals as someone that's seen a [17:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1038s) lot of non-determinism, a lot of 80%, [17:20](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1040s) 70% passage. Uh, CI gating is [17:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1044s) not usually the best idea, but instead [17:27](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1047s) what we do now is we have an internal [17:29](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1049s) platform that hosts our evals that [17:31](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1051s) people can add the evals to. This top [17:35](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1055s) uh, picture you see is an example of [17:37](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1057s) stack. We have a bunch of different [17:39](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1059s) cases. We actually tag these cases so [17:41](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1061s) that we can cluster and organize them a [17:43](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1063s) little bit better. We set a recurring [17:46](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1066s) schedule, so these should run in the [17:48](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1068s) background every week, every day, [17:50](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1070s) whatever. We run it, we see how things [17:53](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1073s) perform. We have a background agent that [17:55](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1075s) then looks at like failures, [17:57](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1077s) regressions, all of those things. Uh, [17:59](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1079s) and then we get a Slack message that [18:01](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1081s) tells us, "Ooh, it looks like [18:02](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1082s) something's regressed. Go investigate [18:05](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1085s) it. Here's what I think broke." [18:07](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1087s) So, getting the value out of your evals [18:09](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1089s) is important. So, takeaways. I'm at [18:12](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1092s) time. Break down a job into discrete, [18:15](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1095s) you know, tasks. Build realistic [18:17](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1097s) environments. [18:18](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1098s) Focus on extracting like value as much [18:21](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1101s) as possible Um, and then figure out how [18:24](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1104s) to consistently put those into workflows [18:26](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1106s) to measure performance. [18:28](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1108s) Cool. Thank you for listening and yeah, [18:30](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1110s) appreciate the time. [18:32](https://www.youtube.com/watch?v=WJTQBENv0KI&t=1112s) >> [applause]