How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

A company keeps a ledger of the bills it owes in its enterprise resource planning system, or ERP. It can also use Ramp to pay those bills. At close, the accountant needs the two systems to agree. That gave Ryan Stevens, who leads applied science at Ramp, a concrete starting point for evaluating Stack, its bookkeeping agent. The team sat down with bookkeepers and accounting firms to learn what a person actually does when the numbers differ.

Which records need to agree? The accountant brings the accounts payable ledger from the ERP into Excel row by row, then brings in the records from the system that holds the bills. If the numbers differ, the accountant works out why and resolves the discrepancy. That last step matters for an end-to-end test. An agent can correctly flag a mismatch and still leave the accountant with the work of reconciling it.

Reconciliation became one of four benchmark categories. Ramp also tested data entry, variance analysis (explaining why numbers moved), and schedules and accruals. Breaking the close into tasks at this level let the team define what a successful result looked like for each one.

Before Stack had enough users to supply a representative set of tasks, data was the bottleneck. The agent would call tools that queried a database. What should those tools find when it went looking for a bill? Ramp built the benchmark in partnership with Mercor. Synthetic businesses let them supply the records on both sides of a reconciliation.

One business was a direct-to-consumer lifestyle company that earned revenue through Shopify and Amazon, followed a calendar year, and closed its books monthly. Alongside the company description, the team built its ERP data, customer and vendor lists, and accounts payable records. A reconciliation could now send Stack into the books of a particular retailer, with its own counterparties and bills.

Keeping those books fixed made the evaluation a back-test. Git gives a coding agent a convenient way to revisit an earlier version of a repository. Replaying a bookkeeping task requires the corresponding state of the business. With a synthetic company, Ramp controlled that state from the start. It could give a changed agent yesterday’s task against yesterday’s records and compare the results. A snapshot of production data could serve the same purpose.

What does the grader inspect once Stack finishes a reconciliation? Ramp paired the task with a fixed business environment, borrowing Mercor’s term world. It let the agent run, then checked the final output against task-specific criteria. A typical task had 10 to 20. The agent could earn full credit regardless of which tools it called, which skills it invoked, or which route it took. Ramp cared about cost and latency too, both of which it could assess without prescribing a sequence of tool calls.

Building the evaluation took four to six months before Stack launched. In Stevens’s terms, the criteria were labels. Once the team had specified them, the same labels could answer two different questions about performance.

Take the example with six criteria. Across three attempts at the same task, the agent satisfied four criteria, then five, then all six.

• Accuracy asks how much of the work was right on a given run. The first attempt scored 4/6, or about 67%. The second scored 5/6 even though one part still needed attention.
• Pass@k asks whether at least one of k attempts satisfied every criterion. In this sequence, pass@1 was 0, pass@2 was still 0, and pass@3 was 1. Only the third attempt produced a complete answer.

An engineer can learn something from the difference between four and five. An accountant handed the second result still has to find the remaining error. Where a single mistake can undermine trust in the whole reconciliation, the chance of getting a complete answer on the first attempt deserves its own measure.

How often did high accuracy translate into a complete answer? In the model comparison Ramp published when Stack launched, the Opus models performed best. Yet criterion accuracy was high while first-attempt pass rates were low. Even the leaders left work unfinished often enough for Ramp to measure improvements. Changing the software around the model gave the team more complete passes.

To decide what to change, the team examined failed runs. Had the agent gathered the wrong information? Had it failed to follow an instruction? Sometimes it had all the data and drew the wrong conclusion, which Ramp classified as a reasoning failure. A fourth category, reflection failure, served largely as a catch-all.

Reasoning was the main failure on these accounting tasks. Think back to accounts payable. Once the agent understands that it needs records of bills the company owes, identifying the relevant files is relatively straightforward. The runs showed that an agent could gather those files successfully and still reason incorrectly about their contents. Better retrieval alone would leave that failure intact.

With a repeatable score, Ramp tried automatically tuning the text of prompts and skills. The idea was to try a rewrite, evaluate the resulting agent, and use the score to search for better wording. That search failed to deliver a gain for Stack. Stevens’s objection came down to direction. When tuning a model, a gradient can indicate which way to adjust its parameters. An evaluation can tell you that one rewrite did better than another and still give you little guidance about which words to change next.

Skill ablation posed a narrower question. A skill takes up space in the agent’s context. How much performance does its presence buy? Ramp borrowed the logic of feature ablation in machine learning, where removing a feature or adding it back helps isolate its contribution. A no-skill baseline measured what the model could do with its training alone as guidance. Runs with a skill present and absent tested the value of supplying that extra guidance on the same work. This experiment varied skills. Stevens suggested tools as a possible follow-up.

Reading an individual result depends on which condition is subtracted from which. Stevens flagged a possible reversal in his reading of the chart signs, so the direction of a particular skill’s effect is uncertain from his explanation. The comparisons did show that changing the skill set changed performance. They gave Ramp a shortlist of skills to examine for deletion or compression, along with a way to test whether a shorter configuration still did the job.

Writing a summary to a memory store and reading it back demonstrates that storage works. Ramp wanted to know whether the contents helped Stack do the next period’s work. Its roll-forward worlds gave the same business the same kind of task in two successive periods. A memory-writing prompt compressed the first run into something the agent could use during the second.

How could a record-matching rule learned in one period reach the next reconciliation? Customers can have conventions for matching records that they never fully spell out. Ramp built an example where satisfying the criteria required inferring those conventions. After the first reconciliation, the experiment could put its memory directly into the next run’s context or let the agent search a set of memories. Direct injection tests whether the information helps once supplied. Search also asks whether the agent can find it when it needs it.

Changing the memory-writing prompt changed what the agent preserved about the task. It also changed how well the agent performed in the following period. Stevens described this as one of Ramp’s largest gains from the evaluations. The second close gave the team a way to judge a summary by what another run could accomplish with it.

A repeatable world still contains a nondeterministic agent. Stevens had seen tests pass 70% or 80% of the time. With that much variation, a single failure makes a poor reason to block a code change in continuous integration. Ramp put its evaluations on an internal platform where people could add cases, tag them into groups, and schedule them to run in the background daily or weekly.

After a scheduled run, a background agent examines the results for failures and regressions. When it spots a possible decline, a Slack message flags the issue and suggests what might have broken. Ramp now uses the benchmark to catch regressions in familiar bookkeeping work, with a concrete lead for the engineer who investigates.