Our Smaller Model Beat Opus on Finance Tasks - Alexander Shevchenko, Ramp

Give Ramp Sheets a stack of spreadsheets and ask it to build a discounted cash flow model. Before it can model anything, the agent has to work out what each sheet contains and where to find the figures its plan calls for. Alexander Shevchenko of Ramp Labs said this was where users experienced the most friction. Opus 4.6 spent a long time planning and exploring the sheets, and Ramp paid for the reasoning tokens it used along the way.

That delay gave Ramp a specific reason to invest in its own model. Waiting for a better frontier release had generally paid off for the company. Shevchenko reported that the number of LLM use cases at Ramp had risen 123 percent over a year. On several major features, growing total token bills coincided with falling costs per use as newer models delivered comparable accuracy more cheaply. But the spreadsheet agent already had a job Ramp could isolate and improve. An accountant waiting for a financial model would benefit from faster answers to the questions that held up the plan.

Ramp split the work at the point where the planner needed information from the spreadsheets. The frontier model retained the overall assignment and could hand a financial question to a specialist, then use the answer to continue building the model. That division gave the team a narrower training target. The specialist needed to answer questions about financial data accurately enough for the planner to rely on it. Responsibility for deciding which questions to ask stayed with the planner.

Working with Prime Intellect, Ramp post-trained a 35-billion-parameter Qwen 3.5 model for the question-answering role. Shevchenko reported that it exceeded Opus 4.6 on exact task accuracy and that Ramp could serve it at Haiku-level latency. Ramp shipped the two-model arrangement. The comparison applies to the questions handed off to Qwen. Accuracy on those questions and the time it takes to return an answer are distinct from the quality and completion time of the finished spreadsheet model, for which Shevchenko gave no figures.

A spreadsheet upload can come as a surprise. Month-end close is on the calendar, and the team already knows much of what the accountant will need on the first of the month, including the ledger and the aging report. Simply making those documents available still leaves the agent with retrieval work when the accountant opens it. Ramp explored how much of that work it could do overnight.

The technique uses a cartridge, a compressed representation of context prepared offline for the model’s key-value cache. This cache holds internal representations the model uses to attend to earlier material. Shevchenko said cartridges could pack up to 20 to 30 times as much context into the cache. For the close, Ramp could build a cartridge from the anticipated documents the night before. The accountant would begin with an agent that already had those materials in context. The reported multiplier describes context capacity. Shevchenko gave no corresponding speedup for completing a close.

In Ramp’s experience, cartridges took less effort than specialist post-training because the self-play procedure used to build them generalized more readily. Their practical advantage here depends on knowing which material to prepare and when someone will need it. Shevchenko also acknowledged that frontier labs might eventually offer the capability through an API. Ramp saw value in learning which workflows benefit from advance preparation even if a provider later takes over the mechanics.

For penetration testing, Ramp could afford to pursue many independent lines of inquiry, provided it had a way to handle the tickets they produced. Using its Ramp Inspect cloud agent infrastructure, the company ran 10,000 agents in parallel to look for vulnerabilities. The search was the map phase. In the reduce phase, LLMs deduplicated the tickets and tried to reproduce the suspected vulnerabilities adversarially. That second pass gave Ramp a way to separate a plausible report from a security issue it could act on.

Shevchenko said the exercise found seven high-severity vulnerabilities, all of which Ramp patched. One was a Lua script injection that could be delivered by email. In his account, each component along the path was safe in isolation. Identifying the vulnerability required following how the components interacted from the email through to script injection. That finding illustrates why a parallel search still needs room for an investigator to reason across component boundaries.

Ramp also tried open-weight models that Shevchenko said were up to five times cheaper than frontier reasoning models. In the swarm, allocating more attempts to the cheaper models brought detection close to the frontier result. Failed investigations still consumed tokens, but automated filtering limited the human time spent on those passes. The relevant comparison for this design is the cost of a search that produces reproducible findings. The price of one attempt tells you only part of that.

A bill grouped by API key or employee tells a finance team who incurred a charge. It can still leave the team unable to distinguish spending on spreadsheet research from spending on a security investigation. Ramp applied the same map-and-reduce idea to understanding LLM use. Agents examined sessions and tagged the activities they found. The categories could emerge from the sessions themselves, allowing Ramp to build a taxonomy without first enumerating every way people might use a model.

Reconciling those categories with spend gives an engineering team a better unit for comparing interventions. For spreadsheet research, the useful observations were question-answering accuracy and how long the planner waited. For month-end close, the opportunity came from moving predictable retrieval work outside the accountant’s session. For security, a token budget could fund more searches as long as the system could reproduce the findings. All three workloads consumed tokens, but each called for a different experiment.