# Our Smaller Model Beat Opus on Finance Tasks - Alexander Shevchenko, Ramp

Alexander Shevchenko, Ramp

[Talk page](https://ai.engineer/talks/T-42NrYjwxM-our-smaller-model-beat-opus-on-finance) · [Video](https://www.youtube.com/watch?v=T-42NrYjwxM)

## How Ramp’s 35B Model Beat Opus 4.6 on Financial Question Answering

Ramp Labs kept Opus in charge of financial modeling, trained a faster model to answer its questions, and found two other ways to save time on agent work: prepare the context overnight or run thousands of investigations in parallel.

### Why do research when waiting keeps paying off?

[0:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=12s)

Alexander Shevchenko had spent about three years on Ramp’s applied AI team, working on products for financial professionals. He described Ramp as a company built on the bitter lesson. Over the preceding year, the number of use cases employing LLMs at Ramp had grown 123 percent. Several of its most-used features consumed more tokens in aggregate even as their unit economics improved. New frontier models let Ramp deliver the same accuracy at a lower cost per use. That record raised a fair question about Ramp Labs. What could an application company gain from doing its own research while model providers were already making its products faster and cheaper?

![From October 2025 to October 2026, the active use cases chart rises 123%. Three smaller charts trend downward, with declines of 78% for approval intelligence, 90% for underwriting and 81% for vendor mailbox.](https://ai.engineer/api/talk-essay-media/550e8d0963b2fbfbdacf2c4a67fa983befd5e90fb3fbeeb992eb94f34d42ea10)

We’re a bitter-lesson-pilled company

Shevchenko recalled building agents with GPT-4 in 2023. Once an agent took a wrong turn, it tended to keep going. The models lacked thinking tokens and had little reinforcement training to backtrack, so engineers wrote strict guardrails around the steps an agent could take. Even substituting a synonym in a prompt could change its behavior. Engineers also pared down the context because too much material could confuse the model. Ramp did that work so financial professionals could use Ramp Tour Guide and other agents without waiting for GPT-4.5 or GPT-5.

By 2026, engineers could give a model tools through MCP and access to a much wider body of context, and it could usually work out what the task required. Ramp’s reason to go further came from watching where people still had to wait. The first example surfaced in a product that already worked.

### A discounted cash flow model exposed the cost of reading the sheets

[3:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=192s)

Ramp launched Ramp Sheets in November 2025, before either of the two frontier labs Shevchenko referred to had released a spreadsheet agent. Customers began using it as a general-purpose spreadsheet tool. Accountants and finance professionals spent much of their day in spreadsheets, as did people working inside Ramp. The team saw an opportunity to reuse the agent’s harness and underlying model in other parts of the product. As it did that, it looked for the step that caused users the most friction.

One revealing assignment was to give Ramp Sheets a collection of spreadsheets and ask it to build a discounted cash flow model or model a financial scenario. Opus 4.6 first had to make a plan. Then it had to explore the available sheets, establish what each contained and find the financial information the plan called for. From the user’s perspective, much of the wait came before the requested model took shape. From Ramp’s perspective, it was paying for Opus thinking tokens while the agent got its bearings.

![A Ramp Sheets workbook shows Anthropic and OpenAI projections beside an assistant’s completed checks. The slide assigns planning and editing to a frontier model and spreadsheet questions to the FastAsk specialist.](https://ai.engineer/api/talk-essay-media/b0637abac2122034038a36fdec9ebe28cfee5befea41637789b09ca65e4ab4e5)

A fast specialist for a smart orchestrator

### Opus keeps the plan and Qwen answers the financial questions

[4:43](https://www.youtube.com/watch?v=T-42NrYjwxM&t=283s)

Ramp split the assignment where the planner needed an answer from the data. The frontier model remained the orchestrator. It could decide what it needed to know, hand a financial question to a faster model and use the answer to continue its plan. Shevchenko described the specialist first as a retrieval model and then, more precisely, as a financial question-answering model. Its job was to return an answer the planner could use, which gave Ramp a specific capability to train and measure.

Working with Prime Intellect, Ramp post-trained a 35-billion-parameter Qwen 3.5 model. On exact task accuracy for financial question answering, it scored above Opus 4.6, the frontier model Ramp was using at the time. Ramp could also serve the specialist at Haiku-level latency. Those comparisons describe two properties of the handoff: how often the planner got the right answer and how long it waited for that answer. Shevchenko gave no end-to-end timing or accuracy figure for the finished financial model. Ramp shipped the specialist and frontier orchestrator together.

![A scatterplot compares exact-task accuracy with time per task relative to Claude Haiku 4.5. FastAsk sits above the dashed frontier at 66.25% accuracy and just over 1.0× Haiku’s time. Opus reaches 61.88% at roughly 1.44×.](https://ai.engineer/api/talk-essay-media/20b815f7d519193dd87cfae714d6b13cbc75691e520fc2b7866c6c949776bbbb)

Opus-level accuracy at Haiku-level speed

### Build a KV cache cartridge the night before month-end close

[5:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=342s)

A user can drop an unfamiliar workbook into Ramp Sheets at any time. Month-end close gives Ramp advance notice of both the task and much of its input. On the first of the month, a financial professional will need to reconcile and close the books. Ramp can anticipate documents such as the ledger and the aging report. Making those files available still leaves the agent with a job when the accountant arrives. It has to retrieve the relevant material and build up a picture of the accounts.

A cartridge moves that preparation offline. It compresses context into the model’s key-value cache, the internal representations the model uses to attend to earlier material. Shevchenko said cartridges could pack up to 20 to 30 times as much context into the cache. In the close example, Ramp could build a cartridge overnight from the anticipated documents. When the accountant opened the agent the next morning, it would begin with that context already available. The multiplier measures how much context fits. The time saved on a particular close depends on how much retrieval and preparation the cartridge covers.

![Two timelines compare reconciliation workflows. In the first, a long reading stage precedes reconciliation. In the second, a narrow yellow mark represents loading a cartridge built overnight, and the reconciliation stage starts almost immediately.](https://ai.engineer/api/talk-essay-media/a1d741ea08504ad123d626ea6e4ff5143189ea01f9df369b9abb5c7d5419c25c)

KV cartridges hand the agent the whole month

Shevchenko found cartridges less work to build than a specialist model because the self-play procedure generalized more readily. He also saw a reason a frontier lab might eventually offer the capability through an API as part of its work on continuous learning. Ramp would still have learned which jobs it could anticipate, what material those jobs required and when to prepare it.

### 10,000 agents search Ramp and a second pass tests their findings

[7:50](https://www.youtube.com/watch?v=T-42NrYjwxM&t=470s)

For penetration testing, Ramp used its cloud agent, Ramp Inspect, to supply the infrastructure for 10,000 agents running in parallel. The system organized their work as an agentic MapReduce. In the map phase, agents investigated Ramp for vulnerabilities and filed tickets for suspected problems. A ticket was a candidate finding. Several agents could discover the same issue, and an agent could offer a plausible account of an exploit that failed when another investigator tried it.

In the reduce phase, LLMs deduplicated the tickets, then tried adversarially to reproduce the suspected vulnerabilities. This gave Ramp a way to filter the search results before spending people’s time on them. The exercise found seven high-severity vulnerabilities, all of which Ramp patched. One was a Lua script injection deliverable by email. In Shevchenko’s account, each component along the path was safe in isolation. Discovering the injection required tracing how the components behaved together from receipt of the email through to script execution.

![A funnel narrows from about 10,000 sessions to seven high-severity findings, all patched. A code example shows how an email closed a Lua string early and ran code in the renderer.](https://ai.engineer/api/talk-essay-media/55c1b45c81ce5a1a6e9ba1718ac955ef18409764dab1bbd5a3802c66b3d658fb)

Pen-testing Ramp with ~10,000 agents

Automated filtering also changed the economics of model selection. A failed investigation cost tokens, but a person could be spared the work of pursuing it. Ramp tried open-weight models that were up to five times cheaper than frontier reasoning models. Their detection rate fell by a smaller factor than their price. Ramp could therefore buy more attempts with the cheaper tokens and bring the swarm’s detection rate close to the frontier result. Shevchenko gave no exact detection percentages.

### Token Spend Management accounts for usage and Ramp Router changes its cost

[10:07](https://www.youtube.com/watch?v=T-42NrYjwxM&t=607s)

Shevchenko connected the experiments to two products. Token Spend Management connects to a company’s API keys and attributes spending by key or user. For finance teams, it also supports accounting categories and analysis of spending that may qualify for tax credits. For someone deciding where to improve an agent, attribution answers only the first question. Knowing who incurred a charge still leaves the question of what work those tokens bought. Ramp analyzes usage in context and builds a taxonomy of activities that can be reconciled with the bill.

Ramp Router applies research to the execution side. Ramp Labs helped develop routing strategies aimed at improving agent latency and cost per token. Shevchenko named NVIDIA Switch Yard and Smart Flex offloading among the examples. He described multiple routing strategies as available, without giving a selection rule or a benchmark for an individual strategy.

![The left chart compares costs of $2.99, $1.11 and $0.72 per million tokens and highlights a 4.15× gap. The Ramp Router chart shows stacked costs falling over time, with a “40% lower” callout.](https://ai.engineer/api/talk-essay-media/26a05cd8fbf42aa04958203c2399e9c63e9215f4a727d2ad8bea457eb791f0d1)

Most teams overpay for intelligence

### Transfer a post-train, revise a monitor and discover usage categories

[12:02](https://www.youtube.com/watch?v=T-42NrYjwxM&t=722s)

A project called Portal addresses a cost that the Qwen result leaves open. What happens to the investment in a specialist when a newer base model arrives? Shevchenko described doing a LoRA post-train on a model such as Qwen 3.5 and learning a latent representation that can be projected onto another model. Ramp explored transfers to later Qwen versions and across model families. Reusing what the first post-train learned lets the team spread its training cost across models.

In a separate project on self-improving monitoring, Ramp used LLMs to create Datadog monitors for product surfaces. Incoming alerts then became input to another step in which the system revised the monitors automatically. For semantic tagging, the input was a collection of user sessions. Agents annotated what happened in each session, and an agentic MapReduce assembled those observations into an open-ended taxonomy. Ramp could discover clusters of LLM use without first asking someone to enumerate every activity in the company. Those resulting categories fed back into Token Spend Management, where finance teams could match the work people actually did to what it cost.

## Key takeaways

- A planner’s recurring questions make a bounded target for post-training. Measure the specialist’s answer accuracy and response latency separately from completion of the full assignment.
- For a scheduled workflow such as month-end close, identify the documents the agent will need and prepare their context before the user starts.
- When comparing models for a parallel search, include the number of attempts a budget buys and the cost of deduplicating and reproducing findings.
- To explain a token bill, classify the activities in user sessions and reconcile those categories with spend by key or user.

## Slides

### We’re a bitter-lesson-pilled company

[0:33](https://www.youtube.com/watch?v=T-42NrYjwxM&t=33s) [From October 2025 to October 2026, the active use cases chart rises 123%. Three smaller charts trend downward, with declines of 78% for approval intelligence, 90% for underwriting and 81% for vendor mailbox.](https://assets.ai.engineer/sites/aiecode2025/sha256/550e8d0963b2fbfbdacf2c4a67fa983befd5e90fb3fbeeb992eb94f34d42ea10.webp)

We’re a bitter-lesson-pilled company

Active use cases    +123%
Oct 2025    Jan 2026    Apr 2026    Jul 2026    Oct 2026

Approval intelligence    −78%
Oct 2025    Feb 2026    May 2026    Aug 2026    Oct 2026

Underwriting    −90%
Oct 2025    Feb 2026    May 2026    Aug 2026    Oct 2026

Vendor mailbox    −81%
Oct 2025    Feb 2026    May 2026    Aug 2026    Oct 2026

### We didn’t wait in 2023 either

[1:32](https://www.youtube.com/watch?v=T-42NrYjwxM&t=92s) [The 2023 Ramp Tour Guide points to the Company profile tab and tells the user where to change card branding. The 2026 diagram traces a winding path from an open circle labeled Task through a series of dots to a yellow square labeled Result.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fb7762513fb8acf9a9f7d595a4eeeabc061d57daf62de8ab14a261b387abce8.webp)

We didn’t wait in 2023 either

2023
Setup guide
Next: Get a quick tour of Ramp
Search    ⌘ K
Inbox    133
Insights
My Ramp
Expenses
Cards
Spend programs
Procurement
Bill Pay

Settings
Company settings
Company profile    Financial accounts    Payments    Security
Ramp Tour Guide
Click on the “Company profile” tab to access the section where you can change the card’s branding.
Business website
ramp.com ↗
Business phone number
Business address
123 Main St
New York City, NY 10003

Ramp Tour Guide, built on guardrails, prompts and context

2026
Task
Result
Agents take the wheel, and the work moves down the stack

### Three ways we put applied research to work

[3:04](https://www.youtube.com/watch?v=T-42NrYjwxM&t=184s) [Three panels illustrate the methods. Nested squares narrow to a yellow center for post-training. A calendar connects a Wednesday build to a Friday session for KV cartridges. Several points converge on one circle for agentic map-reduce.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ca471d0bf85db7ae31b59a5823684d58a75444e1b4b096aa158215580999878.webp)

Three ways we put applied research to work

Into a specialist
Post-training

M    T    W    T    F    S    S
BUILT    SESSION
Precompute ahead of time
KV cartridges

Across thousands of agents
Agentic map-reduce

### A fast specialist for a smart orchestrator

[3:17](https://www.youtube.com/watch?v=T-42NrYjwxM&t=197s) [A Ramp Sheets workbook shows Anthropic and OpenAI projections beside an assistant’s completed checks. The slide assigns planning and editing to a frontier model and spreadsheet questions to the FastAsk specialist.](https://assets.ai.engineer/sites/aiecode2025/sha256/b0637abac2122034038a36fdec9ebe28cfee5befea41637789b09ca65e4ab4e5.webp)

A fast specialist for a smart orchestrator

Ramp Sheets
https://labs.ramp.com/sheets
Anthropic vs OpenAI Revenue & Valuation Analysis.xlsx
Saved    Download    Share
Home    Insert    Page Layout    Formulas    View    Settings
Paste    Calibri    11    Wrap Text    Number
Merge & Center    Conditional Format    Format Table    Cell Styles    Cell Editors
Undo    Clipboard    Font    Alignment    Number    Styles    Cells    Editing
C24    =500/F19

THE 2028 PROFITABILITY RACE

ANTHROPIC P&L PROJECTION
Year    Revenue ($B)    YoY Growth    Cash Burn ($B)    Burn % of Rev    Cumulative Burn ($B)
2024    4.2                2.9    69%    2.9
2025    9.0    114%    3.0    33%    5.9
2026    14.0    56%    2.5    18%    8.4
2027    20.0    43%    1.8    9%    10.2
2028    27.0    35%    0.0    0%    10.2

OPENAI P&L PROJECTION
Year    Revenue ($B)    YoY Growth    Cash Burn ($B)    Burn % of Rev    Cumulative Burn ($B)
2024    13                9    69%    9
2025    18    38%    10    56%    19
2026    29    61%    17    59%    36
2027    50    72%    29    58%    65
2028    85    70%    74    87%    139
2029    130    53%    50    38%    189
2030    180    38%    0    0%    189

EFFICIENCY METRICS
Metric    Anthropic    OpenAI    Winner
Revenue per $1 Burned    7.27    2.67    Anthropic
Valuation per $1 Burned    34.31    =500/F19    Anthropic
Cost to Buy 1yr Profitability Advantage    89.4

KEY INSIGHTS
Primary:    Market paying 40% premium for 14x less burn
Secondary:    Anthropic generates 2x more revenue per dollar burned

1. Revenue Multiple Paradox    2. Profitability Race    3. Funding & Returns    4. Summary Dashboard
Edit    100%

• Anthropic: $350B / $9B = 38.9x multiple
• OpenAI: $500B / $18B = 27.8x multiple
• Premium: 40% ✓
• Historical IPO multiples calculate correctly

Sheet 2 (Profitability Race) ✅
• Anthropic cumulative burn to 2028 breakeven: $10.2B
• OpenAI cumulative burn to 2030 breakeven: $189B
• Actual ratio: ~18.5x (even better than the 14x headline)
• Efficiency metrics calculate correctly

Sheet 3 (Funding & Returns) ✅
• MOIC calculations correct (350x for Series A, 5.7x for Series D, 1.9x for Series F)
• Dilution: $10B / $360B = 2.8% ✓
• Valuation per $ raised: $350B / $35B = 10x ✓

Sheet 4 (Summary Dashboard) ✅
• All cross-sheet references pulling correctly
• Cash to profit shows correct ~18x difference

All formulas are accurate and interlinked properly. The “14x less burn” in the title is slightly conservative - the actual model shows ~18x less burn, which makes Anthropic’s efficiency story even stronger.

5 of 5 tasks completed
What do you need help with?
Fast    Expert

Ramp Sheets, November 2025

Orchestrator
A frontier model plans, reasons and edits the sheet

FastAsk
A post-trained Qwen3.5-35B-A3B that answers questions about the sheet in a fraction of the time

### Opus-level accuracy at Haiku-level speed

[5:15](https://www.youtube.com/watch?v=T-42NrYjwxM&t=315s) [A scatterplot compares exact-task accuracy with time per task relative to Claude Haiku 4.5. FastAsk sits above the dashed frontier at 66.25% accuracy and just over 1.0× Haiku’s time. Opus reaches 61.88% at roughly 1.44×.](https://assets.ai.engineer/sites/aiecode2025/sha256/20b815f7d519193dd87cfae714d6b13cbc75691e520fc2b7866c6c949776bbbb.webp)

Opus-level accuracy at Haiku-level speed

Exact-task accuracy
65%    60%    55%    50%

FastAsk
66.25%
+14 pts
at Haiku-level speed

Claude Haiku 4.5 51.88%
Qwen3.5-35B-A3B, base 56.25%
Claude Opus 4.6 61.88%
Claude Sonnet 4.6 59.38%
Frontier before FastAsk

1.0×    1.2×    1.4×    1.6×
Time per task, relative to Claude Haiku 4.5 →

### KV cartridges hand the agent the whole month

[5:59](https://www.youtube.com/watch?v=T-42NrYjwxM&t=359s) [Two timelines compare reconciliation workflows. In the first, a long reading stage precedes reconciliation. In the second, a narrow yellow mark represents loading a cartridge built overnight, and the reconciliation stage starts almost immediately.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1d741ea08504ad123d626ea6e4ff5143189ea01f9df369b9abb5c7d5419c25c.webp)

KV cartridges hand the agent the whole month

Today
Reading the ledger, aging report, open invoices    Reconcile

With a cartridge
Reconcile
Cartridge built overnight, loaded instantly

Lower lift than post-training:
no verifiers, no rubrics.

Labs may ship this as an API one day.
We’re learning how it works now.

### Pen-testing Ramp with ~10,000 agents

[8:01](https://www.youtube.com/watch?v=T-42NrYjwxM&t=481s) [A funnel narrows from about 10,000 sessions to seven high-severity findings, all patched. A code example shows how an email closed a Lua string early and ran code in the renderer.](https://assets.ai.engineer/sites/aiecode2025/sha256/55c1b45c81ce5a1a6e9ba1718ac955ef18409764dab1bbd5a3802c66b3d658fb.webp)

Pen-testing Ramp with ~10,000 agents

~10,000 sessions
No finding, dropped

~6,000 tickets
Duplicates merged

~3,000 tickets
Reproduced adversarially

7 High, all patched

One email ran code in our renderer
html = [===[ Hi team, ]===] os.execute(...)
Simplified

The email closed a Lua string early. No single component
looked unsafe; the agent traced the flow and proved the exploit
in its sandbox before filing.

Failed passes cost only tokens.

### Cheap models still find real bugs

[9:28](https://www.youtube.com/watch?v=T-42NrYjwxM&t=568s) [Horizontal bars compare detection rates on seven known high-severity findings. GPT-5.5 reaches 52.0%, Kimi K2.6 reaches 36.6%, and DeepSeek V4 Pro reaches 20.6%. A bracket marks the latter two as about five times cheaper per token.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ccb46d1f32521b9d4f043d08e75bf9fd9869dca63d030ce71a47171446ad40f.webp)

Cheap models still find real bugs

Detection rate on the 7 known High findings, 25 attempts each

GPT-5.5
high reasoning
52.0%

Kimi K2.6
open weights · via Fireworks
36.6%

DeepSeek V4 Pro
open weights · via Fireworks
20.6%

0%    20%    40%    60%

~5× cheaper
per token

### Most teams overpay for intelligence

[10:05](https://www.youtube.com/watch?v=T-42NrYjwxM&t=605s) [The left chart compares costs of $2.99, $1.11 and $0.72 per million tokens and highlights a 4.15× gap. The Ramp Router chart shows stacked costs falling over time, with a “40% lower” callout.](https://assets.ai.engineer/sites/aiecode2025/sha256/26a05cd8fbf42aa04958203c2399e9c63e9215f4a727d2ad8bea457eb791f0d1.webp)

Most teams overpay for intelligence

The price of inefficient AI
Same work. Unoptimized teams pay several times more per token.

4.15× higher cost per 1M tokens for
unoptimized teams versus
efficient ones.
OFF    ON

$3.00    $2.00    $1.00    $0.00
$2.99    $1.11    $0.72
Cost per 1M tokens

Token Spend Management   See where every token dollar goes

Cost impact over time   40% lower
$200    $150    $100    $50    $0
1/1    1/9    1/17    1/25    2/2
40% Default    60% Flex

Ramp Router   router.com

## Full transcript

[0:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=12s) Hey, everyone. I'm Alex from Ramp. Um, I'll be speaking about chasing capabilities while waiting for the frontier to advance. So quick, uh, intro about myself. I joined Ramp around three years ago, uh, on the broader applied AI team, working on a bunch of different surfaces and products, applying LLMs and ML to make a great product even better. Uh, to-- and to, to caveat my presentation on Ramp Labs, Ramp is an extremely bitter lesson-built

[0:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=42s) company. Over the last year, the number of use cases that use LLMs has increased a hundred and twenty-three percent, while at the same time, a lot of our top features are growing in token spend, but on a unit economic basis, they're actually becoming cheaper because there are frontier models that are being released, uh, all the time now, and these models are cheaper, faster, uh, and better. And so we're able to provide the same level of accuracy at a much lower cost. And so this is

[1:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=72s) a really bitter lesson-pilled, uh, pattern. And so given all this, a question that I sometimes get: Why not just wait? Why have an applied research function inside of a company like Ramp, an app- uh, application layer company, right? Um, and to answer that, I would like to dry- draw a parallel to twenty twenty-three. Back in twenty twenty-three, uh, you had access to kind of like GPT-4, and creating an agent was a lot of work.

[1:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=102s) Um, you would have to create very, very strict guardrails. Models weren't RL'd to backtrack. They didn't have thinking tokens, and so if it went in the wrong direction, it would just kinda charge ahead and not realize that it made mistakes. You had to sweat over all of the prompts where-- to the point where, um, some synonyms could change the behavior of, um, of your agent because the-- they weren't RL'd properly on, uh, generalizing to the user's intent. Uh, you had

[2:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=132s) to reduce and scope down the context as much as possible because giving too much context to the agent would just confuse it. Um, and even back then, we decided, well, we need to ship these things. We can't wait for GPT-4.5 or GPT-5 to come out. We need to serve our financial professionals now. And we shipped, uh, things like Ramp Tour Guide and many other agents, even though it was not the easy thing to do. And fast-forward now to twenty twenty-six, building an agent is a

[2:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=162s) lot easier. The models have gotten so much better that you can kinda give them MCPs and co-- uh, access to kinda any context, and they do a very, very good job of understanding what they need to do given a task. But I still believe that we can dig a layer down and create even better experiences for our users. It's just it requires a bit more effort now. So to present this, uh, some of our work from Ramp Labs, three specific cases of

[3:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=192s) how we've applied research, uh, into Ramp itself, starting with creating a specialist, uh, model. So in November of last year, we launched Ramp Sheets, which was one of the really early, uh, spreadsheet agents. Uh, this was before the two frontier labs had launched their versions of, uh, spreadsheet agents, and we had a, a very good launch in November. A bunch of, uh, users from great companies had signed up and started using it. And we realized that, well, this is a

[3:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=222s) very general purpose, uh, spreadsheet handling agent. And a lot of the work that people do on Ramp, uh, both in-inside of the company, but also all of our customers, accounting and finance professionals, are spending most of their days inside of spreadsheets. So we can kinda take the spread-- uh, the, the harness and the brains behind Ramp Sheets and add it into a bunch of other, um, surfaces within Ramp. And as we were building this out, we realized, well, we can probably

[4:13](https://www.youtube.com/watch?v=T-42NrYjwxM&t=253s) apply some research, uh, from Ramp Labs to make this even better. And we sat down, and we started looking, where do models fall short on this? And back then, this was, like, Opus four point six, and the place that had the most friction was when you would drop in a lot of spreadsheets into Ramp Sheets and you would ask it to build a DCF or, uh, or model some financial scenario. It would spend a lot of time creating a plan at first and then researching all of the sheets that it had access to, to

[4:43](https://www.youtube.com/watch?v=T-42NrYjwxM&t=283s) try to get a good understanding of and the lay of the land. And Opus four point six is a great model, but it was, like, pretty slow, and the thinking tokens, uh, are also pretty expensive. And so we, we took a step back and, and decided to train a retrieval model or kinda like a financial question-answering model that would, while still keeping these frontier models or orc- as orchestrators, would then be handed off to the, this much faster model that would do, uh, financial question

[5:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=312s) answering for that, uh, planning agent. And so we partnered with the company Prime Intellect to, uh, post-train a Qwen three point five, uh, thirty-five billion, um, model. And we actually got above Opus four point six, which was the frontier back then, uh, exact task accuracy, while also being able to serve it at Haiku, uh, level latencies. So a much faster, but just a smart model that could then be used in,

[5:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=342s) uh, in tandem with the really smart frontier orchestrator. Um, and so th-this was, like, a very successful, uh, example of post-training that we had actually sent to prod, uh, as part of our Ramp Labs effort. Uh, the second, uh, big thing that we are actually exploring up to this day is, uh, KV Cache Tricks. And particularly, cartridges are a really interesting, um, family of techniques where you're

[6:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=372s) able to compress a lot of context, up to twenty to thirty times more context, into the KV cache offline and serve it for your agent. And so if you compare it to the Ramp Sheets example, for example, um, there's a lot of tasks that you know are gonna happen at some point, right? Like month-end close, a month finishes, and on the first of the next month, a financial professional will go and try to reconcile and close their books. You know kinda the ledger, the aging report, and the

[6:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=402s) documents that they will need to be retrieving. And so you could prepare them as context, but the agent will still need to agentically retrieve and, and s- create a good picture and understanding of them. With cartridges, we're able to compress all of that work and build one overnight before the, the financial professional starts their work on the first of the month. And once they open up the, the, the agent, it knows exactly what it needs to do because it has all of the, the context already, uh, built in.

[7:12](https://www.youtube.com/watch?v=T-42NrYjwxM&t=432s) Cartridges are also a much lower lift than post-training because the way that they do self-play is kinda a lot more generalizable. The caveat with stuff like this is that compared to post-training, this is something that the labs might ship at some point this, uh, in their APIs, in their foray into, uh, continuous learning. And even if they were to do it in the API, we would actually be happy because we are developing a lot of this tribal knowledge inside of the company, um, and just learning how best to apply these techniques.

[7:42](https://www.youtube.com/watch?v=T-42NrYjwxM&t=462s) So even if it does ship as an API, we can just hit the ground running on day one knowing how to use them.

[7:50](https://www.youtube.com/watch?v=T-42NrYjwxM&t=470s) And finally, the third example, uh, of applied research that we are sending to prod is our work on, uh, agentic MapReduce and kinda agentic swarms. So more concretely, we published an article on the Ramp Labs, uh, blog on how we, uh, used ten thousand parallel agents to pen test all of Ramp. Um, so at Ramp, we have a, a cloud agent called Ramp Inspect, which helps out with setting up all of the

[8:20](https://www.youtube.com/watch?v=T-42NrYjwxM&t=500s) infrastructure and plumbing for something as ambitious as this. And we were able to spin up ten thousand parallel agents and then kinda map th- this work out to try to find any vulnerabilities, and then also agentically reduce on the tickets that they would create. So deduplicate them with LLMs, and then try to reproduce them adversarially with LLMs to try to make sure that the vulnerabilities are real and important. At the end of all of this, we have actually found seven high vulnera- highest

[8:50](https://www.youtube.com/watch?v=T-42NrYjwxM&t=530s) severity vulnerabilities that we all patched. Interestingly, one of the, um, vulnerabilities that we describe in that article is a Lua script injection that you could send over via email, and no single component of that stack was unsafe, and the kinda swarm had to patch together multiple components and understanding of multiple components end-to-end in order to be able to figure this out. And the other nice thing with this is that this is all running automa- automatically. You don't

[9:20](https://www.youtube.com/watch?v=T-42NrYjwxM&t=560s) need to spend anyone's time. It's just you start it, and the only thing that, uh, failed, failed passes that get filtered out is the cost of the tokens.

[9:30](https://www.youtube.com/watch?v=T-42NrYjwxM&t=570s) The other thing that's interesting outside of just building out all of the infrastructure to support these swarms is that we do research into how to best use models as part of these swarms. One of the interesting insights from that same article that we published is that you can actually use open weight models that are up to five times cheaper compared to the frontier reasoning. And because you're se- setting it up as a swarm, the detection rate is not five times lower. So you can kinda spend more

[9:59](https://www.youtube.com/watch?v=T-42NrYjwxM&t=599s) on these cheaper tokens and have almost as good of a detection rate.

[10:07](https://www.youtube.com/watch?v=T-42NrYjwxM&t=607s) Um, if you follow through all of these examples, you kinda notice a pattern of a lot of them is attempts at saving on time and saving on money. So reducing latency, which is time, and then reducing on token cost, which is money. That's actually Ramp's, uh, slogan, uh, "Saving our customers time and money." Um, and so I'd like to, to point to two, uh, products that we actually offer from Ramp that have been, um, influenced by Ramp Labs' research. The first one is Token

[10:37](https://www.youtube.com/watch?v=T-42NrYjwxM&t=637s) Spend Management that connects to a bunch of, uh, your API keys and is able to give you a really good picture of how LLMs are being used inside of your company. Not only token spend on an API key basis or a user basis, but digging deeper, accounting categories, um, and even things like, um, tax credits that you can get by, uh, analyzing the spend. It also goes deeper, and we create a taxonomy that, uh,

[11:07](https://www.youtube.com/watch?v=T-42NrYjwxM&t=667s) contextually analyzes and semantically analyzes the LLM usage so you can actually understand how LLMs are being used outside of just attributing it to a team or a person inside of your company. And then the second one is Ramp Router, which also leans into improving your l- the latency of your agent, where we have a bunch of routing strategies that we work with Ramp Labs to develop, uh, things like NVIDIA Switch Yard, things like Smart Flex offloading of work to both improve latency

[11:37](https://www.youtube.com/watch?v=T-42NrYjwxM&t=697s) and, um, cost per tokens that you spend. And there's a bunch of different strategies already available for the, the, for routing and even more coming out later. So if you're interested in any of these two products, you can take a photo, uh, of the URLs or, or scan the QR codes there. Give you a second.

[12:02](https://www.youtube.com/watch?v=T-42NrYjwxM&t=722s) Awesome. And then I, I'd also invite you to check out our Ramp Labs, uh, blog where we publish, um, a bunch of really interesting articles that I haven't had the chance to cover as part of this presentation. Um, notably one of the really interesting ones is Portal, where we figured out a way of, uh, transferring learnings from one post-train to another. So you could do something like a LoRA, uh, post-train on one model, so like the Qwen 3.5 that I was describing, but then

[12:32](https://www.youtube.com/watch?v=T-42NrYjwxM&t=752s) transfer it to newer versions of Qwen or even across different families. So you're post-training this latent that then can be projected onto different, uh, models, and that really amortizes the cost that you have to spend on post-training. Uh, more blog posts on agentic MapReduce, so not only the pen testing stuff that we did, but also things like, um, analyzing and creating taxonomies of LLM usage, which is st- stuff that has gone into the token spend product. Um, things like self-improving

[13:02](https://www.youtube.com/watch?v=T-42NrYjwxM&t=782s) monitoring, where we use, uh, LLMs to automatically, um, create monitors with Datadog on a lot of our product surfaces and then as alerts come in, improve those monitors automatically without a person in, in the loop to try to basically create super, super robust and resilient software. Um, and finally, semantic tagging, which is this interesting, uh, foray into creating this, um, agentic MapReduce over a bunch of user sessions where we can annotate and create

[13:32](https://www.youtube.com/watch?v=T-42NrYjwxM&t=812s) taxonomies that are open-ended, so not knowing, um, ahead of time which, what are the clusters of LLM usage and trying to create a taxonomy that can then be reconciled against the spend inside of the companies. So if, uh, if you're interested in this, uh, type of research, axe.com/ramplabs where you can, uh, go and, and check out more of our research.

[13:56](https://www.youtube.com/watch?v=T-42NrYjwxM&t=836s) And, uh, here's my Twitter if you have any other questions or, uh, want to talk about some use case as part of your applied research. And that's it. Thank you.
