1
00:00:04,126 --> 00:00:04,966
Hey, everyone.

2
00:00:05,206 --> 00:00:06,866
I'm Alex from Ramp.

3
00:00:07,786 --> 00:00:10,006
I'll be speaking about chasing capabilities

4
00:00:10,006 --> 00:00:11,826
while waiting for the frontier to advance.

5
00:00:12,186 --> 00:00:14,286
So a quick intro about myself.

6
00:00:14,506 --> 00:00:16,626
I joined Ramp around three years ago

7
00:00:16,626 --> 00:00:18,726
on the broader Applied AI team,

8
00:00:18,866 --> 00:00:21,246
working on a bunch of different surfaces and products,

9
00:00:21,506 --> 00:00:25,126
applying LLMs and ML to make a great product even better.

10
00:00:28,286 --> 00:00:34,506
and to caveat my presentation on ramp labs ramp is an extremely bitter lesson build company

11
00:00:34,506 --> 00:00:39,786
over the last year the number of use cases that use lms has increased 123 percent

12
00:00:39,786 --> 00:00:47,746
while at the same time a lot of our top features are growing in token spend but on a unit economic

13
00:00:47,746 --> 00:00:52,186
basis they're actually becoming cheaper because there are frontier models that are being released

14
00:00:53,466 --> 00:00:57,646
all the time now, and these models are cheaper, faster, and better.

15
00:00:57,826 --> 00:01:01,306
And so we're able to provide the same level of accuracy

16
00:01:01,306 --> 00:01:02,886
at a much lower cost.

17
00:01:03,046 --> 00:01:06,986
And so this is a really bitter lesson-pilled pattern.

18
00:01:07,366 --> 00:01:12,466
And so given all this, a question that I sometimes get,

19
00:01:12,706 --> 00:01:13,826
why not just wait?

20
00:01:13,926 --> 00:01:17,906
Why have an applied research function inside of a company like RAMP,

21
00:01:18,086 --> 00:01:20,586
an application layer company, right?

22
00:01:21,246 --> 00:01:25,846
And to answer that, I would like to draw a parallel to 2023.

23
00:01:26,886 --> 00:01:30,946
Back in 2023, you had access to kind of like GPT-4,

24
00:01:31,246 --> 00:01:34,146
and creating an agent was a lot of work.

25
00:01:35,226 --> 00:01:38,586
You would have to create very, very strict guardrails.

26
00:01:38,966 --> 00:01:40,806
Models weren't RL'd to backtrack.

27
00:01:40,966 --> 00:01:42,186
They didn't have thinking tokens.

28
00:01:42,446 --> 00:01:44,846
And so if it went in the wrong direction,

29
00:01:44,846 --> 00:01:46,346
it would just kind of charge ahead

30
00:01:46,346 --> 00:01:49,046
and not realize that it made mistakes.

31
00:01:49,046 --> 00:01:58,306
You had to sweat over all of the prompts to the point where some synonyms could change the behavior of your agent

32
00:01:58,306 --> 00:02:03,486
because they weren't RL'd properly on generalizing to the user's intent.

33
00:02:03,866 --> 00:02:07,466
You had to reduce and scope down the context as much as possible

34
00:02:07,466 --> 00:02:11,926
because giving too much context to the agent would just confuse it.

35
00:02:13,066 --> 00:02:17,386
And even back then, we decided, well, we need to ship these things.

36
00:02:17,386 --> 00:02:21,246
We can't wait for GPT 4.5 or GPT 5 to come out.

37
00:02:21,486 --> 00:02:24,326
We need to serve our financial professionals now.

38
00:02:24,526 --> 00:02:28,646
And we should, things like RAM Tour Guide and many other agents,

39
00:02:28,966 --> 00:02:30,646
even though it was not the easy thing to do.

40
00:02:31,206 --> 00:02:35,166
And fast forward now to 2026, building an agent is a lot easier.

41
00:02:35,506 --> 00:02:38,826
The models have gotten so much better that you can kind of give them MCPs

42
00:02:38,826 --> 00:02:41,106
and access to kind of any context.

43
00:02:41,106 --> 00:02:43,786
And they do a very, very good job of understanding

44
00:02:43,786 --> 00:02:45,426
what they need to do given a task.

45
00:02:45,426 --> 00:02:53,666
But I still believe that we can dig a layer down and create even better experiences for our users.

46
00:02:54,306 --> 00:02:56,466
just it requires a bit more effort now.

47
00:02:57,906 --> 00:03:07,786
So to present this, some of our work from Ramp Labs, three specific cases of how we've applied research into Ramp itself,

48
00:03:08,126 --> 00:03:11,886
starting with creating a specialist model.

49
00:03:12,546 --> 00:03:15,306
So in November of last year, we launched Ramp Sheets,

50
00:03:15,406 --> 00:03:19,826
which was one of the really early spreadsheet agents.

51
00:03:20,066 --> 00:03:24,606
This was before the two frontier labs had launched their versions of spreadsheet agents.

52
00:03:24,606 --> 00:03:27,046
And we had a very good launch in November.

53
00:03:27,466 --> 00:03:31,426
A bunch of users from great companies had signed up and started using it.

54
00:03:31,546 --> 00:03:37,906
And we realized that, well, this is a very general purpose spreadsheet handling agent.

55
00:03:37,906 --> 00:03:40,786
and a lot of the work that people do on RAMP,

56
00:03:40,946 --> 00:03:43,146
both inside of the company,

57
00:03:43,266 --> 00:03:46,406
but also all of our customers, accounting, and finance professionals,

58
00:03:46,966 --> 00:03:49,346
are spending most of their days inside of spreadsheets.

59
00:03:49,466 --> 00:03:54,366
So we can kind of take the harness and the brains behind RAMP sheets

60
00:03:54,366 --> 00:03:59,906
and add it into a bunch of other surfaces within RAMP.

61
00:04:00,066 --> 00:04:02,026
And as we were building this out,

62
00:04:02,206 --> 00:04:07,766
we realized, well, we can probably apply some research from RAMP Labs

63
00:04:07,766 --> 00:04:09,106
to make this even better.

64
00:04:10,106 --> 00:04:12,306
And we sat down and we started looking,

65
00:04:12,326 --> 00:04:14,626
where do models fall short on this?

66
00:04:14,786 --> 00:04:17,386
And back then, this was like Opus 4.6.

67
00:04:17,566 --> 00:04:20,826
And the place that had the most friction

68
00:04:20,826 --> 00:04:23,386
was when you would drop in a lot of spreadsheets

69
00:04:23,386 --> 00:04:26,706
into RAM sheets and you would ask it to build a DCF

70
00:04:26,706 --> 00:04:28,866
or model some financial scenario.

71
00:04:29,086 --> 00:04:31,566
It would spend a lot of time creating a plan at first

72
00:04:31,566 --> 00:04:33,226
and then researching all of the sheets

73
00:04:33,226 --> 00:04:34,326
that it had access to,

74
00:04:34,466 --> 00:04:37,186
to try to get a good understanding

75
00:04:37,186 --> 00:04:38,766
and the lay of the land.

76
00:04:39,606 --> 00:04:41,706
And Opus 4.6 is a great model,

77
00:04:41,806 --> 00:04:43,186
but it was pretty slow,

78
00:04:43,286 --> 00:04:46,126
and the thinking tokens are also pretty expensive.

79
00:04:46,406 --> 00:04:48,806
And so we took a step back

80
00:04:48,806 --> 00:04:51,566
and decided to train a retrieval model,

81
00:04:51,946 --> 00:04:54,006
or kind of like a financial question-answering model

82
00:04:54,006 --> 00:04:56,886
that would, while still keeping these frontier models

83
00:04:56,886 --> 00:04:58,186
as orchestrators,

84
00:04:58,746 --> 00:05:02,306
would then be handed off to this much faster model

85
00:05:02,306 --> 00:05:04,806
that would do financial question-answering

86
00:05:04,806 --> 00:05:07,066
for that planning agent.

87
00:05:07,806 --> 00:05:16,806
And so we partnered with the company Prime Intellect to post-train a Qwen 3.5, 35 billion model.

88
00:05:17,026 --> 00:05:24,606
And we actually got above Opus 4.6, which was the frontier back then, exact task accuracy,

89
00:05:24,846 --> 00:05:28,806
while also being able to serve it at Haiku level latencies.

90
00:05:28,806 --> 00:05:38,266
So a much faster but just a smart model that could then be used in tandem with the really smart frontier orchestrator.

91
00:05:39,846 --> 00:05:48,586
And so this was a very successful example of post-training that we had actually sent to prod as part of our Ramp Labs effort.

92
00:05:49,786 --> 00:05:56,526
The second big thing that we are actually exploring up to this day is KVCache

93
00:05:56,826 --> 00:06:04,366
tricks and particularly cartridges are a really interesting family of techniques where you're able

94
00:06:04,366 --> 00:06:12,186
to compress a lot of context up to 20 to 30 times more context into the kv cache offline and serve it

95
00:06:12,186 --> 00:06:16,706
for your agent and so if you compare it to the ramp sheets example for example

96
00:06:18,346 --> 00:06:23,346
there's a lot of tasks that you know are going to happen at some point right like month end close

97
00:06:23,346 --> 00:06:28,706
a month finishes and on the first of the next month a financial professional will go and try

98
00:06:28,706 --> 00:06:34,286
to reconcile and close their books you know kind of the ledger the aging report and the documents

99
00:06:34,286 --> 00:06:38,986
that they will need to be retrieving and so you could prepare them as context but the agent will

100
00:06:38,986 --> 00:06:45,086
still need to agentically retrieve and create a good picture and understanding of them with

101
00:06:45,086 --> 00:06:51,066
cartridges we're able to compress all of that work and build one overnight before the the

102
00:06:51,066 --> 00:06:56,466
financial professional starts their work on the first of the month and once they

103
00:06:56,466 --> 00:07:01,046
open up the the agent it knows exactly what it needs to do because it has all

104
00:07:01,046 --> 00:07:05,946
of the context already built in cartridges are also a much lower lift

105
00:07:05,946 --> 00:07:09,986
in post training because the way that they do self play is kind of a lot more

106
00:07:09,986 --> 00:07:14,926
generalizable the caveat with stuff like this is that compared to post training

107
00:07:14,926 --> 00:07:19,746
this is something that the labs might ship at some point in their API's in

108
00:07:19,746 --> 00:07:22,466
their foray into continuous learning.

109
00:07:24,066 --> 00:07:26,166
Even if they were to do it in the API,

110
00:07:26,526 --> 00:07:27,986
we would actually be happy because we

111
00:07:27,986 --> 00:07:29,766
are developing a lot of this tribal knowledge

112
00:07:29,766 --> 00:07:32,926
inside of the company and just learning how best

113
00:07:32,926 --> 00:07:33,986
to apply these techniques.

114
00:07:34,226 --> 00:07:35,646
So even if it does ship as an API,

115
00:07:36,066 --> 00:07:38,266
we can just hit the ground running on day one,

116
00:07:38,466 --> 00:07:39,586
knowing how to use them.

117
00:07:42,546 --> 00:07:47,086
And finally, the third example of applied research

118
00:07:47,086 --> 00:07:51,406
that we are sending to prod is our work on agentic map reduce

119
00:07:51,406 --> 00:07:52,866
and kind of agentic swarms.

120
00:07:54,006 --> 00:07:58,586
So more concretely, we published an article on the Ramp Labs blog

121
00:07:58,586 --> 00:08:03,826
on how we used 10,000 parallel agents to pentest all of Ramp.

122
00:08:05,386 --> 00:08:09,846
So at Ramp, we have a cloud agent called Ramp Inspect,

123
00:08:10,166 --> 00:08:13,266
which helps out with setting up all of the infrastructure and plumbing

124
00:08:13,266 --> 00:08:15,246
for something as ambitious as this.

125
00:08:15,246 --> 00:08:19,326
and we were able to spin up 10,000 parallel agents

126
00:08:19,326 --> 00:08:22,266
and then kind of map this work out,

127
00:08:22,446 --> 00:08:23,626
try to find any vulnerabilities,

128
00:08:24,126 --> 00:08:28,486
and then also agentically reduce on the tickets that they would create.

129
00:08:28,686 --> 00:08:30,626
So deduplicate them with LLMs

130
00:08:30,626 --> 00:08:33,546
and then try to reproduce them adversarially with LLMs

131
00:08:33,546 --> 00:08:37,126
to try to make sure that the vulnerabilities are real and important.

132
00:08:37,526 --> 00:08:38,946
At the end of all of this,

133
00:08:39,166 --> 00:08:42,966
we have actually found seven highest severity vulnerabilities

134
00:08:42,966 --> 00:08:44,266
that we all patched.

135
00:08:44,266 --> 00:08:49,986
Interestingly, one of the vulnerabilities that we described in that article is a Lua

136
00:08:49,986 --> 00:08:52,446
script injection that you could send over via email.

137
00:08:53,186 --> 00:09:00,066
And no single component of that stack was unsafe, and the kind of swarm had to patch

138
00:09:00,066 --> 00:09:05,286
together multiple components and understanding of multiple components end-to-end in order

139
00:09:05,286 --> 00:09:06,706
to be able to figure this out.

140
00:09:07,226 --> 00:09:11,406
And the other nice thing with this is that this is all running automatically.

141
00:09:11,826 --> 00:09:13,466
You don't need to spend anyone's time.

142
00:09:13,466 --> 00:09:19,586
is just use started and the only thing that failed passes that get filtered out is the cost of the

143
00:09:19,586 --> 00:09:25,706
tokens. The other thing that's interesting outside of just building out all of the infrastructure to

144
00:09:25,706 --> 00:09:33,366
support these swarms is that we do research into how to best use models as part of these swarms.

145
00:09:33,606 --> 00:09:34,666
One of the interesting

146
00:09:35,026 --> 00:09:44,366
insights from that same article that we published is that you can actually use open weight models that are up to five times cheaper compared to the frontier reasoning.

147
00:09:44,806 --> 00:09:49,806
Because you're setting it up as a swarm, the detection rate is not five times lower.

148
00:09:50,006 --> 00:09:56,126
So, you can kind of spend more on these cheaper tokens and have almost as good of a detection rate.

149
00:10:00,466 --> 00:10:10,746
If you follow through all of these examples, you kind of notice a pattern of a lot of them is attempts at saving on time and saving on money.

150
00:10:10,986 --> 00:10:15,166
So reducing latency, which is time, and then reducing on token cost, which is money.

151
00:10:15,446 --> 00:10:19,286
That's actually RAMP's slogan, saving our customers time and money.

152
00:10:19,986 --> 00:10:28,246
And so I'd like to point to two products that we actually offer from RAMP that have been influenced by RAMP Labs' research.

153
00:10:28,246 --> 00:10:33,806
The first one is token spend management that connects to a bunch of your API keys

154
00:10:33,806 --> 00:10:38,566
and is able to give you a really good picture of how LMs are being used inside of your company.

155
00:10:38,766 --> 00:10:43,846
Not only token spend on an API key basis or a user basis, but digging deeper.

156
00:10:44,106 --> 00:10:54,626
Accounting categories and even things like tax credits that you can get by analyzing the spend.

157
00:10:54,626 --> 00:11:02,746
It also goes deeper, and we create a taxonomy that contextually analyzes and semantically analyzes the LLM usage,

158
00:11:02,746 --> 00:11:09,926
so you can actually understand how LLMs are being used outside of just attributing it to a team or a person inside of your company.

159
00:11:10,366 --> 00:11:16,786
And then the second one is Ramp Router, which also leans into improving the latency of your agent,

160
00:11:16,986 --> 00:11:21,466
where we have a bunch of routing strategies that we work with Ramp Labs to develop,

161
00:11:21,466 --> 00:11:28,826
things like NVIDIA Switchyard, things like smart flex offloading of work to both improve latency

162
00:11:28,826 --> 00:11:34,486
and cost per tokens that you spend. And there's a bunch of different strategies already available

163
00:11:34,486 --> 00:11:41,526
for routing and even more coming out later. So if you're interested in any of these two products,

164
00:11:41,606 --> 00:11:47,266
you can take a photo of the URLs or scan the QR codes there. I'll give you a second.

165
00:11:54,226 --> 00:11:54,826
Awesome.

166
00:11:55,506 --> 00:12:02,206
And then I'd also invite you to check out our RAMP Labs blog where we publish a bunch

167
00:12:02,206 --> 00:12:06,906
of really interesting articles that I haven't had the chance to cover as part of this presentation.

168
00:12:07,986 --> 00:12:13,686
Notably one of the really interesting ones is PorTAL where we figured out a way of transferring

169
00:12:13,686 --> 00:12:16,126
learnings from one post train to another.

170
00:12:16,126 --> 00:12:21,366
So you could do something like a LoRA post-train on one model,

171
00:12:21,746 --> 00:12:23,726
so like the Qwen 3.5 that I was describing,

172
00:12:23,966 --> 00:12:26,466
but then transfer it to newer versions of QAN

173
00:12:26,466 --> 00:12:28,786
or even across different families.

174
00:12:28,806 --> 00:12:30,926
So you're post-training this latent

175
00:12:30,926 --> 00:12:34,026
that then can be projected onto different models.

176
00:12:34,046 --> 00:12:36,266
And that really amortizes the cost

177
00:12:36,266 --> 00:12:38,166
that you have to spend on post-training.

178
00:12:38,986 --> 00:12:41,526
More blog posts on agentic map-reduce,

179
00:12:41,866 --> 00:12:43,906
so not only the pen testing stuff that we did,

180
00:12:43,906 --> 00:12:48,466
but also things like analyzing and creating taxonomies of LLM usage,

181
00:12:48,686 --> 00:12:52,026
which is stuff that has gone into the token spend product.

182
00:12:52,686 --> 00:12:54,526
Things like self-improving monitoring,

183
00:12:54,806 --> 00:13:00,506
where we use LLMs to automatically create monitors with Datadog

184
00:13:00,506 --> 00:13:02,606
on a lot of our product surfaces,

185
00:13:02,606 --> 00:13:04,646
and then as alerts come in,

186
00:13:04,806 --> 00:13:08,466
improve those monitors automatically without a person in the loop

187
00:13:08,466 --> 00:13:12,806
to try to basically create super, super robust and resilient software.

188
00:13:13,906 --> 00:13:17,466
And finally, semantic tagging, which is this interesting foray

189
00:13:17,466 --> 00:13:22,586
into creating this egentic map reduce over a bunch of user sessions

190
00:13:22,586 --> 00:13:26,046
where we can annotate and create taxonomies that are open-ended,

191
00:13:26,266 --> 00:13:32,446
so not knowing ahead of time what are the clusters of LLM usage

192
00:13:32,446 --> 00:13:35,946
and trying to create a taxonomy that can then be reconciled

193
00:13:35,946 --> 00:13:38,066
against the spend inside of the companies.

194
00:13:38,286 --> 00:13:41,586
So if you're interested in this type of research,

195
00:13:41,586 --> 00:13:48,726
x.com slash ramp labs where you can go and check out more of our research and

196
00:13:49,466 --> 00:13:54,586
there's my Twitter if you have any other questions or want to talk about some

197
00:13:54,586 --> 00:13:59,666
use case as part of your applied research and that's it thank you