1
00:11:31,920 --> 00:11:33,750
It's so good to be here. Thank you for

2
00:11:33,760 --> 00:11:35,269
being with me this morning. First

3
00:11:35,279 --> 00:11:37,030
session of the day. I see everyone here

4
00:11:37,040 --> 00:11:39,910
is a morning person. So I'm here to talk

5
00:11:39,920 --> 00:11:42,470
to you about what will you build when AI

6
00:11:42,480 --> 00:11:44,870
is 10 times faster. We're all builders

7
00:11:44,880 --> 00:11:46,550
here. So we're all thinking about what's

8
00:11:46,560 --> 00:11:48,550
next.

9
00:11:48,560 --> 00:11:50,389
Just a bit about me. Um my name is

10
00:11:50,399 --> 00:11:51,990
Natalie. I'm a co-founder of Gimlet

11
00:11:52,000 --> 00:11:54,150
Labs. At Gimlet, we're building an

12
00:11:54,160 --> 00:11:55,750
inference cloud that's designed to

13
00:11:55,760 --> 00:11:58,310
produce much, much faster tokens at high

14
00:11:58,320 --> 00:12:00,069
efficiency, leveraging different

15
00:12:00,079 --> 00:12:02,470
categories and types of chips. At

16
00:12:02,480 --> 00:12:04,710
Gimlet, I work on our kernel generation

17
00:12:04,720 --> 00:12:06,870
side of things across different hardware

18
00:12:06,880 --> 00:12:10,230
types. Um, another thing about me is I

19
00:12:10,240 --> 00:12:12,470
love agents. I use my coding agents for

20
00:12:12,480 --> 00:12:14,550
everything, but I really, really hate

21
00:12:14,560 --> 00:12:17,509
waiting.

22
00:12:17,519 --> 00:12:19,269
And one thing that given that kind of

23
00:12:19,279 --> 00:12:20,790
surprises me is when you're building an

24
00:12:20,800 --> 00:12:22,310
ultra fast inference cloud, people ask

25
00:12:22,320 --> 00:12:24,230
me this all the time actually. They're

26
00:12:24,240 --> 00:12:26,150
like, isn't inference already fast

27
00:12:26,160 --> 00:12:28,150
enough?

28
00:12:28,160 --> 00:12:30,310
And I'm like, what?

29
00:12:30,320 --> 00:12:32,870
But they have a point, right? Because I

30
00:12:32,880 --> 00:12:34,389
can do things and we can all do things

31
00:12:34,399 --> 00:12:36,389
with our agents so much faster than we

32
00:12:36,399 --> 00:12:38,870
could have done before. For example, uh

33
00:12:38,880 --> 00:12:41,910
this slide here, I actually made pretty

34
00:12:41,920 --> 00:12:44,470
using my agent and it took me with my

35
00:12:44,480 --> 00:12:46,790
agent much less time than it would take

36
00:12:46,800 --> 00:12:48,790
me to do that by myself. I'm not very

37
00:12:48,800 --> 00:12:52,310
good at design. And so I think that from

38
00:12:52,320 --> 00:12:54,069
an endto-end perspective, obviously

39
00:12:54,079 --> 00:12:56,629
we're getting more done, but in terms of

40
00:12:56,639 --> 00:12:58,310
is it fast enough, like that's still a

41
00:12:58,320 --> 00:13:00,629
question. So let's take a look at what

42
00:13:00,639 --> 00:13:02,949
happened when I used my agent to make

43
00:13:02,959 --> 00:13:04,550
that slide look pretty. You can see the

44
00:13:04,560 --> 00:13:08,629
ugly before slide on the right.

45
00:13:08,639 --> 00:13:12,389
So, we're going to press play here.

46
00:13:12,399 --> 00:13:14,230
And I said to my agent, can you improve

47
00:13:14,240 --> 00:13:16,710
the visual look of this slide? I love

48
00:13:16,720 --> 00:13:18,710
Figma. I use Figma for everything. So,

49
00:13:18,720 --> 00:13:20,389
of course, I'm going to use it with my

50
00:13:20,399 --> 00:13:22,310
agents. So, what we can see on the

51
00:13:22,320 --> 00:13:25,030
bottom here is the timeline of what

52
00:13:25,040 --> 00:13:26,949
happened. And it the text is a little

53
00:13:26,959 --> 00:13:28,790
small to see, but red is inference. Red

54
00:13:28,800 --> 00:13:30,310
is the time spent that the model is in

55
00:13:30,320 --> 00:13:32,470
inference. Green is the time it's spent

56
00:13:32,480 --> 00:13:34,389
making tool calls. and blue is the time

57
00:13:34,399 --> 00:13:38,230
that it's spent taking my input. And we

58
00:13:38,240 --> 00:13:39,910
can see that, you know, this this

59
00:13:39,920 --> 00:13:41,910
agent's doing good work. It's doing a

60
00:13:41,920 --> 00:13:44,230
lot of thinking, but I'm waiting here.

61
00:13:44,240 --> 00:13:45,910
I'm actually waiting so long I sped up

62
00:13:45,920 --> 00:13:47,910
this video 9x because I didn't want all

63
00:13:47,920 --> 00:13:49,910
of you to have to wait for the full turn

64
00:13:49,920 --> 00:13:52,069
to end. It ended up being about 2

65
00:13:52,079 --> 00:13:53,990
minutes long. But we can see at the end

66
00:13:54,000 --> 00:13:56,389
of this after my one input, we have this

67
00:13:56,399 --> 00:13:57,990
nice pretty slide at the end. So, you

68
00:13:58,000 --> 00:13:59,910
know, on one hand, this is actually much

69
00:13:59,920 --> 00:14:02,069
faster than a human can do. But on the

70
00:14:02,079 --> 00:14:03,829
other hand, in those about two minutes,

71
00:14:03,839 --> 00:14:05,590
I have probably moved on to another

72
00:14:05,600 --> 00:14:07,350
task. I'm not really focused on this

73
00:14:07,360 --> 00:14:11,509
anymore. I've context switched.

74
00:14:11,519 --> 00:14:13,430
And when we look to, you know, the

75
00:14:13,440 --> 00:14:16,230
inevitable progression of inference

76
00:14:16,240 --> 00:14:17,910
getting faster and faster, I think it

77
00:14:17,920 --> 00:14:20,069
helps to look back in time and look back

78
00:14:20,079 --> 00:14:22,389
to the internet era. Um, I mean, we're

79
00:14:22,399 --> 00:14:23,430
still in the internet, but like you know

80
00:14:23,440 --> 00:14:24,949
what people canonically call the web

81
00:14:24,959 --> 00:14:27,350
era. And what we always see with a new

82
00:14:27,360 --> 00:14:30,150
shift is that with each new major wave,

83
00:14:30,160 --> 00:14:32,470
new capabilities are unlocked. And so we

84
00:14:32,480 --> 00:14:34,310
do things that weren't possible before

85
00:14:34,320 --> 00:14:36,069
that technology, like using an agent to

86
00:14:36,079 --> 00:14:38,470
edit my slide. So I think we should take

87
00:14:38,480 --> 00:14:40,550
a step back to 2000 in the early days of

88
00:14:40,560 --> 00:14:42,069
the internet. I think many people here

89
00:14:42,079 --> 00:14:43,670
weren't even alive, so it'll be a

90
00:14:43,680 --> 00:14:45,829
history lesson for you. And we're going

91
00:14:45,839 --> 00:14:47,990
to look at a little application called

92
00:14:48,000 --> 00:14:50,470
Squirrel Mail. Squirrel Mail is actually

93
00:14:50,480 --> 00:14:52,069
open source, so you can run it today. It

94
00:14:52,079 --> 00:14:53,829
was a web application for uh you know

95
00:14:53,839 --> 00:14:56,550
doing your email back then.

96
00:14:56,560 --> 00:14:58,870
So on the left what we have here is I'm

97
00:14:58,880 --> 00:15:01,189
running squirrel mail and I'm running it

98
00:15:01,199 --> 00:15:05,110
with simulated 2000 dialup speed and

99
00:15:05,120 --> 00:15:07,590
then on the right it'll run with 2026

100
00:15:07,600 --> 00:15:09,189
speed. So the speed that we're used to

101
00:15:09,199 --> 00:15:11,670
today. So the both sides are going to

102
00:15:11,680 --> 00:15:14,230
take the same actions but um the left

103
00:15:14,240 --> 00:15:16,790
side I'm using it with that simulated

104
00:15:16,800 --> 00:15:21,590
lower uh speed internet

105
00:15:21,600 --> 00:15:23,750
and we can see that you know it's doing

106
00:15:23,760 --> 00:15:27,110
its job. Back in 2000 loading an image

107
00:15:27,120 --> 00:15:28,389
like this would have blown people's

108
00:15:28,399 --> 00:15:31,110
minds but our modern eyes are like wow

109
00:15:31,120 --> 00:15:33,910
that took forever to show up.

110
00:15:33,920 --> 00:15:35,910
And so we can see these lags. we can see

111
00:15:35,920 --> 00:15:37,350
a noticeable thing, but on the same

112
00:15:37,360 --> 00:15:38,710
time, someone might have looked at this

113
00:15:38,720 --> 00:15:40,710
back in 2000 and said, "This is fast

114
00:15:40,720 --> 00:15:42,150
enough. I don't actually need this to be

115
00:15:42,160 --> 00:15:46,629
faster."

116
00:15:46,639 --> 00:15:48,870
The thing is though, all of the major

117
00:15:48,880 --> 00:15:51,350
giants back in that era did find out it

118
00:15:51,360 --> 00:15:53,829
mattered. Pretty much every major

119
00:15:53,839 --> 00:15:56,550
company from Amazon to Yahoo to Google

120
00:15:56,560 --> 00:15:59,030
to Bing, all of or being at part of

121
00:15:59,040 --> 00:16:01,189
Microsoft like all of them found that

122
00:16:01,199 --> 00:16:03,910
even very small delays cost them users.

123
00:16:03,920 --> 00:16:06,629
It cost them people coming back. It cost

124
00:16:06,639 --> 00:16:08,790
them people doing as many clicks. It

125
00:16:08,800 --> 00:16:11,189
cost them literal money. Even a hundred

126
00:16:11,199 --> 00:16:13,749
milliseconds of delay added on to the

127
00:16:13,759 --> 00:16:16,150
typical experience had a demonstrable

128
00:16:16,160 --> 00:16:19,030
impact on the bottom line. And even when

129
00:16:19,040 --> 00:16:20,710
you got rid of that delay, those same

130
00:16:20,720 --> 00:16:23,110
users didn't come back to the platform

131
00:16:23,120 --> 00:16:29,509
as often as they did without that delay.

132
00:16:29,519 --> 00:16:31,430
And this makes sense because fast

133
00:16:31,440 --> 00:16:33,430
products keep us in flow state. They

134
00:16:33,440 --> 00:16:36,230
provide a stickier UX. About 100

135
00:16:36,240 --> 00:16:38,790
milliseconds feels instant to us. One

136
00:16:38,800 --> 00:16:41,030
seconds, one second of weight, I still

137
00:16:41,040 --> 00:16:42,790
keep my train of thought. It's not a big

138
00:16:42,800 --> 00:16:44,710
deal. I don't really notice it. 10

139
00:16:44,720 --> 00:16:46,629
seconds, we're going to start turning to

140
00:16:46,639 --> 00:16:48,069
something else from a psychology

141
00:16:48,079 --> 00:16:49,269
perspective. Maybe we pull out our

142
00:16:49,279 --> 00:16:51,590
phone, maybe switch to another agent.

143
00:16:51,600 --> 00:16:53,829
But today, from data from June, the

144
00:16:53,839 --> 00:16:59,749
median coding agent turn is 63 seconds.

145
00:16:59,759 --> 00:17:01,430
Each workload has different speed

146
00:17:01,440 --> 00:17:03,189
requirements. We don't need the same

147
00:17:03,199 --> 00:17:05,350
speed for everything. But over time, our

148
00:17:05,360 --> 00:17:08,390
expectations do rise in the fastest days

149
00:17:08,400 --> 00:17:10,390
of fastest internet in the squirrel mail

150
00:17:10,400 --> 00:17:12,390
days would feel incredibly slow to us

151
00:17:12,400 --> 00:17:14,230
and we can keep we can expect that trend

152
00:17:14,240 --> 00:17:17,990
to continue for inference speed as well.

153
00:17:18,000 --> 00:17:19,590
There's a problem though and the problem

154
00:17:19,600 --> 00:17:21,990
is everyone knows that we have massive

155
00:17:22,000 --> 00:17:24,470
compute scarcity. Today's infrastructure

156
00:17:24,480 --> 00:17:26,870
is under incredible pressure to deliver

157
00:17:26,880 --> 00:17:29,669
two things at the same time. More tokens

158
00:17:29,679 --> 00:17:32,710
and faster tokens. Unfortunately, these

159
00:17:32,720 --> 00:17:35,029
goals are fundamentally at odds because

160
00:17:35,039 --> 00:17:37,430
faster tokens require more energy to

161
00:17:37,440 --> 00:17:39,990
produce than slower tokens. So, with the

162
00:17:40,000 --> 00:17:41,909
same energy footprint, you can't make as

163
00:17:41,919 --> 00:17:44,789
many of them.

164
00:17:44,799 --> 00:17:47,190
Back before this was a session and not a

165
00:17:47,200 --> 00:17:49,350
keynote, thanks to upgrading this to a

166
00:17:49,360 --> 00:17:52,390
keynote everyone. Um, we uh we're going

167
00:17:52,400 --> 00:17:54,950
to play a game that I made uh called

168
00:17:54,960 --> 00:17:57,029
Token Mint Factory. And you can actually

169
00:17:57,039 --> 00:17:58,470
play this. I will not be offended if you

170
00:17:58,480 --> 00:17:59,750
play it right now even, but just so you

171
00:17:59,760 --> 00:18:01,190
know, it does not work on phones. It

172
00:18:01,200 --> 00:18:03,190
only works on your laptop. And basically

173
00:18:03,200 --> 00:18:04,789
in this game, you're running a token

174
00:18:04,799 --> 00:18:06,310
mint factory. You're printing literal

175
00:18:06,320 --> 00:18:08,630
tokens. And there's different modes. The

176
00:18:08,640 --> 00:18:11,350
modes are you can play to maximize how

177
00:18:11,360 --> 00:18:12,950
quickly you can put out one single

178
00:18:12,960 --> 00:18:15,510
token. Or you can maximize how many

179
00:18:15,520 --> 00:18:18,070
tokens you can put out in 60 seconds. Or

180
00:18:18,080 --> 00:18:20,150
you can play the SLA mode where you have

181
00:18:20,160 --> 00:18:22,230
to maintain a 20 second SLA on each

182
00:18:22,240 --> 00:18:23,669
token for it to count, but you're still

183
00:18:23,679 --> 00:18:25,830
trying to put out as many as you can.

184
00:18:25,840 --> 00:18:27,510
And as you play, you'll quickly see that

185
00:18:27,520 --> 00:18:29,590
the strategy is extremely different for

186
00:18:29,600 --> 00:18:31,430
doing fast tokens versus doing slow

187
00:18:31,440 --> 00:18:33,830
tokens. And neither one is actually

188
00:18:33,840 --> 00:18:35,590
better or worse from a decision

189
00:18:35,600 --> 00:18:37,110
perspective. It's just a matter of what

190
00:18:37,120 --> 00:18:38,549
the application needs and what makes

191
00:18:38,559 --> 00:18:41,669
sense for your factory.

192
00:18:41,679 --> 00:18:43,990
So I took real runs from people at my

193
00:18:44,000 --> 00:18:46,150
company that played this game and I made

194
00:18:46,160 --> 00:18:48,870
a paro frontier of these tokens in the

195
00:18:48,880 --> 00:18:51,190
game. And we can see there's a real life

196
00:18:51,200 --> 00:18:53,990
paro frontier of this game as well. If

197
00:18:54,000 --> 00:18:55,830
we have faster tokens, we don't have as

198
00:18:55,840 --> 00:18:57,510
many. And if we have more tokens,

199
00:18:57,520 --> 00:19:00,150
they're not as fast. But there is a way

200
00:19:00,160 --> 00:19:01,830
to get more up and to the right. And up

201
00:19:01,840 --> 00:19:03,430
and to the right is where we want to go.

202
00:19:03,440 --> 00:19:05,510
We want more faster tokens. I don't want

203
00:19:05,520 --> 00:19:08,470
to compromise. I want both. So the game

204
00:19:08,480 --> 00:19:10,390
has a concept of adding upgrades to your

205
00:19:10,400 --> 00:19:12,789
equipment. And what we can see is that

206
00:19:12,799 --> 00:19:14,950
compared to the no upgrade baseline,

207
00:19:14,960 --> 00:19:17,190
these technological innovations help

208
00:19:17,200 --> 00:19:19,430
drive us up into the right and get more

209
00:19:19,440 --> 00:19:23,350
tokens and faster tokens.

210
00:19:23,360 --> 00:19:25,350
So how do we upgrade our real life

211
00:19:25,360 --> 00:19:28,310
assembly lines so to speak to make

212
00:19:28,320 --> 00:19:31,909
making our actual tokens faster? Uh one

213
00:19:31,919 --> 00:19:33,830
thing that we do at Gimlet for this is

214
00:19:33,840 --> 00:19:35,669
we really believe that you have to use

215
00:19:35,679 --> 00:19:37,750
the right accelerator for the right job.

216
00:19:37,760 --> 00:19:40,070
And so we actually take models and we

217
00:19:40,080 --> 00:19:42,310
split them up into different pieces and

218
00:19:42,320 --> 00:19:44,470
we run each piece on a different

219
00:19:44,480 --> 00:19:46,789
category of accelerator. So we might run

220
00:19:46,799 --> 00:19:49,350
one piece of the workload on a GPU and

221
00:19:49,360 --> 00:19:51,029
another piece of the workload on

222
00:19:51,039 --> 00:19:52,390
something like an SRM ccentric

223
00:19:52,400 --> 00:19:54,310
accelerator. And we find that even with

224
00:19:54,320 --> 00:19:56,070
the network overhead, this delivers

225
00:19:56,080 --> 00:19:57,830
dramatically faster tokens. It's

226
00:19:57,840 --> 00:19:59,669
essentially like the same thing as the

227
00:19:59,679 --> 00:20:01,350
upgrade to the equipment in the token

228
00:20:01,360 --> 00:20:03,110
factory when you better match the

229
00:20:03,120 --> 00:20:05,350
accelerator to the job. Unfortunately,

230
00:20:05,360 --> 00:20:07,750
we have ended the time of free lunch

231
00:20:07,760 --> 00:20:09,350
transistor scaling, right? So now we

232
00:20:09,360 --> 00:20:11,029
have to look to other approaches to make

233
00:20:11,039 --> 00:20:13,990
things faster.

234
00:20:14,000 --> 00:20:16,230
And so with this, it's a little hard to

235
00:20:16,240 --> 00:20:18,549
read, but we can see the same real life

236
00:20:18,559 --> 00:20:21,430
paro frontier for a frontier model. If

237
00:20:21,440 --> 00:20:23,110
you look at, you know, simple

238
00:20:23,120 --> 00:20:25,430
homogeneous GPU inference, it's very

239
00:20:25,440 --> 00:20:26,950
high throughput. And then the latency

240
00:20:26,960 --> 00:20:28,470
drops off a little bit. You don't you

241
00:20:28,480 --> 00:20:30,310
can't get it as fast because the y-axis

242
00:20:30,320 --> 00:20:32,549
is how many tokens that I can make and

243
00:20:32,559 --> 00:20:34,870
the x-axis how fast these tokens are for

244
00:20:34,880 --> 00:20:37,430
each user. And we can see that depending

245
00:20:37,440 --> 00:20:39,190
on how you split the model up because

246
00:20:39,200 --> 00:20:41,029
there's different ways of doing it. You

247
00:20:41,039 --> 00:20:43,750
can get much faster tokens for the same

248
00:20:43,760 --> 00:20:45,909
power footprint.

249
00:20:45,919 --> 00:20:48,070
You can get many more tokens for the

250
00:20:48,080 --> 00:20:51,029
same speed or you can get more and

251
00:20:51,039 --> 00:20:54,230
faster tokens. So as an industry we need

252
00:20:54,240 --> 00:20:56,070
to continue pushing pushing pushing on

253
00:20:56,080 --> 00:20:57,510
this because otherwise our poor data

254
00:20:57,520 --> 00:21:01,190
centers are just going to keel over.

255
00:21:01,200 --> 00:21:03,270
So I just wanted to show a quick demo of

256
00:21:03,280 --> 00:21:06,310
this. So um using a coding agent build

257
00:21:06,320 --> 00:21:08,149
the same app using heterogeneous

258
00:21:08,159 --> 00:21:09,590
inference that leverages different chip

259
00:21:09,600 --> 00:21:14,310
types versus the traditional one.

260
00:21:14,320 --> 00:21:16,149
So this is going to be the game snake.

261
00:21:16,159 --> 00:21:17,750
And we can see on the left the

262
00:21:17,760 --> 00:21:20,390
heterogeneous fast one's done. I can see

263
00:21:20,400 --> 00:21:21,909
in my flow state I can keep adding

264
00:21:21,919 --> 00:21:23,830
things. I can keep playing but on the

265
00:21:23,840 --> 00:21:25,669
right side it's still going. And by the

266
00:21:25,679 --> 00:21:27,029
way I still think the left side's too

267
00:21:27,039 --> 00:21:31,750
slow. We need to make that even faster.

268
00:21:31,760 --> 00:21:33,350
And then we can add a competitor to the

269
00:21:33,360 --> 00:21:35,270
game and we can see even with the tool

270
00:21:35,280 --> 00:21:36,710
calls, even with all the things that

271
00:21:36,720 --> 00:21:39,270
it's doing, it's going to complete and

272
00:21:39,280 --> 00:21:40,870
then the other one will still be hanging

273
00:21:40,880 --> 00:21:42,710
because it's fundamentally about how can

274
00:21:42,720 --> 00:21:44,230
you drive more efficiency and better

275
00:21:44,240 --> 00:21:49,029
performance with the same resources.

276
00:21:49,039 --> 00:21:50,230
It's going to go on. I'm not going to

277
00:21:50,240 --> 00:21:53,350
make you guys wait for this.

278
00:21:53,360 --> 00:21:54,870
One thing that people often point out

279
00:21:54,880 --> 00:21:57,909
too is is inference the bottleneck? Our

280
00:21:57,919 --> 00:21:59,510
tool calls the bottleneck. And I think

281
00:21:59,520 --> 00:22:01,590
this is a great point. If we look back

282
00:22:01,600 --> 00:22:04,470
to the turn timeline from the time that

283
00:22:04,480 --> 00:22:07,029
I was spending updating my slide, we can

284
00:22:07,039 --> 00:22:09,590
see that 80% of it was inference and 18%

285
00:22:09,600 --> 00:22:12,470
of it was tool calls. If we simulate

286
00:22:12,480 --> 00:22:15,350
that same exact thing and have 10x

287
00:22:15,360 --> 00:22:18,789
faster inference, we can see that 67% of

288
00:22:18,799 --> 00:22:20,549
that will be tool calls at that point if

289
00:22:20,559 --> 00:22:23,029
we don't optimize the tool calls. That's

290
00:22:23,039 --> 00:22:24,549
why it's important that we have to

291
00:22:24,559 --> 00:22:26,390
constantly attack the bottlenecks of the

292
00:22:26,400 --> 00:22:27,909
application and get the lowhanging fruit

293
00:22:27,919 --> 00:22:29,909
where we can. Today, people are focused

294
00:22:29,919 --> 00:22:31,669
on inference speed and maybe we haven't

295
00:22:31,679 --> 00:22:33,430
optimized the tool call side and how

296
00:22:33,440 --> 00:22:35,270
tools are called as much because it's

297
00:22:35,280 --> 00:22:37,029
not the bottleneck. If I cut away the

298
00:22:37,039 --> 00:22:38,710
tool calls on the upper one, it wouldn't

299
00:22:38,720 --> 00:22:39,830
even really matter to the end

300
00:22:39,840 --> 00:22:41,590
application. But once we get much

301
00:22:41,600 --> 00:22:43,510
faster, we have to think about how to

302
00:22:43,520 --> 00:22:45,190
call these tools much more efficiently

303
00:22:45,200 --> 00:22:46,789
and much more performantly because

304
00:22:46,799 --> 00:22:48,070
otherwise they're going to be the reason

305
00:22:48,080 --> 00:22:53,190
that we're all waiting around.

306
00:22:53,200 --> 00:22:55,350
I think the good news is that optimizing

307
00:22:55,360 --> 00:22:56,630
these speeds and what we're all

308
00:22:56,640 --> 00:22:58,950
collectively doing as an industry is a

309
00:22:58,960 --> 00:23:00,950
verifiable task. If I know what my

310
00:23:00,960 --> 00:23:03,510
inputs and outputs should be, I can have

311
00:23:03,520 --> 00:23:06,710
agents do this very very full stack. And

312
00:23:06,720 --> 00:23:08,549
that's where we're increasingly moving

313
00:23:08,559 --> 00:23:10,710
to. We're having self-improvement loops

314
00:23:10,720 --> 00:23:12,390
with these agents to drive better

315
00:23:12,400 --> 00:23:14,549
efficiency on the fly as our inference

316
00:23:14,559 --> 00:23:16,390
is running. And at Gimlet, we're

317
00:23:16,400 --> 00:23:18,230
building recursive self-optimization for

318
00:23:18,240 --> 00:23:20,710
this that pulls from low-level telemetry

319
00:23:20,720 --> 00:23:22,310
from our data centers all the way up to

320
00:23:22,320 --> 00:23:24,310
the model layer and trying to use all of

321
00:23:24,320 --> 00:23:26,870
it to in to improve the overall

322
00:23:26,880 --> 00:23:28,390
efficiency of the workload because if

323
00:23:28,400 --> 00:23:30,149
you just optimize one small piece, it

324
00:23:30,159 --> 00:23:35,110
might not make that much of a dent.

325
00:23:35,120 --> 00:23:37,029
I want to show another demo because I

326
00:23:37,039 --> 00:23:38,630
think that cyber security is on

327
00:23:38,640 --> 00:23:41,110
everyone's mind right now. Um, we're all

328
00:23:41,120 --> 00:23:43,510
seeing like the huge incredible advances

329
00:23:43,520 --> 00:23:46,310
in AI driving awareness that there's all

330
00:23:46,320 --> 00:23:48,390
these vulnerabilities in our software

331
00:23:48,400 --> 00:23:50,390
and I think that this is an area where

332
00:23:50,400 --> 00:23:53,350
faster inference can really really help.

333
00:23:53,360 --> 00:23:55,909
So this is a dumb fake scenario but it's

334
00:23:55,919 --> 00:23:58,310
a real application and a real demo where

335
00:23:58,320 --> 00:24:00,549
imagine that you're running the dog club

336
00:24:00,559 --> 00:24:02,630
and that's a website for dog lovers to

337
00:24:02,640 --> 00:24:05,110
upload and vote on pictures of dogs.

338
00:24:05,120 --> 00:24:07,909
However, unfortunately, like we often

339
00:24:07,919 --> 00:24:10,710
see, one of your node dependencies has a

340
00:24:10,720 --> 00:24:12,549
vulnerability and this is a real

341
00:24:12,559 --> 00:24:15,190
vulnerability. This is really a node tar

342
00:24:15,200 --> 00:24:17,830
and you're running a security agent on

343
00:24:17,840 --> 00:24:19,830
your application and it's trying to

344
00:24:19,840 --> 00:24:21,830
monitor live traffic and see is there

345
00:24:21,840 --> 00:24:23,190
anything going wrong that I don't know

346
00:24:23,200 --> 00:24:24,870
about. This agent doesn't know about

347
00:24:24,880 --> 00:24:26,630
this vulnerability, but it does know its

348
00:24:26,640 --> 00:24:31,269
job is to protect your application.

349
00:24:31,279 --> 00:24:33,190
So, let's do a quick turn with normal

350
00:24:33,200 --> 00:24:36,149
user traffic. We're going to upload a

351
00:24:36,159 --> 00:24:39,669
dog biscuit.

352
00:24:39,679 --> 00:24:41,350
And on the upper side, we have the

353
00:24:41,360 --> 00:24:43,190
application logs. And on the bottom, we

354
00:24:43,200 --> 00:24:44,789
have the defender. And the defender is

355
00:24:44,799 --> 00:24:46,230
monitoring, but it's not in the critical

356
00:24:46,240 --> 00:24:48,470
path because it's not fast enough to

357
00:24:48,480 --> 00:24:50,390
actually respond to every user request

358
00:24:50,400 --> 00:24:54,470
live. It's monitoring things async.

359
00:24:54,480 --> 00:24:56,310
And we can see biscuit gets uploaded and

360
00:24:56,320 --> 00:24:58,549
we see that this is an okay, you know,

361
00:24:58,559 --> 00:25:00,149
this is an okay upload. The defender

362
00:25:00,159 --> 00:25:02,470
doesn't see any problem.

363
00:25:02,480 --> 00:25:04,070
But now what we're going to do is have

364
00:25:04,080 --> 00:25:05,830
this defender operating when there's a

365
00:25:05,840 --> 00:25:07,830
malicious upload. And this upload is

366
00:25:07,840 --> 00:25:09,750
exploiting the real vulnerability where

367
00:25:09,760 --> 00:25:11,830
in node tar there was a release where

368
00:25:11,840 --> 00:25:13,669
you could basically overwrite files that

369
00:25:13,679 --> 00:25:15,110
you weren't supposed to overwrite based

370
00:25:15,120 --> 00:25:17,190
on the destinations of what was being

371
00:25:17,200 --> 00:25:19,669
unzipped.

372
00:25:19,679 --> 00:25:23,990
So in this one we have Ranger, a fake

373
00:25:24,000 --> 00:25:27,830
bad uh malicious dog. Not really a dog.

374
00:25:27,840 --> 00:25:30,070
And what we're going to see is that the

375
00:25:30,080 --> 00:25:32,230
defender is going to figure out that

376
00:25:32,240 --> 00:25:34,149
this is a malicious thing. The agent is

377
00:25:34,159 --> 00:25:35,990
smart enough to detect that it's doing

378
00:25:36,000 --> 00:25:37,269
something that it shouldn't be doing,

379
00:25:37,279 --> 00:25:38,789
even though it hasn't seen this exact

380
00:25:38,799 --> 00:25:41,830
vulnerability before. But unfortunately,

381
00:25:41,840 --> 00:25:44,950
Cat's malicious upload has taken over.

382
00:25:44,960 --> 00:25:47,110
So your website's gone. It's too late.

383
00:25:47,120 --> 00:25:52,390
You didn't react in time.

384
00:25:52,400 --> 00:25:54,310
But if we retry this same exact

385
00:25:54,320 --> 00:25:56,230
malicious upload, but this time with an

386
00:25:56,240 --> 00:25:58,070
ultra fast defender, and again, this is

387
00:25:58,080 --> 00:26:01,750
a real a real demo fake use case. Um, we

388
00:26:01,760 --> 00:26:03,909
can see that with ultra fast inference,

389
00:26:03,919 --> 00:26:05,669
we can react to things and do things

390
00:26:05,679 --> 00:26:07,750
online much more quickly. And this

391
00:26:07,760 --> 00:26:09,029
should change the way that we think of

392
00:26:09,039 --> 00:26:10,950
how to use agents in the first place.

393
00:26:10,960 --> 00:26:12,310
We're all thinking about how to use them

394
00:26:12,320 --> 00:26:14,310
in an async way, but we should think

395
00:26:14,320 --> 00:26:16,230
about how to use them in online ways as

396
00:26:16,240 --> 00:26:18,230
well. So we can see on the bottom that

397
00:26:18,240 --> 00:26:19,990
the worker blocked their request and it

398
00:26:20,000 --> 00:26:21,430
was like what are you doing? This is

399
00:26:21,440 --> 00:26:26,630
obviously wrong.

400
00:26:26,640 --> 00:26:29,269
So faster internet took us from squirrel

401
00:26:29,279 --> 00:26:32,149
mail to ondemand highresolution

402
00:26:32,159 --> 00:26:34,070
streaming, right? Like if you just look

403
00:26:34,080 --> 00:26:36,149
from this to this, it's crazy. If you

404
00:26:36,159 --> 00:26:37,990
showed this to my like 5-year-old self

405
00:26:38,000 --> 00:26:39,669
or something, I would have been like my

406
00:26:39,679 --> 00:26:41,510
mind would have been blown. And this is

407
00:26:41,520 --> 00:26:42,870
just one of many things that I can

408
00:26:42,880 --> 00:26:45,590
select. So, like what does a 10,000

409
00:26:45,600 --> 00:26:47,510
token per second world look like? We're

410
00:26:47,520 --> 00:26:50,149
at in the like low hundreds today. What

411
00:26:50,159 --> 00:26:51,430
does it look like when we get to a

412
00:26:51,440 --> 00:26:52,870
million tokens per second? Like what

413
00:26:52,880 --> 00:26:56,870
does that unlock for us?

414
00:26:56,880 --> 00:26:58,230
And I think that we should all ask

415
00:26:58,240 --> 00:27:00,230
ourselves this too. It's important in AI

416
00:27:00,240 --> 00:27:01,990
to skate to where the puck is moving,

417
00:27:02,000 --> 00:27:04,230
right? It's changing so fast. We need to

418
00:27:04,240 --> 00:27:06,470
be ready. We need to think boldly. It's

419
00:27:06,480 --> 00:27:08,390
not enough to just do what we've always

420
00:27:08,400 --> 00:27:09,830
been doing. I think we've all had the

421
00:27:09,840 --> 00:27:11,909
experience where we make an audacious

422
00:27:11,919 --> 00:27:13,590
request to our agent and after the

423
00:27:13,600 --> 00:27:16,070
latest model release, it just works. And

424
00:27:16,080 --> 00:27:17,269
it's going to be the same thing with

425
00:27:17,279 --> 00:27:19,269
latency. And so I think that we can

426
00:27:19,279 --> 00:27:21,190
think about ideas that are both coming

427
00:27:21,200 --> 00:27:23,830
soon when we hit a certain latency

428
00:27:23,840 --> 00:27:25,830
threshold, what we can do, but also the

429
00:27:25,840 --> 00:27:27,909
stuff that is real life sci-fi. Like if

430
00:27:27,919 --> 00:27:29,430
you look at something like Jarvis or

431
00:27:29,440 --> 00:27:31,110
something, there's no way today's

432
00:27:31,120 --> 00:27:32,870
inference could be fast enough to power

433
00:27:32,880 --> 00:27:34,710
that experience for Iron Man. But we

434
00:27:34,720 --> 00:27:37,669
will get there. So, I'm really excited

435
00:27:37,679 --> 00:27:40,549
about online use cases, online security,

436
00:27:40,559 --> 00:27:43,269
online things of like I want to just

437
00:27:43,279 --> 00:27:45,190
tell my agent what I want to build and

438
00:27:45,200 --> 00:27:46,390
it's building it right in front of me

439
00:27:46,400 --> 00:27:48,310
and I don't even have to type anything.

440
00:27:48,320 --> 00:27:49,990
I think we're going to have new patterns

441
00:27:50,000 --> 00:27:52,149
of interaction. You know, think tab

442
00:27:52,159 --> 00:27:54,230
complete for everything. How can we have

443
00:27:54,240 --> 00:27:56,389
the user side of things be as quick and

444
00:27:56,399 --> 00:27:58,070
immediate as possible because the

445
00:27:58,080 --> 00:28:00,149
inference side is also just as fast to

446
00:28:00,159 --> 00:28:02,549
accommodate it. And we're going to have

447
00:28:02,559 --> 00:28:04,549
so many more agents. We're going to have

448
00:28:04,559 --> 00:28:06,870
even maybe transformers driving robots,

449
00:28:06,880 --> 00:28:09,029
right? There's so much that we can build

450
00:28:09,039 --> 00:28:11,110
when this gets unlocked. And I think as

451
00:28:11,120 --> 00:28:12,710
builders on the frontier here, we should

452
00:28:12,720 --> 00:28:16,470
all be thinking about this, too.

453
00:28:16,480 --> 00:28:19,110
Thank you so much for being with me um

454
00:28:19,120 --> 00:28:21,430
this morning. Uh you can find me on X at

455
00:28:21,440 --> 00:28:24,549
NSino. Um we're hiring at Gimlet. Would

456
00:28:24,559 --> 00:28:26,310
love to talk to you all. And then if you

457
00:28:26,320 --> 00:28:28,549
want to play the token mint game, uh you

458
00:28:28,559 --> 00:28:30,789
can also find that QR here. Thank you so