1
00:40:03,800 --> 00:40:12,960
Hello, my name is Armin Ronohar and I want to share with you a little bit the lessons that we have learned from building software with agents.

2
00:40:13,260 --> 00:40:19,820
And in particular, I want to make a strong case for not so much building the factory as building the artiller or workshop.

3
00:40:21,520 --> 00:40:25,240
and I want to show some data with this from our own usage but in particular I

4
00:40:25,240 --> 00:40:30,920
want to explain what I mean by building the atelier or the workshop prior to

5
00:40:30,920 --> 00:40:36,540
building the software factory and I think everybody here is being used has

6
00:40:36,540 --> 00:40:40,020
been using agents for quite a while and you have probably noticed that it is

7
00:40:40,020 --> 00:40:45,960
we're sort of an exponential curve in what we can do with them but the

8
00:40:45,960 --> 00:40:49,800
comparison that I want to give here is that you can be an amazing artist and

9
00:40:49,800 --> 00:40:51,640
and use the greatest and latest technology

10
00:40:51,640 --> 00:40:53,600
from CNC machines to 3D printers,

11
00:40:53,860 --> 00:40:55,520
which is very good power tools.

12
00:40:55,760 --> 00:40:57,140
And it can increase the ambition

13
00:40:57,140 --> 00:40:58,640
of what it can do with your art.

14
00:40:58,920 --> 00:41:00,980
And I think we can also look at software

15
00:41:00,980 --> 00:41:02,080
a little bit in that sense.

16
00:41:02,460 --> 00:41:05,320
So less about building much more software

17
00:41:05,320 --> 00:41:07,360
and just building way more ambitious

18
00:41:07,360 --> 00:41:09,020
and high quality software.

19
00:41:10,120 --> 00:41:11,940
So I'm going to show you a little bit

20
00:41:11,940 --> 00:41:13,020
of what I mean by that.

21
00:41:15,160 --> 00:41:17,400
And you might know that Arendelle

22
00:41:17,400 --> 00:41:18,900
is working on the PyCoding Agent.

23
00:41:18,900 --> 00:41:22,420
So we have obviously a little bit of a horse in the race here of how we're looking at this.

24
00:41:23,100 --> 00:41:29,920
But I think what is probably quite interesting is that the PyCoding agent has not really changed in a year on the core harness.

25
00:41:30,160 --> 00:41:33,840
It still has the same tools today as it did in December last year.

26
00:41:34,620 --> 00:41:40,500
And I think this is interesting because it has become much more powerful purely as a fact of the models getting better.

27
00:41:41,440 --> 00:41:48,440
So I'm much more capable today of what I can do with Py without any changes landing in the core functionality of what the harness is doing.

28
00:41:48,900 --> 00:41:58,240
And I think this is an interesting thing to reflect on because it means that we are all simultaneously getting better even without landing any significant changes in how we're doing this.

29
00:41:59,760 --> 00:42:09,720
And you might have remembered throughout the last year and a half maybe that there were a lot of ideas of how you can prompt the machines better or how you can do some interesting tricks with the harness.

30
00:42:10,000 --> 00:42:11,580
Very few of them are still necessary.

31
00:42:11,580 --> 00:42:15,580
For me, a good example here is that I was in love with Pi Auto Research, which was a

32
00:42:15,580 --> 00:42:19,680
plugin that basically allowed an agent to drive towards a goal, like improving performance.

33
00:42:19,920 --> 00:42:23,920
But if you use a present-day state-of-the-art model, you actually discover that the entire

34
00:42:23,920 --> 00:42:25,000
plugin is no longer necessary.

35
00:42:25,320 --> 00:42:29,620
So there's a lot that is just happening in the models that is dramatically improving

36
00:42:29,620 --> 00:42:30,060
how we're doing.

37
00:42:30,240 --> 00:42:31,760
So these machines are getting better.

38
00:42:33,400 --> 00:42:38,520
And I just want to show this from a couple of graphs from basically my Pi sessions where

39
00:42:38,520 --> 00:42:41,140
I filtered them out to have nothing but the default set.

40
00:42:41,520 --> 00:42:43,340
And the total amount of corrections,

41
00:42:43,420 --> 00:42:44,840
and I used Jeff to analyze this,

42
00:42:44,920 --> 00:42:46,580
the total amount of corrections that I had to do

43
00:42:47,340 --> 00:42:49,900
got down and down and down with every single model

44
00:42:49,900 --> 00:42:50,860
that got better and better.

45
00:42:51,080 --> 00:42:56,280
So I'm pushing back a lot less against the agent

46
00:42:57,140 --> 00:42:59,820
with the latest model than I did in December last year.

47
00:43:00,580 --> 00:43:02,560
It's the same thing with how often I'm aborting it.

48
00:43:04,100 --> 00:43:07,060
And I think this to me was the most interesting way

49
00:43:07,060 --> 00:43:11,480
in which I sort of analyzed my sessions, which is the average, so this is the P 90th percentile,

50
00:43:11,600 --> 00:43:17,100
but the 90th percentile of how long an agent was running in response to a task that I gave it

51
00:43:17,100 --> 00:43:22,900
with Opus 4.6 in December last year was about two and a half minutes. And now with the latest

52
00:43:22,900 --> 00:43:27,200
models, they are on the 90th percentile running for 14 minutes or more. And that does actually

53
00:43:27,200 --> 00:43:32,560
change something about how you want to work with a harness, but really, again, nothing in Pi has

54
00:43:32,560 --> 00:43:37,480
changed and still I'm seeing this sort of different behavior of how I'm using

55
00:43:37,480 --> 00:43:43,880
this. And this probably you have noticed in one form or another if you're still

56
00:43:43,880 --> 00:43:46,980
using an agent and we're going to see what it's doing you'll probably notice

57
00:43:46,980 --> 00:43:50,580
that the present-day agents are much more likely to use bash than any of the

58
00:43:50,580 --> 00:43:54,160
other tools that are being provided but again nothing has changed in how we

59
00:43:54,160 --> 00:43:57,260
configure the harness it's just that there's a lot more emergent behavior

60
00:43:57,260 --> 00:44:01,000
coming from the models. What's missing here is Astra and Sol they actually look

61
00:44:01,000 --> 00:44:05,920
very similar to Opus for my testing, I just didn't have enough data in my test set to

62
00:44:05,920 --> 00:44:12,700
show this nicely. You will notice that there's a little purple thing here. I want to talk

63
00:44:12,700 --> 00:44:17,760
about this in the end, which is sort of code mode is sort of slowly creeping up. But the

64
00:44:17,760 --> 00:44:24,660
core of this is even if you have a piece of infrastructure nowadays that is sort of there

65
00:44:24,660 --> 00:44:30,060
and is doing a good job, it's still changing as a result of the models. What has not changed

66
00:44:30,060 --> 00:44:32,340
in the last year, at least this is the case for me,

67
00:44:32,420 --> 00:44:34,080
and I think this is the case for a lot of people,

68
00:44:34,440 --> 00:44:35,900
is that we're still humans.

69
00:44:35,900 --> 00:44:38,220
We're still limited by our fundamental capabilities.

70
00:44:39,040 --> 00:44:40,620
And as we are being more ambitious

71
00:44:40,620 --> 00:44:42,080
and we're doing more with these machines,

72
00:44:43,760 --> 00:44:45,520
I don't want to say we're the bottleneck,

73
00:44:45,560 --> 00:44:48,000
but we are the thing that paces what they can do.

74
00:44:49,240 --> 00:44:52,180
And in particular, human attention is, for me at least,

75
00:44:52,280 --> 00:44:55,880
the limiting factor, and I'm noticing this largely

76
00:44:55,880 --> 00:44:58,220
in how coding agents are deployed in teams,

77
00:44:58,720 --> 00:45:05,640
that this is a stable amount of attention that we have to work with.

78
00:45:07,360 --> 00:45:10,440
And starting new projects is getting incredibly easy with these agents.

79
00:45:10,640 --> 00:45:12,320
And I think particularly when a new model comes out,

80
00:45:12,400 --> 00:45:15,120
we are all super excited of trying the latest and greatest thing you can do

81
00:45:15,120 --> 00:45:18,640
from 3D games to new ways in which we automate things.

82
00:45:18,800 --> 00:45:21,740
But maintaining them has not really gotten easier.

83
00:45:22,920 --> 00:45:26,320
But we are also increasingly taxing a lot of external systems

84
00:45:26,320 --> 00:45:29,260
that actually in many ways are not supposed to be part of this revolution.

85
00:45:30,420 --> 00:45:35,620
I think we all know that GitHub has a lot more commits being pushed up there, a lot more repositories,

86
00:45:35,840 --> 00:45:40,860
but I know quite a few people who have well established old websites on the internet that

87
00:45:40,860 --> 00:45:45,220
found a new life for agents and they are totally not ready for the new volume that's coming from

88
00:45:45,220 --> 00:45:51,060
from us kind of directly hitting them. So I just want to caution a little bit here in saying that

89
00:45:51,060 --> 00:45:55,720
when we are pushing things we're also accidentally simultaneously all of us pushing things

90
00:45:56,400 --> 00:45:59,960
which we're not really ready for what agents are doing.

91
00:46:01,360 --> 00:46:04,020
And this to me I think is an interesting graph

92
00:46:04,020 --> 00:46:08,260
because the number of prompts that I'm actively pushing

93
00:46:08,260 --> 00:46:10,480
over the year is sort of flat.

94
00:46:10,840 --> 00:46:13,180
I know that in some months it's a little bit higher,

95
00:46:13,380 --> 00:46:14,480
in some months it's a little bit lower,

96
00:46:14,680 --> 00:46:19,420
but they're alternating between 120, 190 something like this prompts

97
00:46:19,420 --> 00:46:21,700
that I'm pushing per day.

98
00:46:22,820 --> 00:46:29,300
But you can see that the time that the agent was active for my use went from like less than my

99
00:46:29,300 --> 00:46:35,680
working day to many times my working day. And this November is not even over, right? So you

100
00:46:35,680 --> 00:46:41,020
can see that this is taking an exponential curve up. And this here is the number of projects,

101
00:46:41,180 --> 00:46:44,460
this is Git repositories that I've created over the last year. And this is like obviously

102
00:46:44,460 --> 00:46:49,340
cumulatively going up. So I have 141 different projects that I started. But the number of

103
00:46:49,340 --> 00:46:53,800
project I'm committing on a regular cadence to, which means I'm committing for more than

104
00:46:53,800 --> 00:47:00,960
five days to and I'm returning to them also, is pretty much flat with between 10, 15, something

105
00:47:00,960 --> 00:47:06,120
like this. And I have done this sort of informal survey with a bunch of my friends as well,

106
00:47:06,160 --> 00:47:10,540
and this seems to be a common trend that actually maintaining something regularly is tricky.

107
00:47:11,940 --> 00:47:16,440
And I had this discussion with a friend of mine in relation to this graph, and he said,

108
00:47:16,440 --> 00:47:19,360
like, well, actually, I have an open source project that is now fully automated.

109
00:47:20,540 --> 00:47:22,540
So there's an issue tracker coming in.

110
00:47:23,320 --> 00:47:26,140
There's an agent that's set up to auto-review, commit, and send it back.

111
00:47:26,420 --> 00:47:30,060
And that is maybe a way to scale up, so you take the human out of the whole thing.

112
00:47:30,200 --> 00:47:34,340
But actually, he also turned it off because eventually he didn't care about the project

113
00:47:34,340 --> 00:47:37,720
as much anymore, and he wanted his token specs to use them for other things.

114
00:47:39,020 --> 00:47:43,200
So our attention to an open source project matters even if maybe we have automated it

115
00:47:43,200 --> 00:47:43,520
entirely.

116
00:47:43,520 --> 00:47:49,480
So if we stop using something where we have a tendency to no longer actually actively take care of it.

117
00:47:51,480 --> 00:47:54,840
So if we're pushing towards, and I would make this argument,

118
00:47:55,060 --> 00:47:59,560
first figuring out how to make a really good workshop to build really high quality software,

119
00:47:59,780 --> 00:48:01,460
there's a question of what matters.

120
00:48:01,720 --> 00:48:05,040
And I think for me the three most important things to matter is on the one hand

121
00:48:05,040 --> 00:48:10,280
to really know when to push back against what an agent is doing,

122
00:48:10,560 --> 00:48:12,380
telling it that this is the wrong way of doing it.

123
00:48:12,380 --> 00:48:20,960
And I think that this is a crucial skill because an agent is really rewarded in just completing a goal.

124
00:48:21,440 --> 00:48:25,040
And it is very easy to look at if it has succeeded in this goal or not.

125
00:48:26,000 --> 00:48:32,180
And we can sort of see it in the same way as the RL environment maybe is rewarding the model during training.

126
00:48:32,720 --> 00:48:38,340
But that doesn't mean that what is being actually delivered is something that we are intrinsically happy with over a longer period of time.

127
00:48:38,340 --> 00:48:45,140
And this becomes tricky the more and more commits we're doing, the larger and larger the software is going.

128
00:48:45,220 --> 00:48:48,760
It's very easy to basically create the Simpson car of all the features in a software,

129
00:48:49,080 --> 00:48:56,620
but actually having something that intrinsically fits together where the internal complexity of the system does not explode

130
00:48:57,340 --> 00:49:03,000
still requires us to work with it as it's working.

131
00:49:03,620 --> 00:49:07,040
And I like to call this true skill versus agent skill.

132
00:49:08,640 --> 00:49:13,540
there is clearly a difference between someone who is an experienced software engineer working with an agent

133
00:49:13,540 --> 00:49:16,540
and someone who is not an experienced software engineer.

134
00:49:16,880 --> 00:49:24,080
And I think for me, the truest form of this is that my son is now sending me pull requests on a project that we're doing together.

135
00:49:24,260 --> 00:49:27,980
And there's a very clear quality difference in what his contributions are.

136
00:49:28,380 --> 00:49:30,540
Because he is not a software engineer.

137
00:49:31,280 --> 00:49:34,000
We're using the same system to prompt it, right?

138
00:49:35,660 --> 00:49:41,380
And I have noticed this myself in situations, but I've heard this also particularly from a lot of others.

139
00:49:41,540 --> 00:49:46,780
It is actually a very frustrating experience if you're in a situation that really matters

140
00:49:46,780 --> 00:49:51,000
and other people in the room are basically just going to the agent to answer a question

141
00:49:51,000 --> 00:49:55,500
and no longer have a really good understanding of either what the system is doing

142
00:49:55,500 --> 00:49:59,480
or that the fundamental problem that they're working on, they actually comprehend.

143
00:50:01,340 --> 00:50:04,900
So, there is a lot of value in still understanding what you are doing.

144
00:50:06,460 --> 00:50:12,720
That also means that you kind of have to moderate the speed in which you are doing things to still work within your understanding.

145
00:50:12,960 --> 00:50:17,980
And one of the ways in which I started doing this as my commits started going through the roof basically,

146
00:50:18,520 --> 00:50:22,540
was I looked into how are my trajectories of sessions going.

147
00:50:22,540 --> 00:50:29,160
And can I hook myself into a part before the end of it to find a consistent way for me

148
00:50:29,160 --> 00:50:30,280
to evaluate what's going on?

149
00:50:30,460 --> 00:50:35,360
And so for instance, I started basically prompting the agent that before it's going towards

150
00:50:35,360 --> 00:50:40,660
the verification, which is typically around 70% of it, to start producing a really nice,

151
00:50:40,820 --> 00:50:45,340
short executive summaries and a nice typeset HTML page that can really understand what

152
00:50:45,340 --> 00:50:47,020
it's doing as it's finishing.

153
00:50:48,040 --> 00:50:53,440
particularly with newer models, it's no longer quite as horrible to read the text that comes

154
00:50:53,440 --> 00:50:53,880
out of it.

155
00:50:54,220 --> 00:50:59,000
But this is one way in which I found to balance out that I'm basically no longer reading the

156
00:50:59,000 --> 00:50:59,960
code in a lot of cases.

157
00:51:01,860 --> 00:51:08,520
And this I think is something that increasingly is more important to me is to also show up

158
00:51:08,520 --> 00:51:15,860
with a little bit of a guidance towards not writing all the software over and over.

159
00:51:17,000 --> 00:51:21,340
I'm a huge believer in using fewer dependencies and writing more code yourself.

160
00:51:21,340 --> 00:51:26,500
So I really like the Archantic Revolution for that element of it.

161
00:51:27,540 --> 00:51:31,960
But in German, we have this word called Wildwuchs, which basically means uncontrolled growth.

162
00:51:32,800 --> 00:51:36,580
And uncontrolled growth in software that is supposed to work together

163
00:51:36,580 --> 00:51:38,440
can have a lot of really negative consequences.

164
00:51:39,760 --> 00:51:42,620
You might have seen a blog post that was going around recently

165
00:51:42,620 --> 00:51:46,620
about a TAR archive that has been crafted in a way

166
00:51:46,620 --> 00:51:49,800
that different TAR implementations read it in different ways.

167
00:51:51,400 --> 00:51:53,740
And I think this is a bit of a problem now.

168
00:51:55,420 --> 00:51:58,640
For us to work together, we really want to have standards.

169
00:51:58,880 --> 00:52:00,940
We want to have common specifications.

170
00:52:00,940 --> 00:52:02,440
We want to have common file formats.

171
00:52:02,980 --> 00:52:05,500
And we now can basically make an agent

172
00:52:05,500 --> 00:52:06,980
and implement any format we want.

173
00:52:07,200 --> 00:52:08,740
But there are going to be these differences.

174
00:52:09,620 --> 00:52:13,080
And what really makes it hard for us to agree on standards

175
00:52:13,080 --> 00:52:15,780
and what makes us work together on something

176
00:52:15,780 --> 00:52:17,720
often is not a technical challenge

177
00:52:17,720 --> 00:52:20,560
as much as it is a problem of how do people agree

178
00:52:20,560 --> 00:52:22,680
on what is supposed to be in this file format

179
00:52:22,680 --> 00:52:23,900
and how is it supposed to be implemented

180
00:52:23,900 --> 00:52:25,480
and how is it supposed to be interpreted

181
00:52:25,480 --> 00:52:27,680
because there can be multiple interpretations of it.

182
00:52:28,360 --> 00:52:30,240
And I really think that these open protocols

183
00:52:30,240 --> 00:52:32,300
and file formats are a really important aspect.

184
00:52:33,020 --> 00:52:34,680
You can imagine there is a future

185
00:52:34,680 --> 00:52:36,660
where we all have our own little Photoshop clones,

186
00:52:36,720 --> 00:52:39,300
but we'll no longer be able to send file formats to each other

187
00:52:39,300 --> 00:52:41,960
because we have diverged too much.

188
00:52:42,920 --> 00:52:44,840
And this really comes down to this part

189
00:52:44,840 --> 00:52:48,340
that collaboration between humans is as hard as it has ever been.

190
00:52:49,000 --> 00:52:52,260
And this is slowing us in a way down, right?

191
00:52:53,120 --> 00:52:56,420
And I think it's important to sometimes resist this temptation

192
00:52:56,420 --> 00:52:59,340
of just having the agent solve the problem

193
00:52:59,340 --> 00:53:01,140
and just taking a step back and figuring out

194
00:53:01,140 --> 00:53:02,860
if this is maybe one of those problems

195
00:53:02,860 --> 00:53:04,980
where we should be working together with other people.

196
00:53:05,760 --> 00:53:08,260
It is for me coming from an open source background

197
00:53:08,260 --> 00:53:11,620
fascinating to see how little actual human collaboration

198
00:53:11,620 --> 00:53:13,340
is happening in open source projects now

199
00:53:13,340 --> 00:53:16,960
and how many more projects end up with gazillions of forks

200
00:53:16,960 --> 00:53:20,640
because it's just easier to have a fork happen

201
00:53:20,640 --> 00:53:22,500
than to try to upstream changes.

202
00:53:24,180 --> 00:53:26,220
So I wanna make a very strong argument

203
00:53:26,220 --> 00:53:29,540
for building software that strives towards excellence.

204
00:53:29,860 --> 00:53:31,460
And I understand that this is hard.

205
00:53:32,100 --> 00:53:36,680
We have, I think, seen a pretty big explosion recently of verified software.

206
00:53:37,180 --> 00:53:39,740
It's also very clear that this is in itself hard.

207
00:53:40,020 --> 00:53:42,920
The technology stack is not right there yet where this works.

208
00:53:43,160 --> 00:53:46,480
But I think it is one of those areas where it's really worth pushing towards

209
00:53:46,480 --> 00:53:52,020
and seeing if this is maybe something that can help us over time creating more stable software.

210
00:53:53,400 --> 00:53:58,780
I really am very happy with seeing so many people wanting to build fundamentally faster

211
00:53:58,780 --> 00:54:00,180
and more memory efficient software.

212
00:54:00,180 --> 00:54:06,280
I think it's a great development. For years I was very frustrated with how

213
00:54:06,280 --> 00:54:10,540
little my RAM usage actually goes to things that I think are worthwhile and

214
00:54:10,540 --> 00:54:14,800
now there's a significant trend of people trying to rebuild things in a

215
00:54:14,800 --> 00:54:18,980
native language rather than for instance just building an electron wrapper. So all

216
00:54:18,980 --> 00:54:23,200
of those things are really cool developments but they are in many ways

217
00:54:23,200 --> 00:54:27,780
still building up on technology we have created before times. I really like Rust,

218
00:54:27,780 --> 00:54:31,340
I like that there is now a trend towards building memory-safe software.

219
00:54:31,580 --> 00:54:39,440
I think you've probably all been exposed to what happens with good models finding security issues.

220
00:54:39,540 --> 00:54:41,360
So there's a strong argument towards that.

221
00:54:42,180 --> 00:54:46,180
And it now has also led to people that would never have picked that language before

222
00:54:46,180 --> 00:54:51,120
to actually give it a try because it delivers some value that now can, in quote, come for free.

223
00:54:53,940 --> 00:54:57,900
On the last part, I want to talk a little bit about maybe there is something where Harness

224
00:54:57,900 --> 00:55:03,220
can do something. And this is less so much for me to say, hey, here's a path that we're on,

225
00:55:03,380 --> 00:55:07,900
is to give you some idea of maybe where some of the challenges are as we're pushing this task to

226
00:55:07,900 --> 00:55:14,300
be much more long running. The first one is, if you have a task that only takes two minutes to run,

227
00:55:14,300 --> 00:55:20,660
you're kind of willing to babysit it and if it breaks and it ends early or it

228
00:55:20,660 --> 00:55:24,240
doesn't run to completion or you have to restart the agent to just restart it

229
00:55:24,240 --> 00:55:28,040
manually and tell it to continue. Right so it's like we have we have largely been

230
00:55:28,040 --> 00:55:33,000
working in environment for quite a while but that was acceptable. If you now have

231
00:55:33,000 --> 00:55:38,560
automated agents running on thousands of issues overnight you want to have an

232
00:55:38,560 --> 00:55:43,060
agent actually be able to restore itself. And so we're pretty big believers on this

233
00:55:43,060 --> 00:55:46,860
concept of durable agents, also durable agents that create durable workflows

234
00:55:46,860 --> 00:55:50,640
where they orchestrate themselves in a way where you can start it, stop it,

235
00:55:50,760 --> 00:55:54,120
you can suspend it, you can bring it back. And this not just because we have this

236
00:55:54,120 --> 00:55:57,800
longer-running tasks but also because there's a lot of inefficiency in what

237
00:55:57,800 --> 00:56:04,680
we're doing and we are all compute constrained, we are all constrained in

238
00:56:04,680 --> 00:56:08,200
terms of the cost of inference and being able to be a little bit more

239
00:56:08,200 --> 00:56:11,460
intelligent about how we're doing this I think is really interesting and I wish

240
00:56:11,460 --> 00:56:15,520
more people would be pushing towards that. For instance, if you pay raw token costs, you get a

241
00:56:15,520 --> 00:56:20,160
pretty significant discount on flex scheduling. But that means that maybe for minutes, no token

242
00:56:20,160 --> 00:56:24,340
is coming, right? And so then you want to suspend your agent, you want to suspend also your sandbox

243
00:56:24,340 --> 00:56:30,640
and actually not waste the other resources that are also expensive. So this is one part where I do

244
00:56:30,640 --> 00:56:35,860
think beyond the quality of the models, because the models are getting so good now, we might want

245
00:56:35,860 --> 00:56:40,560
to rethink a little bit of how we are using these resources intelligently.

246
00:56:41,580 --> 00:56:50,320
And the second thing is you probably know that when Pi came out there was a pretty big

247
00:56:51,520 --> 00:56:55,480
motivation to say like bash is everything that you have to run and this is still largely

248
00:56:55,480 --> 00:56:56,100
what we believe.

249
00:56:56,920 --> 00:57:05,300
But now we also see that there is an emergence of desire to orchestrate sub-agents, a desire

250
00:57:05,300 --> 00:57:11,120
to use models which are not LLMs, like decision models or classifier models like Jeff.

251
00:57:11,520 --> 00:57:14,700
And in those cases, you need to start composing workflows.

252
00:57:16,060 --> 00:57:19,840
And this, I think, originally came from the folks from Cloudflare,

253
00:57:20,100 --> 00:57:25,860
say, like, instead of having thousands of tools being put into the code,

254
00:57:26,500 --> 00:57:30,900
into the context, what if we just give the agent, in addition to Bash,

255
00:57:30,900 --> 00:57:35,540
JavaScript, in addition to bash, JavaScript to run and have all the tool calls that the

256
00:57:35,540 --> 00:57:38,020
agent would natively do happen via JavaScript.

257
00:57:38,220 --> 00:57:39,500
Then he can orchestrate them better.

258
00:57:39,860 --> 00:57:44,380
And this has gone from a curiosity, I think, in many ways, to something that is fundamentally

259
00:57:44,800 --> 00:57:45,540
shockingly useful.

260
00:57:46,420 --> 00:57:50,420
And this means that now the harness in many cases, and this is definitely the case already

261
00:57:50,420 --> 00:57:54,520
in Codex, you can do it in Py and many other harnesses at this point, you can have the

262
00:57:54,520 --> 00:57:59,740
harness orchestrate very large invocations of tool calls without having to go through

263
00:57:59,740 --> 00:58:00,600
pretty inefficient context.

264
00:58:01,020 --> 00:58:03,540
And this does require a little bit of a rethinking

265
00:58:03,540 --> 00:58:04,840
of how you're doing things.

266
00:58:05,440 --> 00:58:08,460
And it also allows you to use classifier models.

267
00:58:08,560 --> 00:58:11,120
And I think this is the other big part

268
00:58:11,120 --> 00:58:13,020
that really changed how I was looking at

269
00:58:13,020 --> 00:58:14,960
how far I can push certain technology.

270
00:58:15,520 --> 00:58:17,980
Because you're not being penalized

271
00:58:17,980 --> 00:58:19,700
for not having an active KVCache,

272
00:58:20,460 --> 00:58:23,140
but they're also very cheap and very fast.

273
00:58:23,380 --> 00:58:25,920
And they allow you now to take certain problems

274
00:58:25,920 --> 00:58:29,220
that previously were just too expensive or too slow to do,

275
00:58:29,740 --> 00:58:31,300
to have a decision model do this.

276
00:58:31,760 --> 00:58:34,180
And the way I think you should be thinking on this

277
00:58:34,180 --> 00:58:35,780
is that when you have code mode,

278
00:58:35,980 --> 00:58:38,720
what the harness can do is it can create a little workflow

279
00:58:38,720 --> 00:58:41,120
that can have decision points along the way

280
00:58:41,120 --> 00:58:44,160
where if you, for instance, have to crunch through

281
00:58:44,160 --> 00:58:46,520
3,000 incoming customer support requests

282
00:58:47,040 --> 00:58:48,740
and you have to make decisions along the way

283
00:58:48,740 --> 00:58:52,000
if this is being sent towards having an agent work on it,

284
00:58:52,140 --> 00:58:54,240
towards requiring more feedback from a customer,

285
00:58:54,500 --> 00:58:56,820
you can have the agent create basically

286
00:58:56,820 --> 00:58:59,560
an on-demand workflow and then for every single

287
00:58:59,560 --> 00:59:01,760
if condition that it's doing, it's just going through Jeff.

288
00:59:01,980 --> 00:59:06,080
And then you can crunch through thousands of issues for very little time in no time in

289
00:59:06,080 --> 00:59:08,960
your coding agent, but previously that would have taken you hours.

290
00:59:10,440 --> 00:59:14,540
So with this, I just want to leave this idea.

291
00:59:15,020 --> 00:59:20,840
Now that we can build everything, maybe we should not, but we should really, I think,

292
00:59:20,920 --> 00:59:27,060
push towards much more enjoyable and in many ways much more ambitious software than we

293
00:59:27,060 --> 00:59:27,800
have been doing before.

294
00:59:27,800 --> 00:59:28,920
Thank you.