1
03:42:45,040 --> 03:42:47,590
All set.

2
03:42:47,600 --> 03:42:50,070
>> Hi everyone. Uh I'm Luke. I lead our

3
03:42:50,080 --> 03:42:52,150
core agent team here at Factory. Uh

4
03:42:52,160 --> 03:42:54,550
today I want to talk about autonomy and

5
03:42:54,560 --> 03:42:56,389
why uh the teams that are getting the

6
03:42:56,399 --> 03:42:57,750
most out of coding agents are not

7
03:42:57,760 --> 03:42:59,030
actually the ones that are handing over

8
03:42:59,040 --> 03:43:00,870
like the most work on day one. It's

9
03:43:00,880 --> 03:43:02,870
instead the folks who are ramping up

10
03:43:02,880 --> 03:43:05,030
autonomy gradually, one checkpoint at a

11
03:43:05,040 --> 03:43:06,710
time until what they have starts to

12
03:43:06,720 --> 03:43:09,590
resemble a software factory.

13
03:43:09,600 --> 03:43:11,349
So for those who don't know us, a

14
03:43:11,359 --> 03:43:13,189
factory is a platform uh that builds

15
03:43:13,199 --> 03:43:15,910
droid and then a a whole like surface

16
03:43:15,920 --> 03:43:17,510
for software engineering and building

17
03:43:17,520 --> 03:43:19,990
software on top of that coding agent.

18
03:43:20,000 --> 03:43:21,670
Hundreds of thousands of developers use

19
03:43:21,680 --> 03:43:23,429
us around the world at companies like

20
03:43:23,439 --> 03:43:26,870
Nvidia, T-Mobile, uh and Adobe. What

21
03:43:26,880 --> 03:43:28,469
these companies engineers are building

22
03:43:28,479 --> 03:43:30,790
with us is is really like a software

23
03:43:30,800 --> 03:43:33,189
factory, a system that runs 24/7 and

24
03:43:33,199 --> 03:43:34,870
builds software under the company's

25
03:43:34,880 --> 03:43:38,070
control. And that's a loop that takes in

26
03:43:38,080 --> 03:43:39,590
signals from wherever information is

27
03:43:39,600 --> 03:43:41,269
already available, turns them into

28
03:43:41,279 --> 03:43:43,670
changes, validates them, uh, and then

29
03:43:43,680 --> 03:43:45,670
learns from what production shows. A

30
03:43:45,680 --> 03:43:47,110
signal is really like anything that

31
03:43:47,120 --> 03:43:48,790
tells you how your software should

32
03:43:48,800 --> 03:43:50,950
evolve. So that could be, you know, a

33
03:43:50,960 --> 03:43:53,590
sentry alert or someone's idea on Slack.

34
03:43:53,600 --> 03:43:54,870
Every signal goes through the same

35
03:43:54,880 --> 03:43:56,950
stages. So you standardize you

36
03:43:56,960 --> 03:43:58,870
standardize the the process not the

37
03:43:58,880 --> 03:44:00,630
output.

38
03:44:00,640 --> 03:44:02,550
So we built that platform on five

39
03:44:02,560 --> 03:44:05,110
foundations. It has to be transparent,

40
03:44:05,120 --> 03:44:07,990
modular, governable, efficient and

41
03:44:08,000 --> 03:44:10,550
frontier.

42
03:44:10,560 --> 03:44:12,469
Most people picture a software factory

43
03:44:12,479 --> 03:44:14,389
as this kind of like lights out building

44
03:44:14,399 --> 03:44:16,870
where uh there's no people inside of it.

45
03:44:16,880 --> 03:44:18,870
Requirements come in one end and merged

46
03:44:18,880 --> 03:44:21,510
PRs come out the other. And and really

47
03:44:21,520 --> 03:44:23,269
what we saw firsthand in the field is

48
03:44:23,279 --> 03:44:24,950
that that's not the case. The best ones

49
03:44:24,960 --> 03:44:26,870
aren't set up like that. They're built

50
03:44:26,880 --> 03:44:29,429
around people instead. Uh, humans decide

51
03:44:29,439 --> 03:44:31,189
what matters and how much to trust

52
03:44:31,199 --> 03:44:33,269
droids over time. And those droids earn

53
03:44:33,279 --> 03:44:34,870
their keep as they deliver more work

54
03:44:34,880 --> 03:44:36,950
that's higher quality.

55
03:44:36,960 --> 03:44:38,469
Every company's software factory looks

56
03:44:38,479 --> 03:44:39,990
very different to each other. So, what

57
03:44:40,000 --> 03:44:41,590
we what we sell is not really this

58
03:44:41,600 --> 03:44:43,510
factory, it's the tools and the building

59
03:44:43,520 --> 03:44:46,389
blocks to compose them.

60
03:44:46,399 --> 03:44:48,070
Now, it's it's worth kind of looking at

61
03:44:48,080 --> 03:44:50,150
how we got here in the first place. Back

62
03:44:50,160 --> 03:44:51,750
in April, one of our flagship products

63
03:44:51,760 --> 03:44:53,990
was Missions, which was one of like the

64
03:44:54,000 --> 03:44:56,630
first multi- aent uh products to really

65
03:44:56,640 --> 03:44:58,150
take like a bite out of the human

66
03:44:58,160 --> 03:45:00,389
attention bottleneck that existed in in

67
03:45:00,399 --> 03:45:02,870
software engineering. You'd describe a

68
03:45:02,880 --> 03:45:04,710
goal, you'd like argue with the

69
03:45:04,720 --> 03:45:06,550
orchestrator until you were aligned,

70
03:45:06,560 --> 03:45:08,309
you'd approve the plan, and then it

71
03:45:08,319 --> 03:45:10,309
would go handle implementation for like

72
03:45:10,319 --> 03:45:12,630
days or even weeks. When the goal was

73
03:45:12,640 --> 03:45:14,630
clear and verifiable, you really could

74
03:45:14,640 --> 03:45:16,469
come back to like completed work pretty

75
03:45:16,479 --> 03:45:18,070
consistently.

76
03:45:18,080 --> 03:45:20,229
But as more teams adopted it, we heard

77
03:45:20,239 --> 03:45:22,630
some consistent like pieces of feedback.

78
03:45:22,640 --> 03:45:24,309
The first was how missions asked you to

79
03:45:24,319 --> 03:45:26,150
work. The product assumed that you could

80
03:45:26,160 --> 03:45:27,670
plan everything up front and then it

81
03:45:27,680 --> 03:45:29,830
would just work. Uh and the UX sort of

82
03:45:29,840 --> 03:45:31,110
suggested that you shouldn't interrupt

83
03:45:31,120 --> 03:45:32,710
them. Missions didn't really like ask

84
03:45:32,720 --> 03:45:34,710
you any questions or provide like an

85
03:45:34,720 --> 03:45:36,950
interactive surface for that uh

86
03:45:36,960 --> 03:45:38,630
communication between you and the agent

87
03:45:38,640 --> 03:45:40,790
once it was starting implementation. So

88
03:45:40,800 --> 03:45:43,269
this suits like migrations and big uh

89
03:45:43,279 --> 03:45:45,429
big coding like feature projects but it

90
03:45:45,439 --> 03:45:47,110
didn't really fit fit the bill for like

91
03:45:47,120 --> 03:45:49,750
most kinds of other work.

92
03:45:49,760 --> 03:45:51,990
The second was cost and time and that

93
03:45:52,000 --> 03:45:54,150
came from honestly like the models

94
03:45:54,160 --> 03:45:55,990
getting really really good. I'm sure

95
03:45:56,000 --> 03:45:58,229
you've all seen now adversarial review

96
03:45:58,239 --> 03:46:00,070
is really really good at finding these

97
03:46:00,080 --> 03:46:02,790
issues in your code. And so, uh, at the

98
03:46:02,800 --> 03:46:05,030
time we were adding adversarial review

99
03:46:05,040 --> 03:46:07,590
into every milestone's boundary. That

100
03:46:07,600 --> 03:46:09,830
just kind of like grew the scope of that

101
03:46:09,840 --> 03:46:12,469
mission from something like 60 features

102
03:46:12,479 --> 03:46:15,910
to over 200. And so you ended up having

103
03:46:15,920 --> 03:46:17,750
a token bill and a timeline that matched

104
03:46:17,760 --> 03:46:19,990
that. And then the last one was that

105
03:46:20,000 --> 03:46:21,670
missions were keeping their context to

106
03:46:21,680 --> 03:46:24,070
one person. So that was fine for like a

107
03:46:24,080 --> 03:46:25,670
single project, but it meant that the

108
03:46:25,680 --> 03:46:27,110
rest of the team couldn't really build

109
03:46:27,120 --> 03:46:30,229
on what that mission had learned.

110
03:46:30,239 --> 03:46:32,309
So we went back to the drawing board and

111
03:46:32,319 --> 03:46:34,710
one mental model kept on coming up. When

112
03:46:34,720 --> 03:46:36,630
you have like an org that is running

113
03:46:36,640 --> 03:46:39,510
dozens or hundreds of agents at a given

114
03:46:39,520 --> 03:46:41,670
point in time, you actually have like a

115
03:46:41,680 --> 03:46:44,150
lot of the the problems that operating

116
03:46:44,160 --> 03:46:46,710
systems back in the day solved. How to

117
03:46:46,720 --> 03:46:48,790
schedule work, how to run what many

118
03:46:48,800 --> 03:46:50,790
things at once safely, and how to share

119
03:46:50,800 --> 03:46:53,349
state.

120
03:46:53,359 --> 03:46:55,510
So every operating system starts with

121
03:46:55,520 --> 03:46:57,110
the thing that actually does the work

122
03:46:57,120 --> 03:46:58,710
and we wanted to revisit the agent

123
03:46:58,720 --> 03:47:01,510
itself. Long horizon agents run for

124
03:47:01,520 --> 03:47:03,269
thousands of turns and every

125
03:47:03,279 --> 03:47:05,110
inefficiency across of those gets

126
03:47:05,120 --> 03:47:07,030
multiplied. So you could have, you know,

127
03:47:07,040 --> 03:47:08,389
an extra inference request that's

128
03:47:08,399 --> 03:47:11,590
unnecessary or a a tool result that's

129
03:47:11,600 --> 03:47:13,670
pretty bloated or a prompt cache miss.

130
03:47:13,680 --> 03:47:15,990
All those things compound over time and

131
03:47:16,000 --> 03:47:17,990
lead to waste.

132
03:47:18,000 --> 03:47:19,349
So we decided to make some pretty

133
03:47:19,359 --> 03:47:20,710
significant changes to the harness

134
03:47:20,720 --> 03:47:23,349
itself. The biggest one was programmatic

135
03:47:23,359 --> 03:47:25,750
tool calling, also known as code mode.

136
03:47:25,760 --> 03:47:27,990
Instead of calling one tool, waiting on

137
03:47:28,000 --> 03:47:29,750
the model, and then calling the next, we

138
03:47:29,760 --> 03:47:31,510
have the model write a small program

139
03:47:31,520 --> 03:47:33,830
that executes all those tools in one go,

140
03:47:33,840 --> 03:47:35,269
and then prints out only what's

141
03:47:35,279 --> 03:47:36,790
relevant.

142
03:47:36,800 --> 03:47:38,550
So there's this example here where you

143
03:47:38,560 --> 03:47:40,469
have 150 linear tickets that are getting

144
03:47:40,479 --> 03:47:43,030
moved from one project to another. done

145
03:47:43,040 --> 03:47:44,950
the normal way that's like 30 inference

146
03:47:44,960 --> 03:47:47,590
requests, a million tokens, but as one

147
03:47:47,600 --> 03:47:49,510
model generated program, you actually

148
03:47:49,520 --> 03:47:51,830
only have like 60k tokens. It runs much

149
03:47:51,840 --> 03:47:53,670
faster and the model only sees what is

150
03:47:53,680 --> 03:47:55,830
actually relevant to the task that the

151
03:47:55,840 --> 03:47:57,189
the fact whether it was like successful

152
03:47:57,199 --> 03:47:59,189
or not.

153
03:47:59,199 --> 03:48:00,950
So this layer of the OS is where being

154
03:48:00,960 --> 03:48:03,030
frontier and efficient matters the most.

155
03:48:03,040 --> 03:48:05,349
A new model lands every few weeks and no

156
03:48:05,359 --> 03:48:07,750
two fail fail the same way. So we tune

157
03:48:07,760 --> 03:48:09,830
our harness in such a way that evens out

158
03:48:09,840 --> 03:48:11,590
this jaggedness across all these

159
03:48:11,600 --> 03:48:13,189
different models. The result is that

160
03:48:13,199 --> 03:48:15,189
Droid is now, you know, higher quality

161
03:48:15,199 --> 03:48:17,110
at a lower cost than every other coding

162
03:48:17,120 --> 03:48:19,990
agent on the market.

163
03:48:20,000 --> 03:48:21,910
Now you have a faster agent, but that

164
03:48:21,920 --> 03:48:23,269
that doesn't really give you like a step

165
03:48:23,279 --> 03:48:25,349
function increase in in how much work

166
03:48:25,359 --> 03:48:27,510
gets done inside of your system. For

167
03:48:27,520 --> 03:48:29,030
that, you need many agents running at

168
03:48:29,040 --> 03:48:31,750
once, and they need somewhere to run.

169
03:48:31,760 --> 03:48:33,510
remote missions were kind of held back

170
03:48:33,520 --> 03:48:35,269
that by this like fundamental limitation

171
03:48:35,279 --> 03:48:37,189
in the system where there was one

172
03:48:37,199 --> 03:48:39,750
computer that handled all the work. But

173
03:48:39,760 --> 03:48:41,750
to run a software factory, you need uh

174
03:48:41,760 --> 03:48:43,830
compute that scales with the volume of

175
03:48:43,840 --> 03:48:45,510
work in the system. Otherwise, every

176
03:48:45,520 --> 03:48:47,590
agent is is fighting for the same files

177
03:48:47,600 --> 03:48:49,990
or system resources. And so that's why

178
03:48:50,000 --> 03:48:52,150
we build environments which are reusable

179
03:48:52,160 --> 03:48:53,910
definitions of the sandboxes that your

180
03:48:53,920 --> 03:48:55,990
droids need, complete with the repos,

181
03:48:56,000 --> 03:48:58,309
tools, and dependencies that they that

182
03:48:58,319 --> 03:49:00,229
they need to get started with. The

183
03:49:00,239 --> 03:49:02,150
system is governable. So there's one

184
03:49:02,160 --> 03:49:04,550
comp control plane that orchestrates all

185
03:49:04,560 --> 03:49:06,710
of these sandboxes together wherever you

186
03:49:06,720 --> 03:49:08,870
need them. They can run inside of your

187
03:49:08,880 --> 03:49:12,550
own VPC on on EC2, Kubernetes, whatever

188
03:49:12,560 --> 03:49:14,229
compute you already operate. So your

189
03:49:14,239 --> 03:49:16,150
code, your data and your network are in

190
03:49:16,160 --> 03:49:19,110
your control. Compute becomes pluggable

191
03:49:19,120 --> 03:49:21,670
just like we made the BY options and

192
03:49:21,680 --> 03:49:24,229
automodel uh options that we provide uh

193
03:49:24,239 --> 03:49:25,830
make that provide sort of like this like

194
03:49:25,840 --> 03:49:28,870
model pluggable layer. So now we had

195
03:49:28,880 --> 03:49:30,710
agents that were cheap enough to run all

196
03:49:30,720 --> 03:49:33,349
the time and the compute to run many at

197
03:49:33,359 --> 03:49:35,750
once. And so then the next question was

198
03:49:35,760 --> 03:49:38,389
how do we orchestrate all of this work?

199
03:49:38,399 --> 03:49:40,790
At first we tried making like this DAG

200
03:49:40,800 --> 03:49:43,510
very complex system of agents composed

201
03:49:43,520 --> 03:49:45,429
together by like these very like strong

202
03:49:45,439 --> 03:49:47,830
rules and it didn't really work. You end

203
03:49:47,840 --> 03:49:49,189
up having a lot of the same problems

204
03:49:49,199 --> 03:49:51,429
that plagued missions uh in the first

205
03:49:51,439 --> 03:49:53,910
place because it was too rigid and as a

206
03:49:53,920 --> 03:49:55,510
result it was too expensive to run and

207
03:49:55,520 --> 03:49:57,910
it didn't generalize well.

208
03:49:57,920 --> 03:49:59,670
So what we landed on instead is

209
03:49:59,680 --> 03:50:01,510
something that we call a workstream.

210
03:50:01,520 --> 03:50:03,269
It's a standing goal with a team of

211
03:50:03,279 --> 03:50:05,670
agents to support it. It has this goal.

212
03:50:05,680 --> 03:50:07,910
It has an identity, a an environment

213
03:50:07,920 --> 03:50:10,710
that it runs on and memory.

214
03:50:10,720 --> 03:50:13,349
A new workstream runs as you and it has

215
03:50:13,359 --> 03:50:15,670
your access. When it outgrows one

216
03:50:15,680 --> 03:50:17,349
person, it can be moved to a service

217
03:50:17,359 --> 03:50:19,510
account with its own credentials. So you

218
03:50:19,520 --> 03:50:21,269
stay on as a member, but the work keeps

219
03:50:21,279 --> 03:50:23,510
h happening under an identity that the

220
03:50:23,520 --> 03:50:26,550
the org actually controls.

221
03:50:26,560 --> 03:50:28,710
At the center of every workstream is an

222
03:50:28,720 --> 03:50:30,630
orchestrator and its job looks a lot

223
03:50:30,640 --> 03:50:33,189
like a scheduler in an OS. Unlike the

224
03:50:33,199 --> 03:50:35,429
graph that we started with, it assembles

225
03:50:35,439 --> 03:50:37,110
its agents in whatever way is most

226
03:50:37,120 --> 03:50:38,870
natural for the model. So the

227
03:50:38,880 --> 03:50:40,309
instructions are pretty short, right?

228
03:50:40,319 --> 03:50:43,030
It's like uh delegate first. Give each

229
03:50:43,040 --> 03:50:45,349
agent an area of ownership and then for

230
03:50:45,359 --> 03:50:47,269
tasks that can fall into an existing

231
03:50:47,279 --> 03:50:50,150
area of ownership, delegate to those.

232
03:50:50,160 --> 03:50:52,469
You start by talking to it. uh you just

233
03:50:52,479 --> 03:50:54,229
kind of hand things over as you think of

234
03:50:54,239 --> 03:50:56,550
them and then it fans out the work so

235
03:50:56,560 --> 03:50:58,630
that then uh you can get the the control

236
03:50:58,640 --> 03:51:01,429
back to you. Uh agents report back when

237
03:51:01,439 --> 03:51:03,510
they finish, when they get blocked, when

238
03:51:03,520 --> 03:51:04,790
there's something that changes the plan

239
03:51:04,800 --> 03:51:07,189
overall.

240
03:51:07,199 --> 03:51:08,550
But the work stream doesn't just stop

241
03:51:08,560 --> 03:51:11,030
there because it takes on more on its

242
03:51:11,040 --> 03:51:13,269
own and it learns from your habits. At

243
03:51:13,279 --> 03:51:14,870
some point the orchestrator may ask you

244
03:51:14,880 --> 03:51:16,710
how you decide what to work on and it

245
03:51:16,720 --> 03:51:18,309
learns that maybe bugs are coming from

246
03:51:18,319 --> 03:51:21,349
two Slack channels.

247
03:51:21,359 --> 03:51:23,189
So here it can set up an intake and a

248
03:51:23,199 --> 03:51:25,910
triage listener. Every 30 minutes it can

249
03:51:25,920 --> 03:51:27,830
go in and pull in those reports from

250
03:51:27,840 --> 03:51:30,070
those channels, ddup them against you

251
03:51:30,080 --> 03:51:33,030
know existing linear issues or uh sentry

252
03:51:33,040 --> 03:51:34,790
sentry alerts that already exist and

253
03:51:34,800 --> 03:51:36,309
then weigh each of those against the

254
03:51:36,319 --> 03:51:39,269
workstream's goal. So a Slack complaint

255
03:51:39,279 --> 03:51:41,910
and a customer error report become one

256
03:51:41,920 --> 03:51:45,429
change. The output of triage is a spec

257
03:51:45,439 --> 03:51:48,870
that you review and uh you're capable of

258
03:51:48,880 --> 03:51:50,950
like weighing in on before any code gets

259
03:51:50,960 --> 03:51:53,429
written. Once you approve a spec, the

260
03:51:53,439 --> 03:51:55,990
same worker opens a PR, stewards it

261
03:51:56,000 --> 03:51:58,309
until it's merge ready uh and then make

262
03:51:58,319 --> 03:52:00,469
sure that you know review uh comments

263
03:52:00,479 --> 03:52:03,590
and CI are passing.

264
03:52:03,600 --> 03:52:05,189
So each one of those stages actually

265
03:52:05,199 --> 03:52:07,110
maps to an autonomy level in this

266
03:52:07,120 --> 03:52:10,309
system. At L1, the workstream posits for

267
03:52:10,319 --> 03:52:13,349
human review at the spec creation, at

268
03:52:13,359 --> 03:52:17,030
the PR, and before hitting merge. But

269
03:52:17,040 --> 03:52:19,269
you know, as you level up that th those

270
03:52:19,279 --> 03:52:21,750
levels of autonomy, you can get to L4

271
03:52:21,760 --> 03:52:23,590
where you're automerging everything by

272
03:52:23,600 --> 03:52:25,670
default. The owner gets to set the

273
03:52:25,680 --> 03:52:27,910
level, but the confidence to raise it

274
03:52:27,920 --> 03:52:30,229
comes from watching the work. So when

275
03:52:30,239 --> 03:52:33,030
proposed changes are accurate or PRs

276
03:52:33,040 --> 03:52:34,550
consistently merge at the quality that

277
03:52:34,560 --> 03:52:36,389
you'd expect, that's actually your cue

278
03:52:36,399 --> 03:52:38,550
to move to the next checkpoint. And so

279
03:52:38,560 --> 03:52:40,550
that becomes the dial for you. You don't

280
03:52:40,560 --> 03:52:42,070
need to decide on day one whether you

281
03:52:42,080 --> 03:52:43,910
trust a coding agent to own, you know,

282
03:52:43,920 --> 03:52:46,389
your like CI health workstream off the

283
03:52:46,399 --> 03:52:48,950
bat. You just need to trust it to to

284
03:52:48,960 --> 03:52:52,630
like unlock the next checkpoint.

285
03:52:52,640 --> 03:52:54,870
So in a factory, an agent that starts

286
03:52:54,880 --> 03:52:56,950
from scratch repeats the same mistakes

287
03:52:56,960 --> 03:52:58,790
the last one already did. And so we

288
03:52:58,800 --> 03:53:01,030
built workstreams to have a shared file

289
03:53:01,040 --> 03:53:04,469
system. Agents are are encouraged to

290
03:53:04,479 --> 03:53:07,189
keep memory and version skills and so

291
03:53:07,199 --> 03:53:09,670
their PRs get better over time. They're

292
03:53:09,680 --> 03:53:11,429
also proactive and they go looking for

293
03:53:11,439 --> 03:53:13,590
context that they're missing. So for

294
03:53:13,600 --> 03:53:15,670
example, say that your work notices that

295
03:53:15,680 --> 03:53:17,429
other people are shipping related

296
03:53:17,439 --> 03:53:19,830
changes. It sets up an automation that

297
03:53:19,840 --> 03:53:21,990
pulls in their PRs,

298
03:53:22,000 --> 03:53:23,830
analyzes what they've been working on

299
03:53:23,840 --> 03:53:25,269
and what comments they've been receiving

300
03:53:25,279 --> 03:53:27,510
in those PRs, and then incorporates them

301
03:53:27,520 --> 03:53:30,309
into their memory and skills. So, what

302
03:53:30,319 --> 03:53:32,469
one agent learns, others can use, and

303
03:53:32,479 --> 03:53:36,070
nothing starts cold.

304
03:53:36,080 --> 03:53:38,469
That folder is also how agents work

305
03:53:38,479 --> 03:53:40,309
together on a project. When the

306
03:53:40,319 --> 03:53:42,150
orchestrator spawns an agent, it's

307
03:53:42,160 --> 03:53:44,710
capable of telling, you know, each agent

308
03:53:44,720 --> 03:53:46,469
where does it where where does its

309
03:53:46,479 --> 03:53:48,070
result sort of belong inside of this

310
03:53:48,080 --> 03:53:51,830
shared folder. And so every agent can

311
03:53:51,840 --> 03:53:54,389
read and write to the same place. If you

312
03:53:54,399 --> 03:53:56,950
ask it a workstream why checkout got

313
03:53:56,960 --> 03:53:59,189
slower, for example, it can send dozens

314
03:53:59,199 --> 03:54:01,269
of agents out to track down all those

315
03:54:01,279 --> 03:54:02,950
different leads that you have. And then

316
03:54:02,960 --> 03:54:05,030
ultimately each one can write its

317
03:54:05,040 --> 03:54:06,229
findings to the folder and the

318
03:54:06,239 --> 03:54:09,189
orchestrator can merge them.

319
03:54:09,199 --> 03:54:10,710
Now you still need a barrier between

320
03:54:10,720 --> 03:54:13,269
implementation and merging. And for us

321
03:54:13,279 --> 03:54:14,710
that barrier is platform level

322
03:54:14,720 --> 03:54:17,750
validation. Every PR gets code review,

323
03:54:17,760 --> 03:54:20,229
security review, and dedicated QA

324
03:54:20,239 --> 03:54:22,550
against the live preview. And so once

325
03:54:22,560 --> 03:54:24,870
every review thread is resolved, a risk

326
03:54:24,880 --> 03:54:26,630
judge then goes in and decides whether

327
03:54:26,640 --> 03:54:28,950
it's safe to approve or not. The judge

328
03:54:28,960 --> 03:54:30,870
gets the exact if and a check out of the

329
03:54:30,880 --> 03:54:32,950
branch so it can like build and test it.

330
03:54:32,960 --> 03:54:34,790
Uh but ultimately it reads its policy

331
03:54:34,800 --> 03:54:36,710
from our mainline branch so that the

332
03:54:36,720 --> 03:54:38,790
change can't rewrite the rules that it's

333
03:54:38,800 --> 03:54:40,870
being judged by.

334
03:54:40,880 --> 03:54:43,110
Every week a droid minds the review

335
03:54:43,120 --> 03:54:45,110
comments and goes through figures out

336
03:54:45,120 --> 03:54:47,189
which one authors have actually acted on

337
03:54:47,199 --> 03:54:49,670
and then pulls them pulls those into

338
03:54:49,680 --> 03:54:51,830
like review guidelines so that the the

339
03:54:51,840 --> 03:54:54,790
loop sort of improves over time. We also

340
03:54:54,800 --> 03:54:56,870
have droids that every day run and

341
03:54:56,880 --> 03:54:59,510
analyze reverts, hot fixes, new sentry

342
03:54:59,520 --> 03:55:01,269
errors, for example, and traces them

343
03:55:01,279 --> 03:55:03,189
back to the PRs that that caused them.

344
03:55:03,199 --> 03:55:05,030
And so that also is a signal into this

345
03:55:05,040 --> 03:55:07,590
same loop.

346
03:55:07,600 --> 03:55:09,590
What we're really hell climbing on here

347
03:55:09,600 --> 03:55:12,070
is the regretfree merge rate, the share

348
03:55:12,080 --> 03:55:14,630
of changes that merge and never need a

349
03:55:14,640 --> 03:55:17,269
revert or a follow-up fix. We measure it

350
03:55:17,279 --> 03:55:19,269
for droid ridden and humanridden PRs

351
03:55:19,279 --> 03:55:21,110
side by side. And it's the same way that

352
03:55:21,120 --> 03:55:22,950
we let like self-driving cars onto the

353
03:55:22,960 --> 03:55:24,790
road. We didn't just give them the green

354
03:55:24,800 --> 03:55:27,830
light. We only uh did so once they

355
03:55:27,840 --> 03:55:29,670
crashed less often than human-driven

356
03:55:29,680 --> 03:55:32,309
ones did. And so once AI written changes

357
03:55:32,319 --> 03:55:34,309
break things less often than human ones

358
03:55:34,319 --> 03:55:36,469
and you can actually measure that then

359
03:55:36,479 --> 03:55:38,469
you can let them merge and that's the

360
03:55:38,479 --> 03:55:41,670
bar for turning a workstream to L4.

361
03:55:41,680 --> 03:55:43,429
Workstream is now like the platform that

362
03:55:43,439 --> 03:55:45,510
we build new features on top of. So

363
03:55:45,520 --> 03:55:47,510
migrate is the first one. It sort of

364
03:55:47,520 --> 03:55:50,550
fulfills a niche in our product um like

365
03:55:50,560 --> 03:55:53,110
framework for for what missions used to

366
03:55:53,120 --> 03:55:55,349
be this like dedicated oneshot migration

367
03:55:55,359 --> 03:55:57,750
experience and continuous remediation is

368
03:55:57,760 --> 03:55:59,590
the second where it follows that

369
03:55:59,600 --> 03:56:01,110
circular shape that workstreams are best

370
03:56:01,120 --> 03:56:02,950
suited for because there the work never

371
03:56:02,960 --> 03:56:06,070
ends.

372
03:56:06,080 --> 03:56:08,710
We run factory uh on top of workstreams

373
03:56:08,720 --> 03:56:10,389
ourselves and so this is what it

374
03:56:10,399 --> 03:56:12,389
actually looks like in production. In a

375
03:56:12,399 --> 03:56:14,229
weekend this past October, we shipped

376
03:56:14,239 --> 03:56:16,389
about 800 poll requests in our dev

377
03:56:16,399 --> 03:56:19,189
branch and Droid wrote 99% of them and

378
03:56:19,199 --> 03:56:21,510
reviewed all of them. Now, work landed

379
03:56:21,520 --> 03:56:23,349
in one of two ways. It was either a

380
03:56:23,359 --> 03:56:26,229
human picking up a signal manually uh or

381
03:56:26,239 --> 03:56:28,389
increasingly nowadays, it's a workstream

382
03:56:28,399 --> 03:56:31,110
picking it up uh automatically triaging

383
03:56:31,120 --> 03:56:33,269
it against the goal of the workstream

384
03:56:33,279 --> 03:56:35,269
and then ultimately writing a spec and

385
03:56:35,279 --> 03:56:37,110
merging it.

386
03:56:37,120 --> 03:56:40,070
52% of PRs that we ship right now are

387
03:56:40,080 --> 03:56:41,910
coming from workshops themselves and

388
03:56:41,920 --> 03:56:43,269
some of these are operating on full

389
03:56:43,279 --> 03:56:46,229
autonomy. So for example, the one that

390
03:56:46,239 --> 03:56:48,070
uh fixes bug reports from a dedicated

391
03:56:48,080 --> 03:56:50,950
Slack channel merges 53 PRs per week.

392
03:56:50,960 --> 03:56:52,710
Another one is our flaky test work

393
03:56:52,720 --> 03:56:56,630
stream which merges about 40.

394
03:56:56,640 --> 03:56:59,030
Fully autonomous PRs aren't really the

395
03:56:59,040 --> 03:57:00,950
only thing worth tracking though. It

396
03:57:00,960 --> 03:57:02,229
what matters a lot more is the

397
03:57:02,239 --> 03:57:04,309
progression. how much of your work flows

398
03:57:04,319 --> 03:57:05,670
through work streams and how many

399
03:57:05,680 --> 03:57:08,229
checkpoints does each one still need. As

400
03:57:08,239 --> 03:57:10,070
more work flows unrestricted, your

401
03:57:10,080 --> 03:57:12,070
engineers become the regulators of this

402
03:57:12,080 --> 03:57:13,990
system. They decide, you know, what

403
03:57:14,000 --> 03:57:16,229
signals to watch, uh what are the goals

404
03:57:16,239 --> 03:57:17,990
and the owners of those, what are the

405
03:57:18,000 --> 03:57:19,349
skills and review guidelines that you

406
03:57:19,359 --> 03:57:21,110
need in order to keep this code high

407
03:57:21,120 --> 03:57:22,950
quality and how much autonomy each

408
03:57:22,960 --> 03:57:24,950
workstream gets.

409
03:57:24,960 --> 03:57:26,550
And so there's just one question that

410
03:57:26,560 --> 03:57:28,469
tells you, you know, whether whether you

411
03:57:28,479 --> 03:57:30,229
have a software factory or faster

412
03:57:30,239 --> 03:57:32,550
engineers, and that is, can you name and

413
03:57:32,560 --> 03:57:34,870
track your signals? Every team already

414
03:57:34,880 --> 03:57:37,429
has them. You have bug reports in Slack,

415
03:57:37,439 --> 03:57:40,070
you know, failing CI jobs, error spikes,

416
03:57:40,080 --> 03:57:42,150
tickets that nobody ever picks up, but

417
03:57:42,160 --> 03:57:44,630
most of them wait on a person to notice.

418
03:57:44,640 --> 03:57:47,030
And so my ask to all of you is to write

419
03:57:47,040 --> 03:57:48,790
them down and then pick the one that you

420
03:57:48,800 --> 03:57:51,189
waste the most time on. give it to

421
03:57:51,199 --> 03:57:53,269
workstream on L1 and then set up our

422
03:57:53,279 --> 03:57:54,950
validation experience and then you're

423
03:57:54,960 --> 03:57:56,870
well on your way to building software

424
03:57:56,880 --> 03:57:58,550
that improves itself with all the

425
03:57:58,560 --> 03:58:01,269
signals that you choose to track. Thanks

426
03:58:01,279 --> 03:58:02,550
everyone and I'll be around to answer

427
03:58:02,560 --> 03:58:05,000
any questions today. [applause]