1
03:23:23,720 --> 03:23:24,840
Well, good afternoon, folks.

2
03:23:24,840 --> 03:23:27,700
Wow, that's loud. You guys hear me okay? I take it yes.

3
03:23:28,460 --> 03:23:30,660
I'm going to start with a question here today.

4
03:23:30,940 --> 03:23:35,620
How many people feel like they're getting everything that they could possibly get from their agents today?

5
03:23:36,180 --> 03:23:40,980
And that the promise of AI for your team has been delivered and you're satisfied with where you are?

6
03:23:41,340 --> 03:23:41,920
Show of hands.

7
03:23:43,320 --> 03:23:46,320
This is about what I usually see. Like, there's very few people.

8
03:23:48,900 --> 03:23:51,560
And for us at Amazon, it's actually the very same.

9
03:23:51,560 --> 03:23:56,440
Like when we first started this process, what happened is we were seeing between 10 and 20% gains,

10
03:23:56,700 --> 03:24:00,320
as I show on the chart over here, you know, 1.1x to 1.2x.

11
03:24:00,580 --> 03:24:06,120
And what we noticed was what was happening is teams were bolting AI onto a process that they already had.

12
03:24:06,420 --> 03:24:08,440
So they're taking their normal process that they were doing,

13
03:24:08,620 --> 03:24:11,280
they're trying to attach some AI here, attach some AI there,

14
03:24:11,340 --> 03:24:13,740
and expecting that they're going to get this massive result out of it.

15
03:24:14,700 --> 03:24:18,620
In addition, what we saw was teams were only shrinking the code generation phase.

16
03:24:18,880 --> 03:24:21,120
And as you all know, you guys are all experts in this,

17
03:24:21,120 --> 03:24:25,100
The code generation phase has gotten better and better and better as time has gone on.

18
03:24:25,320 --> 03:24:27,980
But there's a lot more to software development than just code generation.

19
03:24:29,420 --> 03:24:31,220
We did a study in June.

20
03:24:31,460 --> 03:24:35,100
I published a blog post on it as well.

21
03:24:35,580 --> 03:24:40,080
And in the study, we looked at teams who changed how they worked in our Amazon Stores division.

22
03:24:40,260 --> 03:24:43,780
And we looked at what kind of returns were they getting on their AI investment.

23
03:24:44,000 --> 03:24:47,040
Many of these teams were going slower before they were able to go faster.

24
03:24:47,380 --> 03:24:50,520
But the median team was seeing a 4.5x return.

25
03:24:51,200 --> 03:24:52,840
in terms of productivity improvement,

26
03:24:53,020 --> 03:24:54,320
and specifically what this was,

27
03:24:54,400 --> 03:24:56,160
was normalized development velocity

28
03:24:56,160 --> 03:24:58,640
against that own team's historical baseline.

29
03:24:59,220 --> 03:25:00,820
And we had some teams in the study

30
03:25:00,820 --> 03:25:02,640
who exceeded 10x, right?

31
03:25:02,760 --> 03:25:04,600
So we're talking significantly more productive.

32
03:25:05,080 --> 03:25:06,360
And what they did was just,

33
03:25:06,480 --> 03:25:07,900
they changed the way they were working.

34
03:25:08,420 --> 03:25:10,220
But I see this still so often,

35
03:25:10,340 --> 03:25:11,740
and just by your show of hands,

36
03:25:11,820 --> 03:25:12,580
the same thing, right,

37
03:25:12,620 --> 03:25:15,020
where people don't feel like that promise of AI

38
03:25:15,020 --> 03:25:16,240
has quite been delivered yet.

39
03:25:17,840 --> 03:25:19,120
Now, my name's Doug Clausen.

40
03:25:19,160 --> 03:25:20,680
I'm a product manager on the Kiro team.

41
03:25:21,120 --> 03:25:23,920
And I've been on this AI DevTool journey really since the beginning.

42
03:25:24,800 --> 03:25:28,120
And I still see, like I said, teams who aren't quite getting what they expect.

43
03:25:29,780 --> 03:25:34,080
Now, as a PM on Kiro, I get asked literally the same two questions all the time.

44
03:25:34,200 --> 03:25:36,180
Like, this is a story of my life and meetings.

45
03:25:36,480 --> 03:25:37,800
What are you shipping next?

46
03:25:38,240 --> 03:25:40,300
And they want to know, like, what are the new features?

47
03:25:40,460 --> 03:25:43,300
What new use cases might be enabled by those new features?

48
03:25:43,560 --> 03:25:45,140
And, you know, I can walk through that, and that's great.

49
03:25:45,700 --> 03:25:48,280
The other question, and oftentimes the more interesting one,

50
03:25:48,380 --> 03:25:50,360
because everybody builds software slightly differently,

51
03:25:50,360 --> 03:25:52,040
is are we doing this right?

52
03:25:52,700 --> 03:25:55,080
People feel like they've hit some sort of local maximum

53
03:25:55,080 --> 03:25:57,180
and they're scared that somebody else has figured out

54
03:25:57,180 --> 03:25:59,080
how to use the AI better than they have

55
03:25:59,080 --> 03:26:00,380
and they're just going to get surpassed.

56
03:26:01,180 --> 03:26:03,580
We're actually going to spend today on the second question here,

57
03:26:03,700 --> 03:26:04,600
not the first question.

58
03:26:05,760 --> 03:26:07,780
I'm going to walk you through how one of our teams,

59
03:26:07,940 --> 03:26:11,340
our Kuro Crew team, is shipping now thousands of PRs per week,

60
03:26:11,700 --> 03:26:13,500
the big lessons we learned on the way

61
03:26:13,500 --> 03:26:16,200
and sort of the process we went through to get to where we are today.

62
03:26:16,900 --> 03:26:19,020
And then at the end of this, I'm going to leave you with a workshop

63
03:26:19,020 --> 03:26:21,660
that if you want to try to build your own software factory in Kiro Crew,

64
03:26:21,800 --> 03:26:22,820
you can go and do that as well.

65
03:26:25,660 --> 03:26:26,200
Excuse me.

66
03:26:26,340 --> 03:26:28,440
Now, first off, just some bit of context on Kiro.

67
03:26:28,660 --> 03:26:30,760
So Kiro is an agentic development system

68
03:26:30,760 --> 03:26:34,000
that lets you plan, build, and ship software with AI agents.

69
03:26:34,580 --> 03:26:35,760
It works wherever you work.

70
03:26:35,860 --> 03:26:38,220
So up here I'm showing you we have an IDE client,

71
03:26:38,420 --> 03:26:40,280
a CLI, a web, and a mobile client.

72
03:26:40,400 --> 03:26:42,960
And then I just talked just briefly, we have Kiro Crew as well.

73
03:26:44,040 --> 03:26:46,260
We support a variety of models.

74
03:26:46,260 --> 03:26:51,660
So we have both the Anthropic models and the OpenAI models in Kiro, as well as some openweight models.

75
03:26:51,740 --> 03:26:55,100
I'll talk a little bit later how we use that and how that's been beneficial to us.

76
03:26:56,380 --> 03:27:00,260
And just like the rest of AWS, we really push the barrier on cost performance.

77
03:27:00,440 --> 03:27:06,980
So when we start saving money and we make the models cheaper to run and the agents cheaper to run, we end up passing those savings on to you.

78
03:27:07,040 --> 03:27:11,720
So we're really happy with where Kiro has gotten on that Pareto front of sort of cost performance and cost effectiveness.

79
03:27:13,300 --> 03:27:13,840
Excuse me.

80
03:27:13,840 --> 03:27:17,120
Now, like I said, we're going to spend this day talking about KiroCrew, and KiroCrew

81
03:27:17,120 --> 03:27:19,820
was built to run many Kiro agents at the same time.

82
03:27:20,380 --> 03:27:21,480
It's all open source.

83
03:27:21,660 --> 03:27:27,180
It actually started internally as an internal Amazon product where some engineers grabbed

84
03:27:27,180 --> 03:27:29,320
our CLI and they're like, hey, we want to extend this.

85
03:27:29,400 --> 03:27:34,240
It was during the whole OpenClaw craze, if anybody has their Mac minis at home, they've

86
03:27:34,240 --> 03:27:34,820
been running those.

87
03:27:34,980 --> 03:27:38,260
It happened during that craze where people wanted to figure out how they could get these

88
03:27:38,260 --> 03:27:39,900
agents to be more like personal assistants.

89
03:27:39,900 --> 03:27:42,520
And so this MeshClaw product internally was developed,

90
03:27:42,760 --> 03:27:46,920
got around 40,000 new internal Amazon developers on it

91
03:27:46,920 --> 03:27:47,800
in the first two months.

92
03:27:48,220 --> 03:27:49,660
And so us at the Kiro team were like,

93
03:27:49,700 --> 03:27:51,380
hey, since it was built on Kiro,

94
03:27:51,480 --> 03:27:53,160
we should take that and extend it and release it.

95
03:27:53,200 --> 03:27:54,180
And so we've done that now.

96
03:27:54,660 --> 03:27:55,980
And Kiro Crew is released.

97
03:27:56,260 --> 03:27:57,180
It's open source.

98
03:27:58,720 --> 03:28:02,040
And if I talk about what have we seen with the Kiro Crew team

99
03:28:02,040 --> 03:28:04,340
and our team in terms of building and developing this,

100
03:28:05,580 --> 03:28:07,820
we shipped on August 4th externally.

101
03:28:07,980 --> 03:28:09,100
And again, it's open source,

102
03:28:09,100 --> 03:28:11,720
so you can go look at all the source and look at all the PRs that came through.

103
03:28:12,320 --> 03:28:16,360
We go by one philosophy, that if we don't use it and love it, then we don't ship it.

104
03:28:17,200 --> 03:28:22,240
Since we launched, we've merged about 16,500 PRs with a very small team.

105
03:28:22,840 --> 03:28:25,760
Just last week, there was over 3,000 in a single week alone.

106
03:28:26,060 --> 03:28:31,260
Take contributions from external entities, both internal to Amazon and external to Amazon,

107
03:28:31,380 --> 03:28:34,020
people who just want to contribute and take a good idea with open source.

108
03:28:35,620 --> 03:28:39,900
And if you look at the chart here, we sort of leveled out at around, you know, if you

109
03:28:39,900 --> 03:28:45,640
look at just our shipping speed, we sort of leveled out at around high 100, 600 to 800,

110
03:28:45,980 --> 03:28:47,920
sort of the first few months after we shipped.

111
03:28:48,380 --> 03:28:52,520
And then you'll see in like 928, right, all of a sudden we break 1,000 again, and then

112
03:28:52,520 --> 03:28:57,520
we're sort of hovering around, I'll average it out in my head and say 1.2K-ish, right?

113
03:28:57,880 --> 03:29:00,140
And then you can see we made another step function improvement.

114
03:29:00,140 --> 03:29:04,700
And while I'm showing you PRs on this chart, what we really care about among the team is

115
03:29:04,700 --> 03:29:09,420
not necessarily the PRs, but we care about how many sessions can an individual developer

116
03:29:10,080 --> 03:29:11,260
operate at the same time.

117
03:29:12,340 --> 03:29:16,300
And I want to be clear that this is not a one size fits all thing, so as I walk through

118
03:29:16,300 --> 03:29:20,120
this, just keep that in mind that what works for us may not work for you, and so you should

119
03:29:20,120 --> 03:29:23,440
challenge what are you doing that you could learn from this.

120
03:29:25,000 --> 03:29:28,340
Last thing on this one, about one in ten of these is features.

121
03:29:28,920 --> 03:29:31,880
So I always get asked, like, man, 3,300 features,

122
03:29:31,920 --> 03:29:33,180
that's a lot of features in a week.

123
03:29:33,220 --> 03:29:34,420
Well, no, it's not all features, right?

124
03:29:34,660 --> 03:29:36,440
This is what shipping with speed looks like.

125
03:29:36,980 --> 03:29:38,560
A lot of, so I get asked, okay,

126
03:29:38,640 --> 03:29:41,640
so about one in 10 of these is a feature, so about 10%.

127
03:29:42,180 --> 03:29:45,220
And so if 10% are a feature, what are the other 90%?

128
03:29:46,100 --> 03:29:47,640
Well, for us, it's tests.

129
03:29:47,900 --> 03:29:49,200
Tests is a big part of what we do.

130
03:29:50,760 --> 03:29:53,700
It's, I'll say, paper cut fixes for things we've already shipped

131
03:29:53,700 --> 03:29:56,140
that we're just sort of learning about how it's getting used

132
03:29:56,140 --> 03:29:57,620
and we're massaging it a little bit.

133
03:29:58,080 --> 03:29:59,540
It's bug fixes as well.

134
03:29:59,680 --> 03:29:59,960
And that's

135
03:30:00,000 --> 03:30:02,540
really what shipping at a high speed with velocity looks like, right?

136
03:30:03,080 --> 03:30:06,820
It's not all new features, but if you just think about 10% of our features, there's still

137
03:30:06,820 --> 03:30:09,180
hundreds of features, new features being shipped over this timeline.

138
03:30:11,940 --> 03:30:17,500
If I think about sort of how did we evolve with our development cycle, we started, if

139
03:30:17,500 --> 03:30:21,580
I go back to when our CLI product initially launched, it was one session, one person.

140
03:30:21,580 --> 03:30:23,700
So I'm sure many of us experienced this

141
03:30:23,700 --> 03:30:25,240
when these new agents came out,

142
03:30:25,260 --> 03:30:27,500
and I was typing, and it would then work,

143
03:30:27,580 --> 03:30:28,800
and then it would stop working,

144
03:30:28,880 --> 03:30:29,900
and then I would type again, right?

145
03:30:29,900 --> 03:30:30,900
We'd just go back and forth.

146
03:30:31,860 --> 03:30:33,900
And then from that, so that was one session, right?

147
03:30:33,940 --> 03:30:35,760
I said we're trying to optimize sessions per person.

148
03:30:36,140 --> 03:30:37,600
Then we moved to parallel sessions,

149
03:30:37,600 --> 03:30:39,340
where instead of just running one session,

150
03:30:39,500 --> 03:30:41,780
I'm looking at, you know, maybe I'm running two or three,

151
03:30:41,880 --> 03:30:43,360
but I'm running them in the same manner, right?

152
03:30:43,980 --> 03:30:45,840
And the problem with both of these is that

153
03:30:45,840 --> 03:30:48,200
if you make yourself the feedback loop,

154
03:30:48,400 --> 03:30:49,940
you are now the bottleneck, right?

155
03:30:49,940 --> 03:30:52,960
These agents got faster long before you're going to be able to keep up.

156
03:30:53,340 --> 03:30:55,980
And so these first two sessions, they're just going to kill in one chart here,

157
03:30:56,120 --> 03:30:58,620
where it's, you know, we got to two or three sessions per person,

158
03:30:58,660 --> 03:30:59,480
but we sort of topped out.

159
03:31:00,980 --> 03:31:03,280
Now, stage three introduced memory and schedules.

160
03:31:03,600 --> 03:31:08,120
And with a memory, a new session now all of a sudden knows your conventions, right?

161
03:31:08,200 --> 03:31:09,080
So implicit memories.

162
03:31:09,780 --> 03:31:12,960
And then schedules make it so you don't have to kick all of the work off on your own.

163
03:31:13,380 --> 03:31:14,540
Things can happen on a schedule.

164
03:31:15,460 --> 03:31:19,280
The thing we learned here was that schedules only help if your machine is on.

165
03:31:19,280 --> 03:31:23,900
So in the previous phases, most of our engineers were running locally, right, as I'm sure many of you do still today.

166
03:31:24,620 --> 03:31:28,920
But moving to remote ends up making it so you no longer have to worry about, is it going to run?

167
03:31:28,960 --> 03:31:32,160
Because I have an always-on machine, something we're very good at at AWS.

168
03:31:32,460 --> 03:31:34,080
So obviously we are running remote.

169
03:31:35,520 --> 03:31:40,700
But what happened then is this allowed us to sort of, I'll say, build up to between 10 and 20 sessions, right?

170
03:31:40,780 --> 03:31:41,960
And then we sort of plateaued again.

171
03:31:42,240 --> 03:31:45,420
We were unable to sort of break that barrier with just memory and schedules.

172
03:31:46,520 --> 03:31:48,180
because much of what we were doing

173
03:31:48,180 --> 03:31:51,160
was still being the glue between the sessions, right?

174
03:31:52,840 --> 03:31:55,160
And then if we, our next phase was a pipeline,

175
03:31:55,360 --> 03:31:57,660
but not a CICD pipeline like you might be thinking,

176
03:31:57,780 --> 03:31:59,260
but instead more like an assembly line.

177
03:32:00,100 --> 03:32:03,120
I actually have, my previous life, I worked in aerospace,

178
03:32:03,340 --> 03:32:04,200
so a lot of assembly lines,

179
03:32:04,260 --> 03:32:07,020
and the hard thing with assembly lines is nonstandard work, right?

180
03:32:07,160 --> 03:32:09,900
So if everything is exactly the same as it flows through the system,

181
03:32:10,020 --> 03:32:11,040
an assembly line works great.

182
03:32:11,360 --> 03:32:13,040
As soon as you get nonstandard work,

183
03:32:13,120 --> 03:32:14,280
all of a sudden it becomes difficult,

184
03:32:14,280 --> 03:32:16,300
and I'm sure all of you know this

185
03:32:16,300 --> 03:32:18,300
but software doesn't follow the exact same

186
03:32:18,300 --> 03:32:19,140
path every time

187
03:32:19,860 --> 03:32:22,260
and so what we found our own teams doing then

188
03:32:22,260 --> 03:32:24,320
was this, sorry, this got us to maybe like

189
03:32:24,320 --> 03:32:26,420
30, 20, 30, 40

190
03:32:26,420 --> 03:32:28,460
sessions-ish but what happened

191
03:32:28,460 --> 03:32:30,240
was is our team was then trying to

192
03:32:30,240 --> 03:32:32,340
unblock the assembly line or the pipeline, right

193
03:32:32,340 --> 03:32:34,300
so as things came up if we

194
03:32:34,300 --> 03:32:36,380
thought something was a small issue and then it turned into

195
03:32:36,380 --> 03:32:38,280
a big issue, sometimes the assembly line would be

196
03:32:38,280 --> 03:32:40,340
stuck, somebody would have to come in and unblock

197
03:32:40,340 --> 03:32:42,440
that thing, maybe you restart it, maybe you replan

198
03:32:42,440 --> 03:32:47,440
it. And so this is only going to carry us so far because, again, we are in the midst

199
03:32:47,440 --> 03:32:50,740
of the agent. We're blocking the agent more than the agent wants to work.

200
03:32:53,480 --> 03:32:58,460
And so what did we do for that? Well, we launched a thing called Ultra Mode. Ultra Mode

201
03:32:58,460 --> 03:33:02,560
is still, just to be clear, if you go try it and you look for it, it's still in the developer

202
03:33:03,040 --> 03:33:06,220
preview, so it's behind a feature flag. But because it's open source, the code is all

203
03:33:06,220 --> 03:33:10,680
there. If you want to go try it, you can try it out. As I said up front, if we don't love

204
03:33:10,680 --> 03:33:13,980
it, we don't ship it, so we're still sort of finishing, putting the finishing touches

205
03:33:13,980 --> 03:33:14,500
on this one.

206
03:33:15,000 --> 03:33:19,020
But I wanted to talk about it anyway, because it takes you from being, what we did was,

207
03:33:19,140 --> 03:33:24,920
if you look at this middle part here, where we are the person driving the effort, these

208
03:33:24,920 --> 03:33:26,100
are the things that we might have been doing.

209
03:33:26,320 --> 03:33:29,820
So watching for problems was one of the biggest ones, and then trying to fix the assembly

210
03:33:29,820 --> 03:33:31,520
line when the problem came up.

211
03:33:31,940 --> 03:33:34,980
Maybe you're judging a result, or you're deciding what to do next.

212
03:33:35,740 --> 03:33:38,220
What Ultra Mode does is it does that job for you, right?

213
03:33:38,220 --> 03:33:42,980
So you put an agent on top of your agents that then is responsible for doing that, defining

214
03:33:42,980 --> 03:33:46,740
the goal based on whatever you're trying to accomplish, starting things, watching for

215
03:33:46,740 --> 03:33:48,300
problems, unsticking things.

216
03:33:48,740 --> 03:33:51,860
And this is getting us today to 50 plus sessions.

217
03:33:52,120 --> 03:33:56,460
And so if you think about where we started with a single session and a single person,

218
03:33:56,660 --> 03:34:00,820
now if a single developer is doing 50 plus sessions, that's how you get that sort of

219
03:34:00,820 --> 03:34:03,440
the volume in the PR chart I showed you earlier.

220
03:34:04,120 --> 03:34:06,460
And we can already see the next wall that's coming.

221
03:34:07,980 --> 03:34:09,900
The orchestrators working with other orchestrators,

222
03:34:10,100 --> 03:34:12,480
and then the governance around what you want that to be

223
03:34:12,480 --> 03:34:14,280
and the guardrails become even more important, right?

224
03:34:16,320 --> 03:34:17,900
So again, that's how we got the volume.

225
03:34:18,080 --> 03:34:19,620
Those were the five phases we went through.

226
03:34:20,120 --> 03:34:21,920
Ultra mode has been a game changer for us,

227
03:34:22,000 --> 03:34:23,020
as was the assembly line.

228
03:34:23,700 --> 03:34:25,740
And I invite you to try those ideas

229
03:34:25,740 --> 03:34:26,820
if you think they might work for you.

230
03:34:27,860 --> 03:34:29,480
The next thing I wanted to talk about, though,

231
03:34:29,620 --> 03:34:31,580
was what did we learn from this?

232
03:34:31,580 --> 03:34:34,480
Like, what are the biggest things that prevented us from shipping quickly?

233
03:34:35,760 --> 03:34:37,980
The first one is test everything.

234
03:34:38,180 --> 03:34:40,160
And now this is no, this is not a new lesson, right?

235
03:34:40,240 --> 03:34:41,680
Everybody in software knows this.

236
03:34:41,820 --> 03:34:42,660
We should be doing this.

237
03:34:43,000 --> 03:34:47,900
But I still see people too often discount the value this is providing.

238
03:34:48,260 --> 03:34:51,600
And this is gonna build in with the next one I have behind this around reviews.

239
03:34:52,080 --> 03:34:54,580
Like, if you think about where do you get confidence to ship something?

240
03:34:54,960 --> 03:34:58,720
I think many people still feel like, hey, I get confidence because I looked at all the

241
03:34:58,720 --> 03:35:03,740
code and I know exactly what that code does. Again, if you do that, like maybe that's a

242
03:35:03,740 --> 03:35:08,000
good thing, but where we have gotten our confidence, and Amazon's always been very test-driven,

243
03:35:08,660 --> 03:35:12,500
is from the tests. Because you might not be able to look at something every time, but

244
03:35:12,500 --> 03:35:17,160
you can certainly test something every time in the exact same way and test the result,

245
03:35:17,300 --> 03:35:22,980
right? And come out with the result. And so for us, again, quality really becomes a testing

246
03:35:22,980 --> 03:35:27,320
problem and we're spending a significant portion of our time, around 20% just on testing today.

247
03:35:29,120 --> 03:35:33,020
The other big thing we learned, and this is a bigger one, and also a contentious topic,

248
03:35:33,300 --> 03:35:34,260
is, is reviews.

249
03:35:35,200 --> 03:35:39,920
So quick show of hands, how many people have done a review with an agent, thinking that

250
03:35:39,920 --> 03:35:43,460
there's going to be like one or two comments, and then you come back and you're like, oh

251
03:35:43,460 --> 03:35:46,920
my god, like this is terrible code, right?

252
03:35:47,020 --> 03:35:48,120
There's comments everywhere.

253
03:35:48,460 --> 03:35:49,600
We had the same problem.

254
03:35:49,780 --> 03:35:53,300
So we originally wrote a review agent that would review everything, and our goal was

255
03:35:53,300 --> 03:35:54,700
to fix every single comment.

256
03:35:54,700 --> 03:36:00,060
Well, as you guys, I heard the laughter, so as you expect, you submit changes, so you

257
03:36:00,060 --> 03:36:05,660
submit a PR, you get the feedback on that PR, you come back, you submit it again after you

258
03:36:05,660 --> 03:36:08,580
fix it, you come back and now it's three new things that weren't even tested, weren't

259
03:36:08,580 --> 03:36:11,540
even told about before and it's four things on the things you already fixed.

260
03:36:11,760 --> 03:36:14,440
You submit it again and this cycle just continues to repeat.

261
03:36:14,880 --> 03:36:19,780
So what we've done with reviews is now we categorize the reviews or triage them into

262
03:36:19,780 --> 03:36:21,420
blocking and non-blocking issues.

263
03:36:21,420 --> 03:36:24,480
So, for example, something that breaks the build,

264
03:36:25,000 --> 03:36:26,720
if there's something obviously that breaks the build,

265
03:36:26,820 --> 03:36:27,660
that's a blocking thing,

266
03:36:27,660 --> 03:36:29,600
and we need to go fix that before we merge this PR.

267
03:36:30,260 --> 03:36:32,460
Changing a variable name, not a big deal.

268
03:36:32,640 --> 03:36:33,660
We'll take care of that later,

269
03:36:33,680 --> 03:36:34,980
so we let those things slide through.

270
03:36:36,100 --> 03:36:37,640
The other thing that we've done with reviews

271
03:36:38,080 --> 03:36:40,620
is we want to have fast feedback for an agent,

272
03:36:40,640 --> 03:36:42,340
and as you can see by the process here

273
03:36:42,340 --> 03:36:43,220
that I'll walk you through in a second,

274
03:36:43,280 --> 03:36:44,260
this is not that fast.

275
03:36:44,700 --> 03:36:45,900
And so we have a fast gate

276
03:36:45,900 --> 03:36:47,880
that really double-checks things really quickly

277
03:36:47,880 --> 03:36:49,640
to make sure that things are in the right folders

278
03:36:49,640 --> 03:36:50,900
before we move on to our reviews.

279
03:36:51,880 --> 03:36:53,740
We have six reviewers, actually, that run.

280
03:36:53,960 --> 03:36:56,280
Two code reviewers, both in different models,

281
03:36:56,560 --> 03:36:57,580
with different model families.

282
03:36:58,340 --> 03:37:00,400
And the intent there is to really find,

283
03:37:00,620 --> 03:37:02,200
to make sure you don't have a model gap, right,

284
03:37:02,260 --> 03:37:04,220
so that the model wasn't trained on something that we don't catch.

285
03:37:04,460 --> 03:37:07,600
This is where we use the Anthropic and OpenAI models

286
03:37:07,600 --> 03:37:10,040
or an open weight model to be those levers.

287
03:37:10,800 --> 03:37:13,160
We run a design review that asks,

288
03:37:13,300 --> 03:37:14,660
does this thing have the right shape?

289
03:37:14,800 --> 03:37:16,500
Meaning, is what I'm building,

290
03:37:16,700 --> 03:37:18,840
like if I'm adding another cache in this PR,

291
03:37:18,840 --> 03:37:21,860
can the cache I already have take care of this issue or not.

292
03:37:22,260 --> 03:37:24,520
We have a first principles review that says,

293
03:37:24,880 --> 03:37:26,940
is this change addressing the real problem?

294
03:37:27,440 --> 03:37:29,100
We have a UX review as well,

295
03:37:29,280 --> 03:37:31,860
that if you're making it, let's say you're creating a new error message,

296
03:37:32,100 --> 03:37:34,400
we have an agent decide whether or not that error message,

297
03:37:34,560 --> 03:37:36,780
the human will know what to do with that error message

298
03:37:36,780 --> 03:37:38,120
based on what we've published.

299
03:37:38,440 --> 03:37:40,900
And then we're consistently tightening our security posture.

300
03:37:41,640 --> 03:37:43,500
Now, each one of these reviewers gets one vote.

301
03:37:43,640 --> 03:37:45,540
It's either yes or no. It's red or it's green.

302
03:37:45,840 --> 03:37:47,340
If everything's green, it goes on to merge.

303
03:37:47,340 --> 03:37:50,220
If not, we go to a prepare PR skill that's in the repo,

304
03:37:50,240 --> 03:37:51,580
so you can check the skill out if you want.

305
03:37:51,860 --> 03:37:55,080
And that prepare PR skill then tries to resolve the issues as we go.

306
03:37:57,500 --> 03:38:00,380
The third lesson, and sorry, on the reviews, one more thing.

307
03:38:00,700 --> 03:38:02,920
Like I said, we've stopped reviewing code a long time ago.

308
03:38:03,040 --> 03:38:06,340
We have moved on, and what we're responsible for is not the code itself.

309
03:38:06,640 --> 03:38:07,920
See, this actually goes back. It does.

310
03:38:08,220 --> 03:38:10,260
What we're responsible for is the review record.

311
03:38:10,460 --> 03:38:12,400
Did it follow the process as we expected?

312
03:38:12,620 --> 03:38:14,060
And did we get the results we wanted?

313
03:38:14,060 --> 03:38:15,600
If not, we fixed the review process.

314
03:38:17,560 --> 03:38:24,060
Two other things. So, one, the over-engineering problem. So we have a specific role to teach

315
03:38:24,060 --> 03:38:28,120
our agents to subtract, not just add. And so we try to remove code from the process

316
03:38:28,120 --> 03:38:32,100
as we go to prevent that over-engineering. And we use that same philosophy on our roadmap

317
03:38:32,100 --> 03:38:36,720
as well, where saying yes is becoming very cheap or has become very cheap, right? It's

318
03:38:36,720 --> 03:38:40,140
very easy for us now with code to say, like, yes, I'll do that and I'll do all of these

319
03:38:40,140 --> 03:38:43,320
things. Whereas we used to have to be very particular about what we were going to build.

320
03:38:44,320 --> 03:38:46,280
The problem with saying yes all the time

321
03:38:46,280 --> 03:38:48,080
is that you might lose your product shape,

322
03:38:48,080 --> 03:38:50,400
and we're already seeing this in KuroKribit,

323
03:38:50,440 --> 03:38:52,580
that it's moving so fast and so quickly

324
03:38:52,580 --> 03:38:54,940
that you don't want your users to get caught in the noise.

325
03:38:55,100 --> 03:38:57,700
And so, again, use the same principle on the roadmap

326
03:38:57,700 --> 03:38:58,240
as what we do.

327
03:38:59,560 --> 03:39:01,340
And then last thing on lessons is

328
03:39:01,900 --> 03:39:03,980
you're not done once you merge a PR, right?

329
03:39:04,020 --> 03:39:05,060
There's still other phases,

330
03:39:05,120 --> 03:39:07,640
and so for us, merging is not the last check that we do.

331
03:39:08,120 --> 03:39:09,460
Instead, we run a release gate

332
03:39:09,460 --> 03:39:11,140
where the agent will download the repo,

333
03:39:12,200 --> 03:39:14,800
It installs it, runs it, and then checks the core flows

334
03:39:14,800 --> 03:39:17,000
to make sure that the overall system is still in good shape

335
03:39:17,000 --> 03:39:18,460
before we would do a release.

336
03:39:18,960 --> 03:39:20,960
And then we do security all the time as well.

337
03:39:21,040 --> 03:39:23,500
So we have every hour, every day, an always-on,

338
03:39:23,640 --> 03:39:25,360
every hour against our security guidance,

339
03:39:25,680 --> 03:39:27,640
every day against an AppSec review,

340
03:39:27,700 --> 03:39:29,520
and then an always-on pen test agent.

341
03:39:30,280 --> 03:39:32,320
So again, merging is not the end of the process.

342
03:39:32,340 --> 03:39:33,840
You have to crunch everything down.

343
03:39:35,560 --> 03:39:36,940
So I'm going to come back to where I started.

344
03:39:37,440 --> 03:39:39,400
Are you getting everything from your agent?

345
03:39:39,400 --> 03:39:41,680
and many of us don't feel like we are yet.

346
03:39:42,020 --> 03:39:43,440
And what I'm going to offer here today,

347
03:39:43,720 --> 03:39:45,020
or what I've tried to offer is,

348
03:39:45,360 --> 03:39:49,080
the agent was fast enough to work on this a long time ago

349
03:39:49,080 --> 03:39:50,800
and it's only getting better and better and better.

350
03:39:51,140 --> 03:39:54,320
And so what we have been doing is challenging how we work

351
03:39:54,320 --> 03:39:56,980
and figuring out where are we preventing the agent

352
03:39:56,980 --> 03:39:58,340
from working as quickly as possible.

353
03:39:58,920 --> 03:40:00,000
And I've given you numerous

354
03:40:00,040 --> 03:40:00,780
examples today.

355
03:40:01,100 --> 03:40:02,980
Again, I'll just call it a couple of them here.

356
03:40:03,160 --> 03:40:05,120
So it started with our attention span, right?

357
03:40:05,140 --> 03:40:09,120
When I was the feedback loop, I could no longer ship because I was the feedback loop.

358
03:40:09,180 --> 03:40:10,800
When I wasn't there, the agent stopped working.

359
03:40:11,240 --> 03:40:12,720
The laptop lid closing.

360
03:40:12,920 --> 03:40:14,200
All of a sudden, you lose your connection.

361
03:40:14,380 --> 03:40:15,820
Moving to remote can solve that, right?

362
03:40:16,280 --> 03:40:20,960
The fixed pipeline with a fixed shape that can't respond when things go slightly off path.

363
03:40:21,340 --> 03:40:22,440
Again, how do you fix that?

364
03:40:22,940 --> 03:40:24,300
We fixed it with an orchestrator.

365
03:40:24,300 --> 03:40:26,440
ultra-mode. The endless

366
03:40:26,440 --> 03:40:28,400
reviews that I think I've gone into probably

367
03:40:28,400 --> 03:40:29,960
more depth than you were interested in, but

368
03:40:29,960 --> 03:40:32,500
the endless review cycle, and then the loop that only

369
03:40:32,500 --> 03:40:34,380
adds, right? The loop that only adds things

370
03:40:34,380 --> 03:40:35,640
to it and doesn't try to subtract.

371
03:40:38,080 --> 03:40:38,580
And so

372
03:40:38,580 --> 03:40:40,460
I'll leave you with, I also want to be

373
03:40:40,460 --> 03:40:42,180
clear that this is going to be a

374
03:40:43,680 --> 03:40:44,400
continual adjustment,

375
03:40:44,580 --> 03:40:46,440
right? This is not, like, this may be what works for us

376
03:40:46,440 --> 03:40:48,300
today. I promise you another month we're going to find

377
03:40:48,300 --> 03:40:50,480
something else that works, and we're going to continue to evolve

378
03:40:50,480 --> 03:40:52,380
significantly. But like I said in the

379
03:40:52,380 --> 03:40:57,020
beginning, the pace that we're finally able to deliver at, I feel like for us, is that finally

380
03:40:57,020 --> 03:41:01,780
we're meeting that promise of AI, and I still see the disparity among other teams.

381
03:41:03,520 --> 03:41:08,040
And so, like I said, I'm going to leave you with this. If you're interested in trying Kiro Crew and

382
03:41:08,040 --> 03:41:13,340
building your own software factory, scan this QR code. It's a workshop. We have a pretty generous

383
03:41:13,340 --> 03:41:19,160
free trial on Kiro, so you can go sign up for a free trial, go install Kiro Crew, and you'll be

384
03:41:19,160 --> 03:41:19,700
off and running.

385
03:41:21,200 --> 03:41:23,800
And if you don't, if you want to try, like I said, if you want to try CreoCure, you can

386
03:41:23,800 --> 03:41:24,040
do that.

387
03:41:24,200 --> 03:41:26,280
The same free trial works across all of the products.

388
03:41:26,420 --> 03:41:30,700
So the IDE, the CLI, the web client, the mobile client, try them all with that free trial.

389
03:41:31,240 --> 03:41:33,920
And with that, I'm excited to see what you all build.

390
03:41:34,120 --> 03:41:34,820
I appreciate the time.

391
03:41:34,940 --> 03:41:35,180
Thank you.

392
03:41:35,280 --> 03:41:36,100
Enjoy the rest of the conference.