1
03:10:06,240 --> 03:10:08,230
Okay.

2
03:10:08,240 --> 03:10:12,309
All right. Can y'all hear me?

3
03:10:12,319 --> 03:10:16,150
>> All right. My name is Andrew and I think

4
03:10:16,160 --> 03:10:17,349
we're starting now. I don't know if

5
03:10:17,359 --> 03:10:20,150
there's a timer anywhere. Uh my name is

6
03:10:20,160 --> 03:10:22,070
Andrew and I'm excited to talk to you

7
03:10:22,080 --> 03:10:27,349
today about the safe software factory.

8
03:10:27,359 --> 03:10:30,389
Uh but first a little bit about me. Uh I

9
03:10:30,399 --> 03:10:33,990
am a staff software engineer at Sonar. I

10
03:10:34,000 --> 03:10:36,950
work on guitar which is Sonar's AI code

11
03:10:36,960 --> 03:10:40,389
review products. Uh and previously I was

12
03:10:40,399 --> 03:10:43,349
at uh Uber working on the driver trip

13
03:10:43,359 --> 03:10:45,750
experience and navigation.

14
03:10:45,760 --> 03:10:47,590
Uh and then after that I was at a health

15
03:10:47,600 --> 03:10:49,349
tech startup called forward for a little

16
03:10:49,359 --> 03:10:52,630
while uh building a sort of an AI

17
03:10:52,640 --> 03:10:59,429
enabled primary health care system.

18
03:10:59,439 --> 03:11:03,429
Okay. So the software factory this is

19
03:11:03,439 --> 03:11:06,389
like the hottest buzzword in our

20
03:11:06,399 --> 03:11:09,030
industry right now. Everyone's talking

21
03:11:09,040 --> 03:11:10,870
about software factories. Everyone's

22
03:11:10,880 --> 03:11:15,590
trying to build a software factory.

23
03:11:15,600 --> 03:11:19,030
And the question is, well, oh well, so

24
03:11:19,040 --> 03:11:20,870
first of all,

25
03:11:20,880 --> 03:11:22,710
um,

26
03:11:22,720 --> 03:11:25,510
most of you probably know what this is

27
03:11:25,520 --> 03:11:28,950
already. Um, but if you don't, I'll just

28
03:11:28,960 --> 03:11:31,670
give a really quick overview. Uh, this

29
03:11:31,680 --> 03:11:33,670
is sort of the the main software

30
03:11:33,680 --> 03:11:36,469
development life cycle loop uh, as it

31
03:11:36,479 --> 03:11:40,469
existed before the AI era. So we have uh

32
03:11:40,479 --> 03:11:42,950
inputs basically product specs, feature

33
03:11:42,960 --> 03:11:47,750
requests, tickets, things like that. Um

34
03:11:47,760 --> 03:11:49,830
we take those into the planning step if

35
03:11:49,840 --> 03:11:50,870
they're especially if they're

36
03:11:50,880 --> 03:11:52,469
complicated.

37
03:11:52,479 --> 03:11:54,550
Uh and then we then we build then we

38
03:11:54,560 --> 03:11:56,870
write the code

39
03:11:56,880 --> 03:12:00,550
uh and then we um uh put up a PR and

40
03:12:00,560 --> 03:12:03,990
that PR gets uh reviewed and it goes

41
03:12:04,000 --> 03:12:08,150
through CI and then it eventually merges

42
03:12:08,160 --> 03:12:11,429
uh and then we deploy that to production

43
03:12:11,439 --> 03:12:13,429
and then users interact with our

44
03:12:13,439 --> 03:12:15,830
software and they give us feedback or

45
03:12:15,840 --> 03:12:17,750
they find bugs

46
03:12:17,760 --> 03:12:20,630
um or we might also get some monitoring

47
03:12:20,640 --> 03:12:22,870
and alerts as well. And then those go

48
03:12:22,880 --> 03:12:26,950
back in as more inputs and the the cycle

49
03:12:26,960 --> 03:12:28,550
continues

50
03:12:28,560 --> 03:12:32,230
and and so um in the in the era of

51
03:12:32,240 --> 03:12:36,230
agentic coding and AI automation has

52
03:12:36,240 --> 03:12:38,309
followed every bottleneck. So the

53
03:12:38,319 --> 03:12:40,469
bottleneck used to be the the coding

54
03:12:40,479 --> 03:12:43,750
part the building um now with coding

55
03:12:43,760 --> 03:12:47,590
agents that's been automated. Um, and

56
03:12:47,600 --> 03:12:49,269
now the bottleneck has sort of shifted

57
03:12:49,279 --> 03:12:52,389
over to the that review step and we're

58
03:12:52,399 --> 03:12:55,030
starting to introduce uh AI code review

59
03:12:55,040 --> 03:13:00,070
systems for that. Um, and then and we're

60
03:13:00,080 --> 03:13:02,710
seeing uh AI systems and automation

61
03:13:02,720 --> 03:13:05,269
happen really at every step of this now.

62
03:13:05,279 --> 03:13:08,070
Uh so we're starting to see AI systems

63
03:13:08,080 --> 03:13:11,429
for monitoring deployments, controlling

64
03:13:11,439 --> 03:13:15,990
rollouts, triaging uh triaging, you

65
03:13:16,000 --> 03:13:18,790
know, tickets and incidents, planning.

66
03:13:18,800 --> 03:13:21,670
And so this this here is the sort of

67
03:13:21,680 --> 03:13:25,110
software factory that we're that we're

68
03:13:25,120 --> 03:13:28,469
creating.

69
03:13:28,479 --> 03:13:31,030
And and so the question that we all have

70
03:13:31,040 --> 03:13:35,349
now is how do we make this factory safe,

71
03:13:35,359 --> 03:13:37,910
right? Uh in other words, how can we

72
03:13:37,920 --> 03:13:41,349
introduce enough layers of verification

73
03:13:41,359 --> 03:13:43,830
into this so that we can trust that the

74
03:13:43,840 --> 03:13:47,110
software coming out of it is it works as

75
03:13:47,120 --> 03:13:49,910
expected. It's free of bugs, free of

76
03:13:49,920 --> 03:13:54,070
security vulnerabilities.

77
03:13:54,080 --> 03:13:56,550
Um, our CEO

78
03:13:56,560 --> 03:13:59,030
Tar said recently, "Code is abundant,

79
03:13:59,040 --> 03:14:01,429
confidence is not."

80
03:14:01,439 --> 03:14:02,950
So, what I'm going to talk about today

81
03:14:02,960 --> 03:14:04,950
is just how we at Sonar have been

82
03:14:04,960 --> 03:14:08,870
thinking about uh how to introduce uh

83
03:14:08,880 --> 03:14:10,469
confidence

84
03:14:10,479 --> 03:14:12,710
uh at at all sort of all different

85
03:14:12,720 --> 03:14:17,670
points within this software factory.

86
03:14:17,680 --> 03:14:20,469
And uh we're approaching this this

87
03:14:20,479 --> 03:14:22,630
challenge with a defense and depth

88
03:14:22,640 --> 03:14:26,309
approach that comes from adding more

89
03:14:26,319 --> 03:14:28,070
loops. You might not be surprised to

90
03:14:28,080 --> 03:14:29,830
hear

91
03:14:29,840 --> 03:14:31,510
and I'm going to talk about four of them

92
03:14:31,520 --> 03:14:34,070
today in particular. So there's the loop

93
03:14:34,080 --> 03:14:37,990
one which is the PR checks loop.

94
03:14:38,000 --> 03:14:42,150
Uh loop two uh which is the post deploy

95
03:14:42,160 --> 03:14:45,269
verifications loop.

96
03:14:45,279 --> 03:14:48,469
loop three uh which is the maintenance

97
03:14:48,479 --> 03:14:50,870
loop

98
03:14:50,880 --> 03:14:52,469
and loop four which is the knowledge

99
03:14:52,479 --> 03:14:54,229
loop and this is kind of a kind of a

100
03:14:54,239 --> 03:14:57,750
special loop. Uh so first I'm going to

101
03:14:57,760 --> 03:15:01,110
talk about loop one the PR checks loop.

102
03:15:01,120 --> 03:15:02,550
So this is the loop that we probably all

103
03:15:02,560 --> 03:15:06,229
have in some form. Uh you put up a PR,

104
03:15:06,239 --> 03:15:08,150
it goes through some CI checks, it gets

105
03:15:08,160 --> 03:15:10,229
a code review. Nowadays it's probably

106
03:15:10,239 --> 03:15:12,950
getting an AI code review as well. Um,

107
03:15:12,960 --> 03:15:14,309
but we're starting to see a lot more

108
03:15:14,319 --> 03:15:16,710
automation being introduced here as

109
03:15:16,720 --> 03:15:19,830
well. So, we're starting to see agents

110
03:15:19,840 --> 03:15:21,830
pushing fixes to respond to the code

111
03:15:21,840 --> 03:15:25,670
review comments. Um, and this loop kind

112
03:15:25,680 --> 03:15:28,150
of goes around and round until the PR

113
03:15:28,160 --> 03:15:34,469
turns green.

114
03:15:34,479 --> 03:15:37,269
And there's actually a lot more that we

115
03:15:37,279 --> 03:15:40,469
can be doing here. um because catching a

116
03:15:40,479 --> 03:15:42,309
problem at this point is still cheaper

117
03:15:42,319 --> 03:15:45,190
than after it merges into the codebase.

118
03:15:45,200 --> 03:15:47,510
Uh so we have a real opportunity here to

119
03:15:47,520 --> 03:15:50,469
widen this review past the sort of

120
03:15:50,479 --> 03:15:53,670
normal tests and code review that most

121
03:15:53,680 --> 03:15:57,349
of us do. So of course we have AI code

122
03:15:57,359 --> 03:16:00,550
review. We can also do static analysis

123
03:16:00,560 --> 03:16:03,670
here. Many of us do that.

124
03:16:03,680 --> 03:16:05,349
Uh we could do what's called functional

125
03:16:05,359 --> 03:16:07,349
validation. So this is where we can look

126
03:16:07,359 --> 03:16:09,590
at uh what the ticket asks for and see

127
03:16:09,600 --> 03:16:12,229
if the PR actually addresses those

128
03:16:12,239 --> 03:16:15,590
things.

129
03:16:15,600 --> 03:16:19,830
Um we can look at rules that the team

130
03:16:19,840 --> 03:16:23,269
may set for coding conventions

131
03:16:23,279 --> 03:16:26,389
uh see if best practices things like

132
03:16:26,399 --> 03:16:30,870
that see if the PR is adhering to those.

133
03:16:30,880 --> 03:16:32,790
We can look at knowledge. So this can be

134
03:16:32,800 --> 03:16:34,950
knowledge accumulated from company

135
03:16:34,960 --> 03:16:36,469
documentation,

136
03:16:36,479 --> 03:16:37,990
from things that have been learned on

137
03:16:38,000 --> 03:16:43,110
past code reviews or past outages.

138
03:16:43,120 --> 03:16:45,429
Uh we could do cross repo checks. So we

139
03:16:45,439 --> 03:16:46,229
could actually look at other

140
03:16:46,239 --> 03:16:48,389
repositories and see if anything in in

141
03:16:48,399 --> 03:16:51,990
this PR uh will break or cause something

142
03:16:52,000 --> 03:16:55,990
to break in another repository.

143
03:16:56,000 --> 03:16:59,510
Uh we could do cross PR analysis. So, we

144
03:16:59,520 --> 03:17:01,670
can look at all the other open PRs as

145
03:17:01,680 --> 03:17:05,349
well and see if there's anything uh in

146
03:17:05,359 --> 03:17:07,110
this PR that conflicts with one of

147
03:17:07,120 --> 03:17:09,990
those.

148
03:17:10,000 --> 03:17:12,389
We could do security review, check for

149
03:17:12,399 --> 03:17:17,190
vulnerabilities, secrets, dependencies.

150
03:17:17,200 --> 03:17:19,190
We could do endto-end testing. So, for

151
03:17:19,200 --> 03:17:20,790
example, if this is a web app, we could

152
03:17:20,800 --> 03:17:22,790
open up in a browser and and click

153
03:17:22,800 --> 03:17:27,670
around and see if it works as expected.

154
03:17:27,680 --> 03:17:30,150
Um and finally we can do risk

155
03:17:30,160 --> 03:17:32,630
assessment. So we could sort of uh look

156
03:17:32,640 --> 03:17:36,469
at this and compute uh the blast radius

157
03:17:36,479 --> 03:17:38,950
and sort of categorize this into like

158
03:17:38,960 --> 03:17:41,429
low, medium or high risk. And that can

159
03:17:41,439 --> 03:17:43,190
actually help us determine if this is a

160
03:17:43,200 --> 03:17:45,110
PR that a human even needs to look at at

161
03:17:45,120 --> 03:17:47,670
all.

162
03:17:47,680 --> 03:17:49,830
And so all of these things can flow into

163
03:17:49,840 --> 03:17:51,910
this the sort of decision this merge

164
03:17:51,920 --> 03:17:53,830
gate this decision of whether this PR is

165
03:17:53,840 --> 03:17:57,349
mergeable. And of course uh every time a

166
03:17:57,359 --> 03:18:00,550
fix is pushed all of these run again at

167
03:18:00,560 --> 03:18:04,710
least in some degree.

168
03:18:04,720 --> 03:18:06,389
And and with all of this it's important

169
03:18:06,399 --> 03:18:09,269
to gate the that merge decision not just

170
03:18:09,279 --> 03:18:11,349
on the fact that a test ran or that it

171
03:18:11,359 --> 03:18:13,670
passed but on the evidence. So that

172
03:18:13,680 --> 03:18:15,349
would be things like the actual code

173
03:18:15,359 --> 03:18:17,590
review findings, the results of the

174
03:18:17,600 --> 03:18:19,510
quality gates,

175
03:18:19,520 --> 03:18:22,389
uh any vulnerabilities,

176
03:18:22,399 --> 03:18:26,790
um the risk level. So all of that can be

177
03:18:26,800 --> 03:18:29,030
uh basically fed into this merge gate

178
03:18:29,040 --> 03:18:31,269
and checked against criteria that the

179
03:18:31,279 --> 03:18:35,110
team can set for what is a mergeable PR.

180
03:18:35,120 --> 03:18:37,990
And when you record that evidence uh to

181
03:18:38,000 --> 03:18:41,590
the PR, it just makes this uh the merge

182
03:18:41,600 --> 03:18:44,950
gate much stronger.

183
03:18:44,960 --> 03:18:47,190
And so what we're seeing now uh with

184
03:18:47,200 --> 03:18:49,910
teams who do this well uh is that they

185
03:18:49,920 --> 03:18:51,510
can actually delegate more and more

186
03:18:51,520 --> 03:18:54,950
trusts uh to this type of automation on

187
03:18:54,960 --> 03:18:58,550
the PR review. Uh so we're seeing teams

188
03:18:58,560 --> 03:19:00,870
begin to turn on not just automatic

189
03:19:00,880 --> 03:19:05,110
merge blocking but uh even auto approve

190
03:19:05,120 --> 03:19:07,670
in some cases for low-risk changes and

191
03:19:07,680 --> 03:19:09,910
and even automerge for certain types of

192
03:19:09,920 --> 03:19:11,830
PRs that are that are lowrisk and

193
03:19:11,840 --> 03:19:14,150
approved

194
03:19:14,160 --> 03:19:15,750
and this can just free up a lot of

195
03:19:15,760 --> 03:19:17,910
review bandwidth for human reviewers to

196
03:19:17,920 --> 03:19:22,710
review the the more high-risisk changes.

197
03:19:22,720 --> 03:19:25,510
Okay. So now um I'm going to talk about

198
03:19:25,520 --> 03:19:27,670
loop two which is our post deploy

199
03:19:27,680 --> 03:19:32,229
verification loop. Uh so this is I would

200
03:19:32,239 --> 03:19:34,070
say very much in its early stages right

201
03:19:34,080 --> 03:19:35,990
now. People are experimenting with all

202
03:19:36,000 --> 03:19:39,429
kinds of things here. Um but the idea is

203
03:19:39,439 --> 03:19:42,389
that once your code is deployed uh you

204
03:19:42,399 --> 03:19:45,190
can have you you can build AI systems to

205
03:19:45,200 --> 03:19:48,309
check uh to read like production logs,

206
03:19:48,319 --> 03:19:50,469
traces, metrics and see if your code is

207
03:19:50,479 --> 03:19:53,590
actually behaving as you expect it to in

208
03:19:53,600 --> 03:19:55,590
production.

209
03:19:55,600 --> 03:19:59,110
And then from there you can actually um

210
03:19:59,120 --> 03:20:00,389
uh we're starting to see things like

211
03:20:00,399 --> 03:20:03,429
agentic rollout where agents can decide

212
03:20:03,439 --> 03:20:06,229
whether to widen a roll out or roll back

213
03:20:06,239 --> 03:20:09,269
turn a feature flag off

214
03:20:09,279 --> 03:20:11,910
and you can also do uh AI outage

215
03:20:11,920 --> 03:20:15,750
monitoring. So you can uh monitor alerts

216
03:20:15,760 --> 03:20:19,190
and uh use AI systems to help you uh

217
03:20:19,200 --> 03:20:24,790
root cause and fix outages.

218
03:20:24,800 --> 03:20:29,110
Okay. Uh so moving on to loop three. Uh

219
03:20:29,120 --> 03:20:30,550
so this is what I call the maintenance

220
03:20:30,560 --> 03:20:33,510
loop. Uh this is kind of a a parallel

221
03:20:33,520 --> 03:20:35,990
loop that runs alongside your main

222
03:20:36,000 --> 03:20:39,349
software development uh life cycle loop.

223
03:20:39,359 --> 03:20:41,349
And this is where we continuously scan

224
03:20:41,359 --> 03:20:44,070
the codebase on some schedule. And we

225
03:20:44,080 --> 03:20:45,510
check for a lot of the same things that

226
03:20:45,520 --> 03:20:47,429
we check for on the PR. We we could do

227
03:20:47,439 --> 03:20:49,750
code review, static analysis, security

228
03:20:49,760 --> 03:20:53,110
checks, etc. And then we can if we find

229
03:20:53,120 --> 03:20:55,269
an issue, we can open up a remediation

230
03:20:55,279 --> 03:20:59,269
PR to address that. And this lets us

231
03:20:59,279 --> 03:21:01,030
catch things that we couldn't catch in

232
03:21:01,040 --> 03:21:03,030
the PR. So for example, you could

233
03:21:03,040 --> 03:21:04,550
imagine

234
03:21:04,560 --> 03:21:07,910
um maybe a new rule was introduced after

235
03:21:07,920 --> 03:21:10,710
code merged or a new vulnerability was

236
03:21:10,720 --> 03:21:13,670
published after code merged.

237
03:21:13,680 --> 03:21:16,950
um or even something like slow drift

238
03:21:16,960 --> 03:21:19,510
where you you slowly get duplication or

239
03:21:19,520 --> 03:21:21,510
complexity building up that you can't

240
03:21:21,520 --> 03:21:24,150
really see on a single PR. Uh you can

241
03:21:24,160 --> 03:21:27,269
catch these kinds of things here

242
03:21:27,279 --> 03:21:28,790
with these uh with the continuous

243
03:21:28,800 --> 03:21:33,429
codebase scans.

244
03:21:33,439 --> 03:21:35,110
Okay. Okay. And so finally we move over

245
03:21:35,120 --> 03:21:38,070
to the knowledge loop and and this is

246
03:21:38,080 --> 03:21:39,670
becoming we're seeing this become a very

247
03:21:39,680 --> 03:21:42,550
important uh loop for a lot of

248
03:21:42,560 --> 03:21:46,070
companies. Um every company has a lot of

249
03:21:46,080 --> 03:21:50,550
knowledge documentation product specs

250
03:21:50,560 --> 03:21:54,550
um business decisions

251
03:21:54,560 --> 03:21:58,229
um and the agents do a better job if

252
03:21:58,239 --> 03:22:01,559
they have that knowledge. Right? So

253
03:22:01,569 --> 03:22:03,269
[clears throat] this loop basically

254
03:22:03,279 --> 03:22:04,550
collects all that knowledge from

255
03:22:04,560 --> 03:22:06,950
different sources as well as things that

256
03:22:06,960 --> 03:22:09,670
were learned on prior code reviews and

257
03:22:09,680 --> 03:22:13,110
from prior outages. It's curated and

258
03:22:13,120 --> 03:22:16,950
then it can be uh fed into all of the

259
03:22:16,960 --> 03:22:19,750
other loops, all of these other agents

260
03:22:19,760 --> 03:22:25,830
um so they can uh do a better job.

261
03:22:25,840 --> 03:22:27,830
And and for example, this is really

262
03:22:27,840 --> 03:22:29,990
powerful with your coding agents because

263
03:22:30,000 --> 03:22:32,150
if they have some of this knowledge,

264
03:22:32,160 --> 03:22:33,670
they can probably do a better job the

265
03:22:33,680 --> 03:22:37,030
first time, fewer issues.

266
03:22:37,040 --> 03:22:39,110
Um, and then you'll you'll wind up with

267
03:22:39,120 --> 03:22:41,990
basically fewer review laps at the uh

268
03:22:42,000 --> 03:22:47,830
review stage.

269
03:22:47,840 --> 03:22:51,429
Okay. So, what does this mean for our

270
03:22:51,439 --> 03:22:53,990
role as human software engineers in this

271
03:22:54,000 --> 03:22:57,269
factory? Uh, you might be asking, okay,

272
03:22:57,279 --> 03:22:59,670
so if if we nail all of this, can we

273
03:22:59,680 --> 03:23:02,070
just uh turn the lights out and run a

274
03:23:02,080 --> 03:23:04,150
dark factory?

275
03:23:04,160 --> 03:23:06,950
Uh, and I would say probably not. Not

276
03:23:06,960 --> 03:23:09,830
right now. Uh, I like to say the factory

277
03:23:09,840 --> 03:23:12,710
runs dim, not dark. It's still very

278
03:23:12,720 --> 03:23:15,670
important for us to [cough]

279
03:23:15,680 --> 03:23:18,389
intervene in places where our expertise

280
03:23:18,399 --> 03:23:20,870
is is really important.

281
03:23:20,880 --> 03:23:23,429
Uh so that would be things like auditing

282
03:23:23,439 --> 03:23:25,830
risky code.

283
03:23:25,840 --> 03:23:29,349
Um verifying that evidence, verifying

284
03:23:29,359 --> 03:23:32,870
things are actually working as expected.

285
03:23:32,880 --> 03:23:35,190
Handling exceptions like rollbacks and

286
03:23:35,200 --> 03:23:37,830
incidents.

287
03:23:37,840 --> 03:23:41,110
Uh reviewing architecture.

288
03:23:41,120 --> 03:23:43,670
Agents are not very good at this yet, I

289
03:23:43,680 --> 03:23:46,309
would argue.

290
03:23:46,319 --> 03:23:49,030
um and supplying the why or or helping

291
03:23:49,040 --> 03:23:51,590
to build that knowledge that that powers

292
03:23:51,600 --> 03:23:54,309
all these agents and all these loops,

293
03:23:54,319 --> 03:23:56,550
supplying the product intent, the

294
03:23:56,560 --> 03:23:58,469
trade-offs, the business decisions,

295
03:23:58,479 --> 03:24:01,750
things like that.

296
03:24:01,760 --> 03:24:04,229
And of course, we engineer the factory

297
03:24:04,239 --> 03:24:06,309
ourselves, right? We're we've sort of

298
03:24:06,319 --> 03:24:08,070
moved into this era where we're we're

299
03:24:08,080 --> 03:24:09,990
building the software factory as a

300
03:24:10,000 --> 03:24:13,750
whole.

301
03:24:13,760 --> 03:24:17,349
So with that, I hope that gave you sort

302
03:24:17,359 --> 03:24:19,990
of a framework for how to think about um

303
03:24:20,000 --> 03:24:21,670
not just how to build your software

304
03:24:21,680 --> 03:24:24,710
factory uh but how to build a safe

305
03:24:24,720 --> 03:24:28,309
software factory. So thank you. That's

306
03:24:28,319 --> 03:24:30,000
all for me. [applause]