# Building a Safe Software Factory Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [03:10:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11406s) Okay. [03:10:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11408s) All right. Can y'all hear me? [03:10:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11412s) >> All right. My name is Andrew and I think [03:10:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11416s) we're starting now. I don't know if [03:10:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11417s) there's a timer anywhere. Uh my name is [03:10:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11420s) Andrew and I'm excited to talk to you [03:10:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11422s) today about the safe software factory. [03:10:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11427s) Uh but first a little bit about me. Uh I [03:10:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11430s) am a staff software engineer at Sonar. I [03:10:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11434s) work on guitar which is Sonar's AI code [03:10:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11436s) review products. Uh and previously I was [03:10:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11440s) at uh Uber working on the driver trip [03:10:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11443s) experience and navigation. [03:10:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11445s) Uh and then after that I was at a health [03:10:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11447s) tech startup called forward for a little [03:10:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11449s) while uh building a sort of an AI [03:10:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11452s) enabled primary health care system. [03:10:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11459s) Okay. So the software factory this is [03:11:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11463s) like the hottest buzzword in our [03:11:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11466s) industry right now. Everyone's talking [03:11:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11469s) about software factories. Everyone's [03:11:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11470s) trying to build a software factory. [03:11:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11475s) And the question is, well, oh well, so [03:11:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11479s) first of all, [03:11:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11480s) um, [03:11:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11482s) most of you probably know what this is [03:11:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11485s) already. Um, but if you don't, I'll just [03:11:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11488s) give a really quick overview. Uh, this [03:11:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11491s) is sort of the the main software [03:11:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11493s) development life cycle loop uh, as it [03:11:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11496s) existed before the AI era. So we have uh [03:11:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11500s) inputs basically product specs, feature [03:11:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11502s) requests, tickets, things like that. Um [03:11:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11507s) we take those into the planning step if [03:11:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11509s) they're especially if they're [03:11:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11510s) complicated. [03:11:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11512s) Uh and then we then we build then we [03:11:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11514s) write the code [03:11:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11516s) uh and then we um uh put up a PR and [03:12:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11520s) that PR gets uh reviewed and it goes [03:12:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11524s) through CI and then it eventually merges [03:12:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11528s) uh and then we deploy that to production [03:12:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11531s) and then users interact with our [03:12:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11533s) software and they give us feedback or [03:12:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11535s) they find bugs [03:12:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11537s) um or we might also get some monitoring [03:12:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11540s) and alerts as well. And then those go [03:12:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11542s) back in as more inputs and the the cycle [03:12:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11546s) continues [03:12:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11548s) and and so um in the in the era of [03:12:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11552s) agentic coding and AI automation has [03:12:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11556s) followed every bottleneck. So the [03:12:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11558s) bottleneck used to be the the coding [03:12:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11560s) part the building um now with coding [03:12:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11563s) agents that's been automated. Um, and [03:12:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11567s) now the bottleneck has sort of shifted [03:12:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11569s) over to the that review step and we're [03:12:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11572s) starting to introduce uh AI code review [03:12:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11575s) systems for that. Um, and then and we're [03:13:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11580s) seeing uh AI systems and automation [03:13:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11582s) happen really at every step of this now. [03:13:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11585s) Uh so we're starting to see AI systems [03:13:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11588s) for monitoring deployments, controlling [03:13:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11591s) rollouts, triaging uh triaging, you [03:13:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11596s) know, tickets and incidents, planning. [03:13:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11598s) And so this this here is the sort of [03:13:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11601s) software factory that we're that we're [03:13:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11605s) creating. [03:13:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11608s) And and so the question that we all have [03:13:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11611s) now is how do we make this factory safe, [03:13:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11615s) right? Uh in other words, how can we [03:13:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11617s) introduce enough layers of verification [03:13:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11621s) into this so that we can trust that the [03:13:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11623s) software coming out of it is it works as [03:13:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11627s) expected. It's free of bugs, free of [03:13:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11629s) security vulnerabilities. [03:13:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11634s) Um, our CEO [03:13:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11636s) Tar said recently, "Code is abundant, [03:13:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11639s) confidence is not." [03:14:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11641s) So, what I'm going to talk about today [03:14:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11642s) is just how we at Sonar have been [03:14:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11644s) thinking about uh how to introduce uh [03:14:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11648s) confidence [03:14:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11650s) uh at at all sort of all different [03:14:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11652s) points within this software factory. [03:14:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11657s) And uh we're approaching this this [03:14:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11660s) challenge with a defense and depth [03:14:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11662s) approach that comes from adding more [03:14:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11666s) loops. You might not be surprised to [03:14:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11668s) hear [03:14:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11669s) and I'm going to talk about four of them [03:14:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11671s) today in particular. So there's the loop [03:14:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11674s) one which is the PR checks loop. [03:14:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11678s) Uh loop two uh which is the post deploy [03:14:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11682s) verifications loop. [03:14:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11685s) loop three uh which is the maintenance [03:14:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11688s) loop [03:14:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11690s) and loop four which is the knowledge [03:14:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11692s) loop and this is kind of a kind of a [03:14:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11694s) special loop. Uh so first I'm going to [03:14:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11697s) talk about loop one the PR checks loop. [03:15:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11701s) So this is the loop that we probably all [03:15:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11702s) have in some form. Uh you put up a PR, [03:15:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11706s) it goes through some CI checks, it gets [03:15:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11708s) a code review. Nowadays it's probably [03:15:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11710s) getting an AI code review as well. Um, [03:15:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11712s) but we're starting to see a lot more [03:15:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11714s) automation being introduced here as [03:15:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11716s) well. So, we're starting to see agents [03:15:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11719s) pushing fixes to respond to the code [03:15:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11721s) review comments. Um, and this loop kind [03:15:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11725s) of goes around and round until the PR [03:15:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11728s) turns green. [03:15:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11734s) And there's actually a lot more that we [03:15:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11737s) can be doing here. um because catching a [03:15:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11740s) problem at this point is still cheaper [03:15:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11742s) than after it merges into the codebase. [03:15:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11745s) Uh so we have a real opportunity here to [03:15:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11747s) widen this review past the sort of [03:15:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11750s) normal tests and code review that most [03:15:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11753s) of us do. So of course we have AI code [03:15:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11757s) review. We can also do static analysis [03:16:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11760s) here. Many of us do that. [03:16:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11763s) Uh we could do what's called functional [03:16:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11765s) validation. So this is where we can look [03:16:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11767s) at uh what the ticket asks for and see [03:16:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11769s) if the PR actually addresses those [03:16:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11772s) things. [03:16:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11775s) Um we can look at rules that the team [03:16:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11779s) may set for coding conventions [03:16:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11783s) uh see if best practices things like [03:16:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11786s) that see if the PR is adhering to those. [03:16:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11790s) We can look at knowledge. So this can be [03:16:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11792s) knowledge accumulated from company [03:16:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11794s) documentation, [03:16:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11796s) from things that have been learned on [03:16:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11798s) past code reviews or past outages. [03:16:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11803s) Uh we could do cross repo checks. So we [03:16:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11805s) could actually look at other [03:16:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11806s) repositories and see if anything in in [03:16:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11808s) this PR uh will break or cause something [03:16:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11812s) to break in another repository. [03:16:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11816s) Uh we could do cross PR analysis. So, we [03:16:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11819s) can look at all the other open PRs as [03:17:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11821s) well and see if there's anything uh in [03:17:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11825s) this PR that conflicts with one of [03:17:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11827s) those. [03:17:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11830s) We could do security review, check for [03:17:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11832s) vulnerabilities, secrets, dependencies. [03:17:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11837s) We could do endto-end testing. So, for [03:17:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11839s) example, if this is a web app, we could [03:17:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11840s) open up in a browser and and click [03:17:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11842s) around and see if it works as expected. [03:17:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11847s) Um and finally we can do risk [03:17:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11850s) assessment. So we could sort of uh look [03:17:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11852s) at this and compute uh the blast radius [03:17:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11856s) and sort of categorize this into like [03:17:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11858s) low, medium or high risk. And that can [03:17:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11861s) actually help us determine if this is a [03:17:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11863s) PR that a human even needs to look at at [03:17:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11865s) all. [03:17:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11867s) And so all of these things can flow into [03:17:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11869s) this the sort of decision this merge [03:17:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11871s) gate this decision of whether this PR is [03:17:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11873s) mergeable. And of course uh every time a [03:17:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11877s) fix is pushed all of these run again at [03:18:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11880s) least in some degree. [03:18:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11884s) And and with all of this it's important [03:18:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11886s) to gate the that merge decision not just [03:18:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11889s) on the fact that a test ran or that it [03:18:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11891s) passed but on the evidence. So that [03:18:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11893s) would be things like the actual code [03:18:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11895s) review findings, the results of the [03:18:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11897s) quality gates, [03:18:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11899s) uh any vulnerabilities, [03:18:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11902s) um the risk level. So all of that can be [03:18:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11906s) uh basically fed into this merge gate [03:18:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11909s) and checked against criteria that the [03:18:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11911s) team can set for what is a mergeable PR. [03:18:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11915s) And when you record that evidence uh to [03:18:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11918s) the PR, it just makes this uh the merge [03:18:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11921s) gate much stronger. [03:18:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11924s) And so what we're seeing now uh with [03:18:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11927s) teams who do this well uh is that they [03:18:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11929s) can actually delegate more and more [03:18:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11931s) trusts uh to this type of automation on [03:18:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11934s) the PR review. Uh so we're seeing teams [03:18:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11938s) begin to turn on not just automatic [03:19:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11940s) merge blocking but uh even auto approve [03:19:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11945s) in some cases for low-risk changes and [03:19:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11947s) and even automerge for certain types of [03:19:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11949s) PRs that are that are lowrisk and [03:19:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11951s) approved [03:19:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11954s) and this can just free up a lot of [03:19:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11955s) review bandwidth for human reviewers to [03:19:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11957s) review the the more high-risisk changes. [03:19:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11962s) Okay. So now um I'm going to talk about [03:19:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11965s) loop two which is our post deploy [03:19:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11967s) verification loop. Uh so this is I would [03:19:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11972s) say very much in its early stages right [03:19:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11974s) now. People are experimenting with all [03:19:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11976s) kinds of things here. Um but the idea is [03:19:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11979s) that once your code is deployed uh you [03:19:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11982s) can have you you can build AI systems to [03:19:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11985s) check uh to read like production logs, [03:19:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11988s) traces, metrics and see if your code is [03:19:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11990s) actually behaving as you expect it to in [03:19:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11993s) production. [03:19:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11995s) And then from there you can actually um [03:19:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=11999s) uh we're starting to see things like [03:20:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12000s) agentic rollout where agents can decide [03:20:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12003s) whether to widen a roll out or roll back [03:20:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12006s) turn a feature flag off [03:20:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12009s) and you can also do uh AI outage [03:20:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12011s) monitoring. So you can uh monitor alerts [03:20:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12015s) and uh use AI systems to help you uh [03:20:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12019s) root cause and fix outages. [03:20:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12024s) Okay. Uh so moving on to loop three. Uh [03:20:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12029s) so this is what I call the maintenance [03:20:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12030s) loop. Uh this is kind of a a parallel [03:20:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12033s) loop that runs alongside your main [03:20:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12036s) software development uh life cycle loop. [03:20:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12039s) And this is where we continuously scan [03:20:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12041s) the codebase on some schedule. And we [03:20:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12044s) check for a lot of the same things that [03:20:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12045s) we check for on the PR. We we could do [03:20:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12047s) code review, static analysis, security [03:20:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12049s) checks, etc. And then we can if we find [03:20:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12053s) an issue, we can open up a remediation [03:20:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12055s) PR to address that. And this lets us [03:20:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12059s) catch things that we couldn't catch in [03:21:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12061s) the PR. So for example, you could [03:21:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12063s) imagine [03:21:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12064s) um maybe a new rule was introduced after [03:21:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12067s) code merged or a new vulnerability was [03:21:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12070s) published after code merged. [03:21:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12073s) um or even something like slow drift [03:21:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12076s) where you you slowly get duplication or [03:21:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12079s) complexity building up that you can't [03:21:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12081s) really see on a single PR. Uh you can [03:21:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12084s) catch these kinds of things here [03:21:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12087s) with these uh with the continuous [03:21:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12088s) codebase scans. [03:21:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12093s) Okay. Okay. And so finally we move over [03:21:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12095s) to the knowledge loop and and this is [03:21:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12098s) becoming we're seeing this become a very [03:21:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12099s) important uh loop for a lot of [03:21:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12102s) companies. Um every company has a lot of [03:21:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12106s) knowledge documentation product specs [03:21:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12110s) um business decisions [03:21:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12114s) um and the agents do a better job if [03:21:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12118s) they have that knowledge. Right? So [03:22:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12121s) [clears throat] this loop basically [03:22:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12123s) collects all that knowledge from [03:22:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12124s) different sources as well as things that [03:22:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12126s) were learned on prior code reviews and [03:22:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12129s) from prior outages. It's curated and [03:22:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12133s) then it can be uh fed into all of the [03:22:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12136s) other loops, all of these other agents [03:22:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12139s) um so they can uh do a better job. [03:22:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12145s) And and for example, this is really [03:22:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12147s) powerful with your coding agents because [03:22:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12150s) if they have some of this knowledge, [03:22:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12152s) they can probably do a better job the [03:22:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12153s) first time, fewer issues. [03:22:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12157s) Um, and then you'll you'll wind up with [03:22:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12159s) basically fewer review laps at the uh [03:22:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12162s) review stage. [03:22:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12167s) Okay. So, what does this mean for our [03:22:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12171s) role as human software engineers in this [03:22:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12174s) factory? Uh, you might be asking, okay, [03:22:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12177s) so if if we nail all of this, can we [03:22:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12179s) just uh turn the lights out and run a [03:23:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12182s) dark factory? [03:23:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12184s) Uh, and I would say probably not. Not [03:23:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12186s) right now. Uh, I like to say the factory [03:23:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12189s) runs dim, not dark. It's still very [03:23:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12192s) important for us to [cough] [03:23:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12195s) intervene in places where our expertise [03:23:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12198s) is is really important. [03:23:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12200s) Uh so that would be things like auditing [03:23:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12203s) risky code. [03:23:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12205s) Um verifying that evidence, verifying [03:23:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12209s) things are actually working as expected. [03:23:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12212s) Handling exceptions like rollbacks and [03:23:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12215s) incidents. [03:23:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12217s) Uh reviewing architecture. [03:23:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12221s) Agents are not very good at this yet, I [03:23:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12223s) would argue. [03:23:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12226s) um and supplying the why or or helping [03:23:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12229s) to build that knowledge that that powers [03:23:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12231s) all these agents and all these loops, [03:23:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12234s) supplying the product intent, the [03:23:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12236s) trade-offs, the business decisions, [03:23:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12238s) things like that. [03:24:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12241s) And of course, we engineer the factory [03:24:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12244s) ourselves, right? We're we've sort of [03:24:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12246s) moved into this era where we're we're [03:24:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12248s) building the software factory as a [03:24:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12250s) whole. [03:24:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12253s) So with that, I hope that gave you sort [03:24:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12257s) of a framework for how to think about um [03:24:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12260s) not just how to build your software [03:24:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12261s) factory uh but how to build a safe [03:24:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12264s) software factory. So thank you. That's [03:24:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=12268s) all for me. [applause]