Rebuilding CI for AI-Scale Commit Volume — Gautam Korlam, Sonar

Hello, everyone. Uh, welcome to my talk. Uh, today we're gonna be talking about how we are rebuilding CI for AI scale commit volume. And this comes from a lot of experience from build and merge infrastructure that I have been working on for a decade plus at this point, and kind of want to walk you through all the new bottlenecks we're seeing in software factories, and also specifically for highly regulated industries and banks. How does that translate? How do we keep things more compliant and auditable

while being able to move fast? So let's dive in. My name is Gautam. I am a co-founder and CTO at Gitar, and recently joined Sonar as a principal engineer after the acquisition. Pre-previous to Gitar, I was at Uber for almost a decade working on build and merge infrastructure for monorepos. So everything from build, CI/CD, IDs, you name it, to make sure developers are productive. And now at Gitar as part of Sonar, I'm trying to make sure that code that's

been checked in is verified and all the code written by agents is of high quality before it goes in. So before I get started, I wanna just paint a bit of history picture. Uh, at Uber, we had agent-scale commit volume before they were agents. So what I mean by this is we had one giant monorepo, and we were doing, like, thousands of changes a day and with very long running builds. This meant that if you just kept everything serialized and merged them

one at a time, so as not to break your, uh, main branch, you would take a, a really long time, up to days, twenty days in this case, to make sure that everything was green. And if we skipped everything and just let everyone merge to main, we would see red main all the time because changes would conflict with each other. And even though the CI would be green on one of them, combined, they might actually break. So we would see, like, our iOS monorepo would be, like, red all the time, and that was not productive for our developers. To fix this, what

we did was we made a system called SubmitQueue, which we have a paper on, and we tested all the likely possibilities of commits that are coming in, uh, and making sure that we can, like, reduce the wait time, so we can test these things in parallel. We could also predict what could break by testing all the combinations. So once we know that a PR might break, we might demote it back from the queue and let the author know that they have to go fix it before they can push it back up after rebasing. And finally, like, to keep

the trunk green, only if it passes with all the combination of commits coming in, it would be let in to go directly onto the trunk. So we'd want to make sure that the overhead was worth it for keeping our trunk green. And this is still running today at Uber. I'm sure now we have, like, merge queues that GitHub came up with and all the other, uh, code hosts. But for back then, there was not a system, so we developed this. So the problem with this approach is currently all these queues that we have, so that's either CI or review

on a centralized bottleneck in the SDLC. So all these queues were built for human speed, and once you build it from human speed, what happens is they are not able to keep up once you go to agents. Because humans generate changes-- used to generate changes, like, few times an hour, so we could do testing and review one at a time. Now, with agents, developers are writing, like, several PRs, uh, in many in a day, and they won't always wait for the queue. They all want to, like, come in at the same time, which means your

CI and review queues are piling up and having pressure. And add to that, like, the patterns of software factories and background agents triggering on, like, various workflows, suddenly you see, like, everything slowing down to a crawl because your bottlenecks are still gonna be CI and review, and they are over capacity in this case.

So what does this mean, uh, for teams today, right? So just to put things into perspective, uh, our own, uh, rate of commits has almost like tenx'd since we started Gitar, uh, about two years ago. And it's just, like, uh, unbelievable with the same size team. We are still very small. We're able to, like, uh, push so much more code. Uh, and we even see this on the industry trends on Gitar. So this graph shows you the number of PRs that we see that Gitar reviews over time. And if you look at the growth trajectory,

it's pretty exponential. Every month, almost, we're growing twenty-five percent month over month of just the amount of PRs we have to review, fix CI, and other things. So this is also, I think, like, corroborated by evidence that, uh, industry players like Gitar have been putting out, where they're seeing, like, uh, staggering growth in PRs. And this is now coming for every single team. Like, if our small team is seeing this, I'm sure, like, the larger teams and banks are seeing it, like, at a much higher degree, uh, as they are going through their AI adoption

journeys. So the problem I was talking about, right? So if agents are opening PRs and if you make them, uh, check everything through the CI system, uh, individually they might pass, but once you kind of merge them back onto main, your trunk might still be broken. So how do you, like, solve for this? Like, how do you keep things green, uh, at this high rate of pace, right? So this is very important, kind of like how we solve SubmitQueue. The changes themselves are not, uh, testing against each other. So there needs to be

some sort of change in either the CI system as well as how we architecture code and in the queues to be smarter to be able to solve this. And the problem we're also seeing with everyone trying to adopt, like, software factories, which is having agents write all the code, uh, from the spec and then just having humans be responsible for the spec and review, is that the factory line that exists today was built for human speed, as I mentioned. Which means that as agents push more changes, uh, every part of the factory

line is gonna be slowing down. So for example, with build and test, you might have lots of rebuilds whenever a compilation breaks, or you might have integration tests. You might have flaky tests that are not reproducible locally, and that just slows everything down. So-- 'cause you have to retry them, and it could take hours in many cases with a complex build. Uh, add to the fact that now you have review and approval. So a lot of you probably already use like AI to review your code. Hopefully, uh, you're not doing every single review with,

uh, with humans. You ideally are reviewing with AI as a first pass. Now the reviewers become the queue if you don't have that system in place. And even after you review, there might be some criteria you might want additionally to make sure that it's approved fully. Because what's happening now in the industry is a lot of folks are just rubber-stamping them because they cannot keep up with the high rate of PRs. They are relying on other signals now to be able to just get through. So the process has to change. And the other problem right now we're having is, like, a lot of these checks are running.

There's, uh, a lot of records that we keep that are expiring. So like your old logs don't exist as they expire out. There is no stamp of, like, what was actually checked because there is, like, just an approval stamp. So there needs to be some record of these things for auditability and compliance checks to happen.

The other issue we're seeing is that as the code volume increases, the time taken for repair also is adding a ton of overhead. Uh, what we see on our platform, at least thirteen percent of PRs had a required CI failure. This is just one or more CI failures, which means in all these cases, there is a separate round trip to go either back to the developer machine or for an agent to understand the logs, root cause the failure, fix it, push another commit, and then keep iterating till it's

green. And this is just CI. Because most of the time, if a human reviewer wants to look at a PR, they're not gonna take a look unless CI is green because that's the first line of defense. So that also, like, slows, uh, the review process because they can't go in parallel, because usually we wanna look at something once it's finalized. And every lab is now paying for a CI review. So if your review comments come in and you're doing another pass, CI might break again, and it kind of like repeats as a cycle. So

how do we make this better? Like, what, what do we do to make this problem more manageable as agents are pushing down more code? So we have discovered this, uh, obviously back at Uber, but also in, uh, in Gitar's team right now. But the most obvious thing is ob-- of course, like make your CI really quick and fast, which means that you may want to, like, remove a lot of the expensive checks, try to offload them to like post, uh, merge even, and try to make the CI step itself really quick because your repairs will be much more efficient. So you can

parallelize things, you can shard, you can do, like, uh, caching. We'll talk about some of the techniques that we have used as well. You also want to make sure the review is fast. So not just like a deep review, but you also want to have maybe multiple passes that are, like, giving quicker signal to your developers, so they're able to, like, use them as a way to understand in what scenarios they may want to go deeper for higher risk changes or maybe for like lower risk ones even, like let the AI approve and merge them. And for the checks that are already running in CI,

if you can move them to the left, it's way cheaper to cache them. And this actually, uh, is very relevant to checks that happen even past CI. For example, you might have like expensive regression tests or like things that might fail in production. Uh, not every time it's possible to, like, move things to the left, but as much as possible, if you can move them, it means you have less bottleneck down the line. And locally, agents have done more context anyway to fix the issue that they're facing. And you also want to have policies. This is very important for regulated industries.

So, uh, a story we have at back at Uber is we lost, like, the license to operate in London because there was a glitch in the software that allowed drivers to get on the platform without all the proper checks, and that was because of a feature flag change. So we had to put in a process in place so that every change affecting London that was going through the SDLC would need some sort of like a, a lineage of, okay, does this change affect London? If so, what are the controls in place? And we were able to win our license back. Now, a lot of you might be

facing these kind of like challenges where agents just make it much harder to understand what's happening in your code base. So how do you ensure your policies are actually being respected? So you need to have a way to, like, enforce these rules beyond CI. Like, just because the code compiles doesn't mean that it's following, uh, particular specifications. It could be like HIPAA. It could be like, uh, uh, TORA. There's so many other policies that these agents are not going to be aware of unless your agents are enforcing it on the other end. And also, you wanna make sure that you

record that it was enforced for auditability. So just a quick story about how we made our CI faster as well. So we were at a baseline where we were, I would say like still fast, but like very annoying when you're trying to, like, push more code. Uh, we were around like ten minutes for our CI, and we wanted to, like, bring that down significantly. So we started think-- looking at how can we cache outputs. Uh, we use Rust for our build system, so we started, like, caching as much as possible and reusing them. And we also looked at ways

to, like, reduce churn in the compiler. So we added some more fixes to, like, ensure that the compilation part is more cacheable and is invalidated less often. And by adding more fixes on top of this, by sharding and by removing, like, these overheads, we are able to bring the time of CI almost like less than a minute for incremental steps, which means once the cold CI is warmed up, incremental builds are way, way quick, quicker, and we can push a ton more code. The other issue is also review. So just because

the CI is green, what was funny was when we were measuring our, uh, P90 time of like PRs making it into production, the vast majority of time was spent in review back at Uber. And even today, to be honest, like most teams spend more time reviewing code and doing other kinds of testing before they are comfortable merging the code into main. So this is something that needs to be addressed on top of CI as well, because right now what's happening is agents are Of course, like,

gen-generating code at the same time, the AI agents are also, like, finding issues on, on your, uh, code review system. And these automated reviews are also findings that you would have to now go back and address. So either you have to pull context locally for your agent or have an agent on CI iterate and fix. And humans are able to review whatever is left after the AI took its first pass. And this needs to be part of, uh, the equation, so you're not wasting precious human cycles on more and more agent-generated PRs.

Uh, here's an example for us. Uh, we were refactoring our code base to have faster CI, so we were using nextest to parallelize our, uh, tests. And we had this, like, shared database that we were spinning up to make sure that we could set it up once and all our database tests would reuse it. But the problem was that there were some guards we didn't put in place, which meant that when CI gets very busy, uh, there is, like, uh, cross-contamination and tests might become flaky. Gitar actually, as a code reviewer,

it caught it on our PR before we merged it, so we didn't have to waste time after. And Gitar also fixed the issue for us because it suggested the fix, committed it, and added the engineer as a co-author. By doing this, we are able to, like, prevent expensive mistakes before the code made it all the way into main. The other thing that we do at Gitar, uh, interestingly, and as we see more and more of our customers start doing, is we are using policies and criteria to decide when is it appropriate to actually even, like,

approve changes. There's a lot of changes that are, like, routine, uh, low risk, documentation, dependency bumps, of course, if they're not breaking any APIs or security patches. Uh, even things like, uh, low-risk changes for internal tooling that can potentially be auto-approved. Uh, so Gitar supports these things out of the box, and you can also add additional custom rules, which are plain English, that you can use to, like, gate the approval and merge on. And we are seeing, like, teams, uh, who are, like, very AI native starting to adopt these

more and more. We see, I think, on our platform, the rate at which folks are adopting, like, the auto-approval and auto-merge grow really quickly across teams, even from large teams, like hundred plus, two thousand plus developers. So here's an example of approval policies. So if it's a dependency bump and doc changes, we should be approving them. Any changes under sensitive stuff like payments or auth might need a human approver on top. And then, of course, you don't want to have the agent who made the change approve its change, because that would not

be, uh, zero trust. So in this case, for-- as agents are generating PRs, the PRs are getting approved. When there is a new change that doesn't meet the criteria, then we would have to escalate to another approver. And this can be done with your own agents, of course, but also Gitar supports this out of the box if you wanna try.

And the other problem with, uh, just relying on CI today is that just because CI is green, as I mentioned, doesn't mean the code is compliant in many cases. So in this case, uh, the CI is passing and there is some metadata that we have about it. On the rerun, the flaky test, uh, is gonna fail, which means that when you force push to fix that thing, now the branch is pointing at another piece of code. So the problem with this is that over time, most CI data is gonna be

expired. So unless you have a record of what happened and what was fixed, you will not be able to, like, be compliant or be able to audit what actually happened. Which is why I think things like agent trajectories and, like, your, uh, outcomes after each push are very, very important. So whatever information you can record on the PR or any other system of record is gonna be a source of truth as your artifacts expire. So the control here is, uh, making sure that the change was approved and tested by someone

who did not write it. If you don't have that information, it could be a problem. So here's an example. We have a pipeline that failed in the left and a pipeline that passed. But the difference here is that if you record properly why the failure occurred when someone else was able to approve it because it was a known failure mode, and it was saved with the change that made it in, this thing actually passes your checks versus something where you don't have a record of what happened, but you just blindly trust the agent pushing code. 'Cause it could

be a test that was actually, uh, not high quality. There was no record of all the compliance checks that were, like, recorded. So you wanna make sure that on the, uh, more sensitive, uh, industries, you want to have all of these decisions recorded as rules or compliance checks on top of your PR in addition to code review.

So the-- What are some of the kinds of records that we want to keep? So here's a few that we keep on Gitar, which I think are just examples. It could be different for folks. So for example, which coding agent made, uh, the PR? Where did the PR originate from? What was the session that originated it? And who actually started it? Could it be a trigger from an automated workflow, or could it be just an engineer doing prompting? And for each of these, like, CIs that run, what is the commit where CI

was trusted? And the same test report, not just the status. And making sure that each finding has, like, a record of if-- how it was resolved. Was it dismissed? Was it followed up? Was it, like, fixed on the PR? And then making sure the approver is someone apart from the bot that was prompted by the same person. And making sure that any overrides are explicitly named. So if there is a red build or, like, a controlled code coverage skipped, you wanna make sure there's a reason for when-- why that check failed and why we're okay with that

in that scenario. And making sure that the code that merges actually has all this and it's not, like, a small change on top that was merged. So all the checks run every single time. So in this case, we wanna make sure that agents cannot approve their own changes. Engineer who prompted it is also not an author. And if you have these practices in place, now we can actually start removing some of these bottlenecks because the CI being faster will unlock your builds, so your

flow is much nicer. And making sure you can, like, move some checks, checks left will also make sure that your test load is gonna be reduced quite a bit on the CI side. If you can automate review for the most part and only escalate to humans for exceptional cases, and potentially even auto-approve low risk or internal changes, then you remove further bottlenecks, and then you have less, uh, that has to reach humans, uh, for a s-software factory to perform to its, uh, potential. And of course,

after every single approval and merge, you wanna record what happened because as factory cycle ramp up, if you don't have an idea of what happened in the past, that context is gonna be very useful on how to, like, iterate and make your software better over time. Uh, that's it from me. Uh, would love for you folks to also try out Gitar and see how you can improve your software factories. And with that, uh, thank you so much for, for your time.