AI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber

Loading slides…

Read the talk

AI Writes Code Faster Than Uber Can Merge It

A 44% rise in hourly merge requests pushed Uber’s 95th-percentile land time from 30 minutes to three or four hours. To bring it back down, the developer platform team changed which requests could run together and which checks could wait until after merge.

From a talk by Preetam Dwivedi and Manjari Akella

At a glance

Ideas worth remembering

  • An edit’s merge cost depends on how much of the build graph it affects. Deferring broad changes protected peak-hour capacity, and persisting the request relieved its author of the retry while leaving the wait for capacity in place.

  • Weak overlaps can tie many requests into one serial group. Relaxing two edges in the six-request example yielded three independent queues. Choosing edges at entry let existing speculative builds remain useful.

  • A verified-main pointer gives consumers a passing commit during a trunk failure. The pointer can advance as soon as a later integrated state passes, while repairing the failure remains a separate operational job.

How two green pull requests break trunk

Uber’s merge queue has to answer a question that a passing pull request leaves open. Will this change still build after someone else’s change lands? Preetam Dwivedi and Manjari Akella work on that problem across six language-based monorepos. More than 5,000 developers contribute over 36,000 commits and about 27,000 pull requests a week across those repositories. The Go monorepo is the largest and the focus of their account.

Take two developers who open PRs A and B against the same commit on trunk. Both builds pass. A merges, and the build for its merge commit passes too. What did B’s green check establish? CI tested B on the original base. When B merges, trunk contains A and B together for the first time, and that build fails. Both developers had good reasons to expect their changes to work. Neither had a test result for the state they actually produced.

SubmitQueue, Uber’s merge queue, brings that combined state into CI before it reaches trunk. With two requests, there are three candidate futures worth considering: A alone, B alone, and A plus B. A speculative build checks one of those futures while the changes are still queued. If the combined build passes, the queue can land the pair with evidence for the state it will create. If it fails, the two individual passes are insufficient to land that combination safely.

0:230:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:04 · section reference included

How SubmitQueue kept a green trunk moving

Testing possible futures creates its own scheduling problem. As more requests arrive, which ones need a combined build? Which speculative build should get resources first? The original queue used three optimizations.

  • Target analysis. The queue identifies the build and test targets a change affects. Two developers working on different services can have disjoint sets of targets. The queue can build and test those changes independently and merge them without making one wait for the other.
  • Probabilistic speculation. Build history and characteristics of the incoming change provide a score for how likely it is to succeed. That score helps prioritize the speculative builds most likely to be needed to land the change. The build itself still provides the pass or fail result.
  • Bypassing large changes. A slow build can already be in progress when a smaller change arrives. If the smaller change can build and land first, the queue lets it pass. Arrival order alone needn’t hold a ready change behind a long build.
3:163:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:51 · section reference included

At 130 requests an hour, the long wait was measured in hours

How much more traffic did it take to turn a half-hour wait into an afternoon? In January 2026, the queue handled roughly 90 requests an hour, with a 95th-percentile land time of 30 minutes. By March it was handling 130 requests an hour, a 44% increase that the team attributed primarily to AI-generated changes. At that rate, P95 land time reached three to four hours. The queue still kept trunk green. The cost showed up in how long developers at the slow end of the distribution waited to merge.

5:465:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:33 · section reference included

Admitting cheap changes without making library owners schedule their own merges

The first change put a gate at the entrance to the active queue. One way to estimate the cost of a request was already available from target analysis. A change affecting a few targets leaves room to build unrelated changes alongside it. A change to a widely used library can affect much of the build graph. It can hold resources while the queue validates it against many other requests. A small edit to that library can therefore be an expensive merge.

During peak hours, the gate admitted cheap changes for the usual builds and tests. It rejected expensive ones and left their authors to retry outside peak hours. The business-hours land rate rose from 130 to about 160 an hour, a gain of over 20%. That policy worked particularly well for developers changing individual services. For a library maintainer, routine work could repeatedly trigger a rejection. The gate had effectively given that maintainer a second job as the scheduler.

A secondary queue took over the retry. When an expensive request arrived during peak hours, the gate persisted it in the deferred queue. Once capacity permitted, for example after the peak, the system replayed that same request into active SubmitQueue. The developer could submit once and go home. The expensive change still had to wait for an opening, but nobody had to wake up early or stay late to press the button.

8:088:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:04 · section reference included

Two weak links tie six requests to one queue

Admission gating controlled what entered the queue. A separate constraint governed what the admitted requests could do. Imagine six requests, A through F, each touching a set of targets. Draw a link between two requests when their sets overlap. A shares a target with B, B shares one with C, and further overlaps connect the rest. A and F can end up in the same connected component without sharing a target themselves. Under the strict conflict rule, the queue schedules the whole component as one serial group.

Conflict relaxation gives each link a strength. SubmitQueue uses Jaccard scoring for dependency overlap, which measures the fraction of distinct targets across two sets that they share. It also accounts for new dependency edges introduced by the changes. The combined score ranges from zero to one, with higher values indicating stronger overlap. The scheduler keeps the stronger links and can drop weaker ones to meet a configured parallelism target. The score measures overlap. An actual build is still needed to find out whether the combined changes work.

Which links would free enough work to run in parallel? In the six-request example, a target of at least 50% parallelism calls for at least three independently working queues. The two weakest edges are B–C and B–E. Dropping those two scheduling constraints splits the single component into three components of two requests each. Each pair still has an ordering constraint inside its own queue, while the three queues can make progress at the same time. The shared targets still exist in the code.

12:2712:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:19 · section reference included

Assign conflicts on arrival to preserve speculative builds

When should the scheduler choose which edges to relax? A global calculation could reconsider the entire conflict graph whenever work arrived or finished. But suppose a speculative build was already running for a combination chosen under the previous graph. Reassigning the requests could make a different combination the one that matters. Frequent updates would make it harder to use work the queue had already scheduled.

Uber chose a local decision at entry. On arrival, a request is compared with everything already in the queue. The scheduler decides which of the arriving request’s edges to keep, then reuses that assignment. The next arrival gets its own comparison. Existing requests keep the relationships that their speculative builds assumed, so those builds can still help the requests land.

Conflict relaxation delivered the largest throughput gain of the three changes. Peak throughput rose from about 160 lands an hour to nearly 250, sometimes higher, an increase of over 50%. It also changed the promise SubmitQueue could make. Two requests separated by a dropped edge could each pass pre-merge validation and land before anything tested their combination. Uber had deliberately reopened the possibility of the A-plus-B failure.

16:2516:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:19 · section reference included

Stovepipe tracks the last trunk state that passed

Return to A and B. SubmitQueue relaxes their conflict, tests each independently, then lands A followed by B. The combined build fails on trunk. Stovepipe, Uber’s post-merge validation system, checks the code as it actually landed. That gives consumers two references on the same history. main follows the latest landed commit. Verified main follows the latest trunk state that Stovepipe has validated successfully.

How far apart can the two references move? In a fresh example, both start at known-good commit A. Then B, C and D land, moving main to D. Stovepipe checks the state produced by B through D on top of A. The build fails, so verified main stays at A. When E, F and G land, the next check covers B through G on top of A, including the changes from the failed check. This time the build passes. Verified main can advance directly from A to G even though the earlier state at D failed.

Keeping a usable pointer takes one action on failure: leave it at the last passing commit. Repairing trunk takes human work. Stovepipe pages the on-call engineer, who prepares a fix or a revert and sends it through SubmitQueue. Once the repair lands, Stovepipe validates the resulting state. A passing check lets verified main move forward again.

Where do the two references point after each build?

main at ALatest landed — Known-good starting commit.
main at DLatest landed — B, C and D have landed on A.
main at GLatest landed — E, F and G have since landed.
Verified main at ALatest verified — Both references start together.
Still at ALatest verified — The integrated state at D fails.
Verified main at GLatest verified — The integrated state at G passes.
Follow the top row for what has landed and the bottom row for what consumers can use as verified. At D the references diverge. The passing check at G brings them back together.
Read the diagram as text
  • main at A. Latest landed — Known-good starting commit.
  • main at D. Latest landed — B, C and D have landed on A.
  • main at G. Latest landed — E, F and G have since landed.
  • Verified main at A. Latest verified — Both references start together.
  • Still at A. Latest verified — The integrated state at D fails.
  • Verified main at G. Latest verified — The integrated state at G passes.
  • main at A → main at D: Land B through D.
  • main at D → main at G: Land E through G.
  • main at D → Still at A: Check B–D on A · fail.
  • Verified main at A → Still at A: Hold pointer.
  • main at G → Verified main at G: Check B–G on A · pass.
  • Still at A → Verified main at G: Advance pointer.
19:2719:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:15 · section reference included

Back to a 30-minute P95, with recovery work still to do

The team’s final peak figure was roughly 300 lands an hour while maintaining a P95 land time of 30 minutes. That is about 233% more lands per hour than January’s baseline of 90, at the same reported P95. SubmitQueue and Stovepipe were both being developed in the open. The next engineering decisions concerned build cost, the scope of a green result, and the time spent recovering from a red one.

  • Batch related changes. SubmitQueue could group related requests so that fewer builds cover the same work. The aim is to land more changes with the available build resources.
  • Track green states by service or project. A single verified-main pointer describes the whole repository. A service-level result could tell a team that its service still passes at a commit where another part of the repository fails. Rollouts of healthy services could then proceed while the affected project recovers.
  • Automate fixes and reverts. The on-call engineer still owns the repair after a failed post-merge check. The team wanted to automate preparing a fix or revert and landing it through SubmitQueue to shorten the time trunk spends broken.

Uber’s hybrid approach gives a merge and a verified commit different meanings. Before merge, SubmitQueue still checks individual changes and the combinations linked by stronger conflicts. After merge, Stovepipe runs exhaustive validation on the state those changes actually produced. A developer can see a change land while consumers who need a passing build continue to use the last verified commit.

6:2323:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:42 · section reference included

Transcript download
Full transcript

Loading transcript…