1
00:00:04,380 --> 00:00:12,560
 Hello everyone. We are here to present our talk, merging at AI scale. This is essentially

2
00:00:12,560 --> 00:00:21,280
 Uber's journey from going to a fully green trunk to somewhat broken trunk. I'm Preetam

3
00:00:21,280 --> 00:00:25,480
 Dwivedi. I'm a staff engineer at Uber working in developer platform.

4
00:00:25,480 --> 00:00:31,960
 Hi, I'm Manjiri. I'm a senior engineer working alongside Preetam in the developer plan at Uber.

5
00:00:32,500 --> 00:00:36,100
 Great. So here's the outline for the talk.

6
00:00:36,400 --> 00:00:40,680
 We're going to go over how the merging works at Uber at the moment,

7
00:00:41,280 --> 00:00:44,460
 and then how AI changed the landscape,

8
00:00:44,720 --> 00:00:48,620
 and then how to respond to that, how we adapt it,

9
00:00:48,780 --> 00:00:52,860
 and then discuss the results and talk about what are we planning next.

10
00:00:52,860 --> 00:01:05,320
 So at Uber, we have six language-based monorepos, where more than 5,000 developers are contributing

11
00:01:05,320 --> 00:01:11,100
 over 36,000 commits, 27,000 PRs every week.

12
00:01:12,340 --> 00:01:18,880
 And pretty much every changes to the services libraries are being done in one of these monorepos.

13
00:01:19,860 --> 00:01:24,000
 For the purpose of this talk, we're going to focus mostly on Go code when we talk about

14
00:01:24,000 --> 00:01:28,600
 anything because that is the biggest one and that's where the most of the scaling challenges

15
00:01:28,600 --> 00:01:38,240
 we face. So let's understand the problem a little bit and understand why we need a merge queue here.

16
00:01:38,820 --> 00:01:48,100
 So given here, basically we have two users, user A and B, grading PR A and B on a base edge from

17
00:01:48,100 --> 00:01:54,140
 the trunk and when the PRs are created and CI is run, the CI is green, builds are passing,

18
00:01:54,320 --> 00:02:02,820
 everything is good. Now when A is merged into the trunk, we can see the builds for the merge

19
00:02:02,820 --> 00:02:09,960
 commit are still green, builds are still passing, but when the PR B is merged into the trunk,

20
00:02:10,820 --> 00:02:16,200
 builds are no longer green. And the reason is that B was never tested with the changes that

21
00:02:16,200 --> 00:02:24,280
 were merged by A. So we never tested the combination of those two things before merging,

22
00:02:24,620 --> 00:02:32,880
 and hence when B is merged, it's broken. Now imagine when we have thousands of

23
00:02:32,880 --> 00:02:40,220
 engineers working in a monorepo and everyone is making changes, like hundreds and thousands of

24
00:02:40,220 --> 00:02:48,540
 those every day. This is a constant problem where trunk is broken very frequently and there is a

25
00:02:48,540 --> 00:02:56,380
 burden on fixing and there is a toll across the company to sort of figure it out and fix it.

26
00:02:57,060 --> 00:03:03,020
 So here we're going to present a solution to that problem and that solution is Summit Queue.

27
00:03:03,640 --> 00:03:09,520
 That is our Uber's merge queue that we deploy to solve this problem.

28
00:03:09,720 --> 00:03:13,800
 And in this case, we can see our two examples here, PRA and PRAB.

29
00:03:14,500 --> 00:03:21,060
 And here, essentially, what we are doing is we are predicting the futures that could be true on main.

30
00:03:21,240 --> 00:03:23,460
 So there are three things here.

31
00:03:23,800 --> 00:03:26,700
 A could land on its own to the main.

32
00:03:26,880 --> 00:03:28,960
 B could land on its own to the main.

33
00:03:29,200 --> 00:03:31,120
 And then the combination of A and B.

34
00:03:31,700 --> 00:03:37,820
 In general, I think what we're going to do is that we want to land the change that is going to keep the trunk green as a whole.

35
00:03:38,080 --> 00:03:41,620
 So in this case, when we land A and B together, the trunk is green.

36
00:03:43,220 --> 00:03:49,580
 And that's how basically SummitQ works and keeps the trunk green for our monorepos.

37
00:03:50,620 --> 00:03:53,880
 Now, how does it scale in this version of it?

38
00:03:54,080 --> 00:03:56,400
 There are three things that SummitQ employs.

39
00:03:57,000 --> 00:03:58,900
 The very first thing is target analysis.

40
00:03:58,900 --> 00:04:06,780
 What it means is when two users are working across two different services, they don't really have any overlapping changes.

41
00:04:07,220 --> 00:04:12,820
 So those two things can be built and tested independently of each other and then can be merged.

42
00:04:13,040 --> 00:04:18,880
 So this allows us to basically get higher throughput and not serialize every change that is entering into the queue.

43
00:04:20,220 --> 00:04:26,620
 Second is probabilistic speculation. What it means is we are using historical

44
00:04:27,360 --> 00:04:34,580
 the build information and also the the change characteristics to predict what is the success

45
00:04:34,580 --> 00:04:41,140
 score for a given change. And then we are using that to basically decide what our builds are

46
00:04:41,140 --> 00:04:46,760
 likely needed in order to land that change. And so in that case we prioritize the builds that are

47
00:04:46,760 --> 00:04:50,060
 most likely needed and are important for that change

48
00:04:50,060 --> 00:04:53,080
 to be much to the trunk rather than like every other build

49
00:04:53,080 --> 00:04:54,700
 that comes from the speculation.

50
00:04:55,540 --> 00:04:58,340
 The third part is bypassing the large changes.

51
00:04:58,560 --> 00:05:00,720
 What it means is that if we have a large change

52
00:05:00,720 --> 00:05:03,300
 sitting in the queue, which is taking longer to build,

53
00:05:03,420 --> 00:05:06,300
 and at the same time if we have a smaller changes entering

54
00:05:06,300 --> 00:05:09,340
 into the queue, those can be built faster.

55
00:05:09,680 --> 00:05:12,360
 And so those changes don't need to wait for large changes

56
00:05:12,360 --> 00:05:13,940
 to land before they can land.

57
00:05:14,160 --> 00:05:16,380
 So in this case, we can land the smaller changes

58
00:05:16,380 --> 00:05:18,680
 without landing the larger changes first.

59
00:05:18,960 --> 00:05:22,280
 And this optimization allows us to basically move fast

60
00:05:22,280 --> 00:05:23,860
 and get a higher throughput of the queue.

61
00:05:24,260 --> 00:05:26,360
 So this is what has worked

62
00:05:27,480 --> 00:05:32,080
 when it was just humans landing changes, creating PRs.

63
00:05:33,500 --> 00:05:36,080
 But then we had AI,

64
00:05:36,240 --> 00:05:41,520
 and the changes at which the PRs were created and landed,

65
00:05:41,720 --> 00:05:42,720
 it all changed.

66
00:05:43,240 --> 00:05:44,800
 So to understand it better,

67
00:05:45,020 --> 00:05:46,140
 let's look at some of the numbers.

68
00:05:46,380 --> 00:05:54,620
 So in January 2026, we were doing roughly 90 requests every hour.

69
00:05:55,460 --> 00:05:59,620
 And by March, we were doing 130 requests every hour.

70
00:05:59,860 --> 00:06:02,940
 So that's roughly a 44% increase in two months.

71
00:06:03,360 --> 00:06:08,600
 And the reason for this increase is primarily because of the AI-generated changes.

72
00:06:08,600 --> 00:06:18,180
 Now, given the way the system was built, the rate of these changes had effect on it.

73
00:06:18,520 --> 00:06:26,960
 So looking into those, we had 30 minutes of P95 line times when we were doing 90 requests every hour.

74
00:06:27,740 --> 00:06:34,620
 And that went to roughly three to four hours when we were doing 130 requests every hour.

75
00:06:34,620 --> 00:06:45,840
 So in this case, we were still keeping the trunk green at all times, but at the cost of developers essentially waiting for three to four hours to land their changes to the trunk.

76
00:06:46,940 --> 00:06:54,740
 Now I'm going to hand over to Manjri, and she's going to talk about our response to it and how we adapted to the situation.

77
00:06:55,880 --> 00:06:56,920
 Great. Thanks, Pritam.

78
00:06:56,920 --> 00:07:06,780
 So, Pritam just walked us through a few optimizations that SubmitQ had that worked so far in keeping up with the demand that humans generated.

79
00:07:07,180 --> 00:07:12,440
 But once AI came into the picture, the optimizations we had were no longer sufficient.

80
00:07:12,800 --> 00:07:20,540
 So, our response essentially was another set of optimizations to our existing SubmitQ.

81
00:07:20,740 --> 00:07:23,380
 And we approached the response in three different ways.

82
00:07:23,380 --> 00:07:31,140
 The first was we introduced this concept of admission gating, which helped us to shape the traffic that we accept into the queue.

83
00:07:31,680 --> 00:07:43,680
 Next, we introduced the concept of deferrals, which is a secondary queue that we introduced that helps with developer experience from the pain that comes with admission gating.

84
00:07:43,680 --> 00:07:53,880
 And lastly, we implemented what we call conflict relaxation, which helps us buy back more parallelism within the queue itself.

85
00:07:54,680 --> 00:08:03,420
 Next, we will look at each of these interventions in a little bit of depth and understand exactly what each of them are doing.

86
00:08:03,740 --> 00:08:08,160
 So the first one, admission gating, is basically what it sounds like.

87
00:08:08,280 --> 00:08:12,580
 There is a gate at the admission of every request into the main submit queue.

88
00:08:12,580 --> 00:08:18,880
 we look at the request that comes in and we categorize it as one of two things.

89
00:08:19,120 --> 00:08:22,040
 Either it's a small change or it's an expensive change.

90
00:08:22,260 --> 00:08:25,040
 And the definition of this could be anything.

91
00:08:25,320 --> 00:08:31,920
 One of the definitions that we use is the target analysis that Pritam talked about earlier in the slides.

92
00:08:32,580 --> 00:08:35,520
 A small change touches fewer targets.

93
00:08:35,520 --> 00:08:41,740
 an expensive change might be something that spans across multiple targets,

94
00:08:41,920 --> 00:08:43,820
 the entire build graph, something like that.

95
00:08:44,040 --> 00:08:49,320
 So a smaller change is more likely to give you higher parallelism.

96
00:08:49,580 --> 00:08:54,420
 And we basically say admit it during peak hours and add it to the active

97
00:08:54,420 --> 00:08:58,880
 submit queue, build and test as usual and move along.

98
00:08:59,180 --> 00:09:04,800
 But an expensive change has the capability to basically be a bottleneck

99
00:09:04,800 --> 00:09:10,940
 by holding resources needing to validate against everything else that's in the queue.

100
00:09:11,260 --> 00:09:21,520
 So to protect the rest of the queue, what we do in this phase is just to reject any expensive change during peak hours.

101
00:09:21,740 --> 00:09:28,480
 What this means is that the developer has to come back and retry their expensive change outside peak hours.

102
00:09:28,480 --> 00:09:39,520
 So this gives us control into what enters the queue by moving the burden onto a very small subset of changes and developers.

103
00:09:40,980 --> 00:09:43,600
 Let's look at the trade-off of admission gating.

104
00:09:44,040 --> 00:09:48,860
 After we implemented this, we definitely observed an increase in our throughput.

105
00:09:49,080 --> 00:09:51,600
 We were able to achieve over 20% increase.

106
00:09:51,920 --> 00:09:57,420
 Our land rate went from 130 to about 160 during business hours.

107
00:09:57,420 --> 00:10:04,120
 but it came at a cost. And the cost is that developers become the schedulers.

108
00:10:04,580 --> 00:10:10,200
 Not all developers, since we do protect the smaller changes, but you can imagine if you are

109
00:10:10,200 --> 00:10:18,940
 a developer who owns any types of libraries that span the entire code, you will be particularly

110
00:10:18,940 --> 00:10:25,360
 targeted in this because the work that you do naturally falls into the expensive change bucket.

111
00:10:25,360 --> 00:10:31,920
 So we managed to protect the capacity during peak hours, but had to make developers wait.

112
00:10:32,880 --> 00:10:33,720
 Next slide.

113
00:10:34,080 --> 00:10:44,480
 So the next thing that we do is targeted more towards bettering the developer experience of the cons that come from admission gating.

114
00:10:45,160 --> 00:10:48,980
 What if instead of outright rejecting expensive changes,

115
00:10:49,360 --> 00:10:54,040
 we are able to record them and replay them back into the main queue

116
00:10:54,880 --> 00:10:57,520
 based on the status of the queue

117
00:10:57,520 --> 00:11:01,140
 when we have, let's say, a better handle on the capacity.

118
00:11:01,580 --> 00:11:03,920
 And that's exactly what deferrals implement.

119
00:11:04,160 --> 00:11:07,140
 During peak hours, if an expensive request comes in,

120
00:11:07,360 --> 00:11:11,020
 we no longer reject it, but put it into a deferred queue

121
00:11:11,020 --> 00:11:12,240
 where it is perfected.

122
00:11:13,180 --> 00:11:20,440
 and once the gates permit, usually upon, for example, we can say after peak hours,

123
00:11:20,740 --> 00:11:25,920
 we put this persisted request back onto the active submit queue.

124
00:11:26,360 --> 00:11:32,620
 So essentially, we shape the traffic in a way where we release more expensive changes

125
00:11:32,620 --> 00:11:35,380
 when the queue is able to handle it.

126
00:11:35,560 --> 00:11:42,160
 So what this solves is a developer submits once and then we automatically release it for them

127
00:11:42,160 --> 00:11:43,720
 when the capacity allows.

128
00:11:44,420 --> 00:11:45,040
 Next slide.

129
00:11:45,320 --> 00:11:47,200
 So the trade-off here is

130
00:11:47,200 --> 00:11:49,640
 it solves the problem for the developer.

131
00:11:49,900 --> 00:11:51,300
 They don't have to wake up early,

132
00:11:51,560 --> 00:11:52,240
 stay up late,

133
00:11:52,720 --> 00:11:54,700
 try to find a particular window

134
00:11:54,700 --> 00:11:56,140
 where the queue looks good

135
00:11:56,140 --> 00:11:58,380
 and hit the button on their end.

136
00:11:58,500 --> 00:12:00,320
 We will handle that for them.

137
00:12:00,480 --> 00:12:02,700
 But the other side of this,

138
00:12:02,840 --> 00:12:04,720
 the path that comes with this

139
00:12:04,720 --> 00:12:06,940
 is that it doesn't really solve

140
00:12:06,940 --> 00:12:08,940
 the waiting for the developer.

141
00:12:09,300 --> 00:12:10,500
 You still have to wait

142
00:12:10,500 --> 00:12:11,800
 the same amount of time

143
00:12:11,800 --> 00:12:16,000
 for your expensive change to go into the queue.

144
00:12:16,920 --> 00:12:17,800
 Next slide.

145
00:12:18,220 --> 00:12:23,380
 So before we look at the next and final optimization that we did,

146
00:12:23,600 --> 00:12:29,780
 I want to touch a little bit on the concept of what a target overlap is

147
00:12:29,780 --> 00:12:33,280
 and how that serializes work in some way.

148
00:12:33,600 --> 00:12:37,640
 So recall that Fritam mentioned the target analysis that we perform

149
00:12:37,640 --> 00:12:42,680
 to determine whether two requests should be serialized

150
00:12:42,680 --> 00:12:44,940
 or can they go in parallel.

151
00:12:45,280 --> 00:12:47,660
 So an example that we are looking at here

152
00:12:47,660 --> 00:12:49,960
 is a strict one connected component.

153
00:12:50,180 --> 00:12:52,940
 What this means is that each of these requests,

154
00:12:53,100 --> 00:12:56,700
 A through F, touch some subset of targets

155
00:12:56,700 --> 00:13:00,680
 and there's at least one overlap of targets

156
00:13:00,680 --> 00:13:02,580
 between each pair of views.

157
00:13:02,720 --> 00:13:05,920
 So like A and B overlap on at least one target,

158
00:13:05,920 --> 00:13:13,780
 B and C overlap at least one target, similarly for C, D, E, and F. And what this makes SubmitQ do is

159
00:13:13,780 --> 00:13:19,660
 group all of these requests into a single component. And that means that all of these

160
00:13:19,660 --> 00:13:28,320
 have to go together. And this is basically what prevents us from doing work faster.

161
00:13:29,020 --> 00:13:34,040
 and the next optimization we do is how do we break this?

162
00:13:34,280 --> 00:13:37,080
 Is there a way where we can break this up in a smart way?

163
00:13:37,820 --> 00:13:38,700
 Next slide.

164
00:13:39,780 --> 00:13:43,560
 So conflict relaxation is what helps us break these links

165
00:13:43,560 --> 00:13:45,400
 in an informed fashion.

166
00:13:46,240 --> 00:13:49,400
 Before is basically the status quo.

167
00:13:49,660 --> 00:13:52,860
 Any relevant overlap between two requests

168
00:13:52,860 --> 00:13:55,560
 makes them conflict with each other

169
00:13:55,560 --> 00:13:59,020
 and makes them serialize one behind the other.

170
00:13:59,280 --> 00:14:03,480
 But with relaxation, we essentially do three steps.

171
00:14:03,680 --> 00:14:05,340
 First, we rank the overlap.

172
00:14:05,400 --> 00:14:08,340
 So it's no longer a binary decision of,

173
00:14:09,320 --> 00:14:12,600
 is there at least one overlap between these two requests?

174
00:14:12,960 --> 00:14:16,580
 Think of it more as a, how strong is this overlap?

175
00:14:17,260 --> 00:14:20,380
 Once we have that, we keep the stronger links

176
00:14:20,380 --> 00:14:25,020
 and relax the weaker ones to buy back parallelism

177
00:14:25,020 --> 00:14:28,460
 based on a config value that we set.

178
00:14:29,040 --> 00:14:29,720
 Next slide.

179
00:14:30,120 --> 00:14:33,340
 So the first step is essentially scoring these edges

180
00:14:33,340 --> 00:14:35,200
 or these overlaps.

181
00:14:35,420 --> 00:14:36,920
 We use a Jaccard scoring

182
00:14:36,920 --> 00:14:39,660
 for computing this dependency overlap.

183
00:14:39,940 --> 00:14:42,260
 We basically have overlapping targets.

184
00:14:42,540 --> 00:14:44,080
 We check for that

185
00:14:44,080 --> 00:14:46,940
 and any new dependency edges that are introduced.

186
00:14:46,940 --> 00:14:49,760
 We compute a combination of these two

187
00:14:49,760 --> 00:14:53,220
 into a continuous valued score between 0 and 1

188
00:14:53,220 --> 00:14:57,840
 where a higher value of score indicates a stronger overlap

189
00:14:57,840 --> 00:15:02,080
 and a lower value of the score indicates a weaker edge

190
00:15:02,080 --> 00:15:05,360
 with fewer targets that overlap between the requests.

191
00:15:06,120 --> 00:15:06,580
 Next slide.

192
00:15:06,820 --> 00:15:08,600
 Once we are armed with this knowledge,

193
00:15:08,800 --> 00:15:12,940
 we basically make this conflict graph again, right?

194
00:15:13,040 --> 00:15:16,140
 So taking the same example and adding some numbers

195
00:15:16,140 --> 00:15:20,060
 to the conflicts that we have from the earlier slide.

196
00:15:20,480 --> 00:15:22,120
 Let's say we have these six changes

197
00:15:22,120 --> 00:15:29,700
 and the config value we set says we want at least 50% parallelism in the queue at all times.

198
00:15:29,840 --> 00:15:34,800
 What this means is that we need at least three separate queues

199
00:15:34,800 --> 00:15:39,140
 that are working independent of each other.

200
00:15:39,880 --> 00:15:45,920
 And based on the overlap scores that we have computed,

201
00:15:46,220 --> 00:15:50,300
 we are able to make this choice of dropping certain edges

202
00:15:50,300 --> 00:15:55,580
 in order to be able to achieve that target queue size that we have.

203
00:15:55,740 --> 00:16:00,660
 So in this case, the weakest edges are between B and C and D and E.

204
00:16:01,000 --> 00:16:06,140
 And we drop those two, which gives us three separate connected components.

205
00:16:06,720 --> 00:16:12,440
 And this is how we essentially go from a serial chain of six requests

206
00:16:12,440 --> 00:16:18,520
 to a parallelized chain where two each exist in three separate queues.

207
00:16:19,220 --> 00:16:20,060
 Next slide.

208
00:16:20,300 --> 00:16:29,260
 So next comes a question of where exactly do we make this determination of dropping the edges, right?

209
00:16:29,420 --> 00:16:31,940
 So there are two places where we can do this.

210
00:16:32,600 --> 00:16:34,720
 One is at a global level.

211
00:16:35,200 --> 00:16:44,680
 Every time we are trying to assign conflict drafts and building them, we can basically recompute this every time.

212
00:16:44,680 --> 00:16:58,360
 And what that does is it introduces some level of instability into the graph, which is determined by any new request that enter the queue, any work that finishes and exits the queue.

213
00:16:58,360 --> 00:17:02,060
 So essentially, we can do that.

214
00:17:02,240 --> 00:17:09,140
 But what we chose to do instead was a more local assignment of overlaps.

215
00:17:09,420 --> 00:17:16,200
 And the place that we choose to do this at is only at the new arrival checkpoint.

216
00:17:16,560 --> 00:17:25,640
 So once a new request comes in, we essentially take that and compute the edges of this with everything that exists in the queue,

217
00:17:25,640 --> 00:17:33,060
 make the decisions and that's how it stays. The next decision making will only happen when another

218
00:17:33,060 --> 00:17:40,120
 new arrival comes in. So the concept is simple. We assign once at entry time and basically reuse this.

219
00:17:40,720 --> 00:17:49,380
 What this helps us do is any work that we schedule based on previous conflict graphs that we have

220
00:17:49,380 --> 00:17:57,300
 created is preserved and it helps us hang on to speculative builds that can help us move faster

221
00:17:57,300 --> 00:18:06,780
 towards a finalization of its best. So the trade-off of this final intervention was the

222
00:18:06,780 --> 00:18:13,880
 biggest that we saw. We saw over 50% increase in the throughput. So we went from about 160

223
00:18:13,880 --> 00:18:21,780
 lands per hour to almost 250, sometimes even greater during peak hours. So it worked in that

224
00:18:21,780 --> 00:18:30,420
 it not only helped us significantly improve our throughputs while giving us extremely good

225
00:18:30,420 --> 00:18:37,980
 P95 land times, but it came at a cost as everything does. And the trade-off that we have given away

226
00:18:37,980 --> 00:18:44,460
 here is basically what Pritham alluded to in the very first slide talking about the integration

227
00:18:44,460 --> 00:18:53,160
 testing. Some combinations of changes will land untested because we are dropping edges from our

228
00:18:53,160 --> 00:19:01,700
 conflict graph. What this means is main can now be broken and this creates essentially a new

229
00:19:01,700 --> 00:19:10,640
 obligation where we have to validate what lands after it has landed. And to talk a little bit more

230
00:19:10,640 --> 00:19:19,180
 about how we do this, I will hand it back to Pritam. Great. Thanks Manjri for walking us through

231
00:19:20,160 --> 00:19:27,560
 those three solutions that we employed. So taking back this example that we had initially,

232
00:19:27,560 --> 00:19:38,240
 We had basically two changes here, A and B, and say we do a pre-merge validation independently after relaxing those conflict between those two things.

233
00:19:39,200 --> 00:19:40,020
 And then say we landed.

234
00:19:40,440 --> 00:19:50,340
 So now the landed state is essentially A lands first, then B, and now the combination of A plus B fails on the truck.

235
00:19:51,080 --> 00:19:55,420
 So this is where we started at the start of this talk.

236
00:19:55,420 --> 00:19:58,660
 And so now we have a broken trunk again.

237
00:19:59,700 --> 00:20:02,180
 So now how are we dealing with this?

238
00:20:03,220 --> 00:20:06,720
 So we have a system here called StowPipe.

239
00:20:07,080 --> 00:20:11,900
 And StowPipe is our post-land validation system.

240
00:20:12,460 --> 00:20:15,740
 It's responsible for validating any commit that lands on

241
00:20:15,740 --> 00:20:18,440
 to the main and then making sure that

242
00:20:19,180 --> 00:20:23,580
 it detects basically if builds are broken. And if so, then

243
00:20:23,580 --> 00:20:28,280
 and it creates a recovery strategy for that.

244
00:20:28,900 --> 00:20:32,460
 Now, here we have basically two things.

245
00:20:32,960 --> 00:20:36,080
 One is every change that is being landed by SummitQ

246
00:20:36,840 --> 00:20:38,460
 lands onto the main trunk.

247
00:20:38,780 --> 00:20:41,080
 And then we have the second reference here,

248
00:20:41,380 --> 00:20:45,800
 or the pointer, if you call, is the verified main.

249
00:20:45,920 --> 00:20:47,660
 And the verified main is essentially the one

250
00:20:47,660 --> 00:20:51,780
 that is being managed by StowPipe.

251
00:20:51,780 --> 00:20:54,640
 So let's take an example to understand it better.

252
00:20:54,920 --> 00:21:01,700
 So imagine we have this commit A where main and verified main both are pointing to it.

253
00:21:01,820 --> 00:21:02,980
 This is a good known commit.

254
00:21:03,640 --> 00:21:05,200
 This is building.

255
00:21:05,460 --> 00:21:07,120
 This is green on the trunk.

256
00:21:07,720 --> 00:21:14,220
 Now let's say we have another set of commits that land onto the main, say B, C, and D.

257
00:21:14,840 --> 00:21:18,840
 Now main moves to point to the D commit.

258
00:21:18,840 --> 00:21:25,120
 So at this point, a store pipe is going to take this range of commits, essentially B, C, and D,

259
00:21:25,340 --> 00:21:35,760
 on top of A and validate these newly landed changes to see if this new state is still buildable and is still passing or not.

260
00:21:36,280 --> 00:21:38,580
 In this case, we can see it fails.

261
00:21:38,920 --> 00:21:44,320
 And so now a store pipe is not going to move the verified main pointer in this case.

262
00:21:44,320 --> 00:21:50,040
 it will just keep it at A because that's what the good known last commit is on the trunk.

263
00:21:51,120 --> 00:22:00,080
 Now over time more commits land under the trunk and main moves to commit G. So now between the

264
00:22:00,080 --> 00:22:07,200
 new state that we have G and the previously good known state which is A, we have more commits in

265
00:22:07,200 --> 00:22:15,000
 between. So we have B, C, D, E, F, and G commits in between. So now store pipe is going to take

266
00:22:15,000 --> 00:22:21,260
 those commit range and basically validate and see if those are buildable and if builds are passing.

267
00:22:22,180 --> 00:22:29,240
 So now say in this case builds are passing and so the store pipe will basically take the verified

268
00:22:29,240 --> 00:22:36,960
 main pointer and point it to the newly validated good known commit G in this case.

269
00:22:37,200 --> 00:22:38,920
 So that's what a store pipe does.

270
00:22:39,240 --> 00:22:43,860
 It is essentially observing the newly landed commits onto the main,

271
00:22:44,200 --> 00:22:46,080
 verifying if they can build or not.

272
00:22:46,260 --> 00:22:48,480
 And if they pass, move the pointer.

273
00:22:48,700 --> 00:22:54,280
 If not, then continuously look for more changes and keep validating.

274
00:22:55,260 --> 00:23:01,740
 So now let's look at when a commit landed on main breaks the trunk.

275
00:23:02,180 --> 00:23:04,200
 So what happens in that case?

276
00:23:04,200 --> 00:23:09,220
 As I said earlier, StowPipe is essentially validating and then detecting that it failed.

277
00:23:09,440 --> 00:23:11,520
 And on failure, it basically alerts.

278
00:23:11,980 --> 00:23:13,380
 So OnCall gets paged.

279
00:23:13,980 --> 00:23:25,900
 And as a response to this broken trunk, OnCall is responsible to prepare a fix or revert that change, get it reviewed, and then land through SummitQ.

280
00:23:25,900 --> 00:23:29,760
 and then eventually it merges back into the trunk

281
00:23:29,760 --> 00:23:33,420
 and then a stovepipe continuously validates the new state

282
00:23:33,420 --> 00:23:36,060
 to see if it has been fixed or not

283
00:23:36,060 --> 00:23:40,560
 and then accordingly move the pointer to the newly landed comment.

284
00:23:42,380 --> 00:23:48,280
 So with all of this, I want to just recap on basically what we have achieved.

285
00:23:49,240 --> 00:23:55,780
 So at peak times, we are able to do roughly 300 Lens per hour

286
00:23:55,780 --> 00:24:00,140
 we are still able to maintain our P95 land time to 30 minutes.

287
00:24:00,580 --> 00:24:08,120
 And this is essentially a 233% increase since the January numbers that we have,

288
00:24:08,300 --> 00:24:10,500
 which were like 90 lands per hour.

289
00:24:12,640 --> 00:24:17,720
 So with this, we are doing all this work in the open.

290
00:24:17,720 --> 00:24:25,760
 We have SummitQ and StovePipe being developed and everything is in the open source.

291
00:24:26,140 --> 00:24:35,900
 So if you want to explore and contribute, do scan the QR code, explore the project, and feel free to contribute to the project.

292
00:24:38,000 --> 00:24:39,360
 What are we doing next?

293
00:24:39,360 --> 00:25:00,440
 And as we have already done this work, we are still looking ways to make it better, optimize for throughput, handle the recovery process in a better way, and also improve on how we tag what is green or not.

294
00:25:00,440 --> 00:25:07,160
 So as a first thing we are thinking about is on SummitQ side basically batching related changes together.

295
00:25:07,400 --> 00:25:14,880
 So that way we run less number of builds and we can do more work with the same amount of resources.

296
00:25:15,500 --> 00:25:25,220
 Second is today we have a single pointer, verified main, which points whether a repository head is green or not, or a given commit is green or not.

297
00:25:25,220 --> 00:25:30,480
 We want to do better where basically we can say a given service is green, a given project is green.

298
00:25:31,040 --> 00:25:40,240
 So even if a commit on the trunk is broken, we can essentially tell which part of the trunk is broken and not all of it.

299
00:25:40,400 --> 00:25:49,560
 So that would allow us to basically do a better sort of rollouts of those services and changes, and we don't have to block it.

300
00:25:49,560 --> 00:25:55,920
 And the third one is basically, as I said, the recovery process from a broken trunk is manual.

301
00:25:56,160 --> 00:26:12,780
 So most of the investment is on automating the fixes or reverting the fixes and then landing those to Summit Q in an automated way so that our time to recovery can be reduced and we can fix the broken fronts faster.

302
00:26:13,920 --> 00:26:21,800
 And so as a closing point, I want to call out that we chose a hybrid approach to validating

303
00:26:21,800 --> 00:26:31,700
 the changes. One, we are still doing the pre-merge validations. We are still making sure that we

304
00:26:32,220 --> 00:26:40,380
 test the changes where it matters and relax the changes where we can take that hit and validate

305
00:26:40,380 --> 00:26:46,160
 validate them later after they land to the trunk. And then we are running the whole exhaustive

306
00:26:46,900 --> 00:26:53,540
 validation with the help of a store pipe post landing to the trunk. And that allows us to

307
00:26:53,540 --> 00:26:59,160
 basically have this balance where we are validating whatever matters before and then validating

308
00:26:59,160 --> 00:27:06,100
 everything at the end to make sure it is still right. And our consumers can consume the good

309
00:27:06,100 --> 00:27:14,360
 state. And with that, that would be our end of the talk. If you want to learn more about Summit

310
00:27:14,360 --> 00:27:20,400
 Queue and inner working software, feel free to scan these QR codes. These are the previously

311
00:27:20,980 --> 00:27:27,640
 talks that we have done on Summit Queue in depth, and they go over in detail on how it

312
00:27:27,640 --> 00:27:31,160
 functions internally. And again, thank you. Thank you.