1 04:42:54,320 --> 04:42:56,070 Hey everyone, thanks for tuning in. 2 04:42:56,080 --> 04:42:57,190 Today I'm going to walk you through how 3 04:42:57,200 --> 04:42:59,110 we use AI agents at Dualingo to write 4 04:42:59,120 --> 04:43:01,030 and ship more than 200,000,000 lines of 5 04:43:01,040 --> 04:43:03,110 unit tests on iOS codebase. I'll share 6 04:43:03,120 --> 04:43:04,550 what we built and what we learned along 7 04:43:04,560 --> 04:43:06,550 the way. Uh I'm Kush. I'm a staff 8 04:43:06,560 --> 04:43:07,590 engineer on the platform team here at 9 04:43:07,600 --> 04:43:09,030 Dualingo. My job is to build developer 10 04:43:09,040 --> 04:43:10,390 tooling and AI assisted workflows for 11 04:43:10,400 --> 04:43:11,750 the 100 plus engineers who contribute to 12 04:43:11,760 --> 04:43:13,510 our client apps. Before Duelingo, I 13 04:43:13,520 --> 04:43:14,708 spent close to a decade at Stripe and 14 04:43:14,718 --> 04:43:17,190 Uber working across a few teams there. 15 04:43:17,200 --> 04:43:18,788 Let's start with some background on the 16 04:43:18,798 --> 04:43:21,350 scale of mobile at Dualingo. Uh we ship 17 04:43:21,360 --> 04:43:23,510 our apps to the app store every week. Um 18 04:43:23,520 --> 04:43:25,190 we're also a very experiment-driven 19 04:43:25,200 --> 04:43:26,550 company. So at any given point of time, 20 04:43:26,560 --> 04:43:28,310 we have more than 600 AB tests running 21 04:43:28,320 --> 04:43:30,310 live. And our code bases are pretty 22 04:43:30,320 --> 04:43:31,990 mature. Both the iOS and Android apps 23 04:43:32,000 --> 04:43:33,110 have been around for more than 10 years 24 04:43:33,120 --> 04:43:35,190 at this point. 25 04:43:35,200 --> 04:43:36,788 Here's also the number of executable 26 04:43:36,798 --> 04:43:38,708 lines of app code in our iOS repository. 27 04:43:38,718 --> 04:43:40,390 So, as you can see, back in 2019, we 28 04:43:40,400 --> 04:43:42,150 were at about 200,000 lines, and today 29 04:43:42,160 --> 04:43:44,550 we're hit close to 2 million. And as AI 30 04:43:44,560 --> 04:43:45,910 assisted coding picks up, you can see 31 04:43:45,920 --> 04:43:48,470 that curve starting to bend upward. And 32 04:43:48,480 --> 04:43:49,670 it's not just the amount of code that's 33 04:43:49,680 --> 04:43:51,510 changing, it's also who's writing it. 34 04:43:51,520 --> 04:43:53,110 So, last year about 200 people 35 04:43:53,120 --> 04:43:54,550 contributed to the iOS repository. And 36 04:43:54,560 --> 04:43:56,390 this year, we've already crossed 250. 37 04:43:56,400 --> 04:43:57,990 And a lot of these folks aren't even iOS 38 04:43:58,000 --> 04:43:59,430 engineers. We have backend engineers, 39 04:43:59,440 --> 04:44:01,030 Android engineers, even PMs and 40 04:44:01,040 --> 04:44:02,470 designers all shipping code to the iOS 41 04:44:02,480 --> 04:44:04,628 app and that's mostly been unlocked by 42 04:44:04,638 --> 04:44:06,070 AI coding tools like cursor and cloud 43 04:44:06,080 --> 04:44:08,310 code. Um, another thing to note about 44 04:44:08,320 --> 04:44:09,750 our codebase is that it's a codebased in 45 04:44:09,760 --> 04:44:11,430 transition. So a few years ago we 46 04:44:11,440 --> 04:44:12,788 standardized on the MVVM architecture 47 04:44:12,798 --> 04:44:14,708 pattern. Today about half the code in 48 04:44:14,718 --> 04:44:16,070 our repository follows that pattern and 49 04:44:16,080 --> 04:44:17,510 the other half is still legacy code that 50 04:44:17,520 --> 04:44:19,350 we're in the process of migrating. Since 51 04:44:19,360 --> 04:44:21,030 the MVVM half is the easiest to test, 52 04:44:21,040 --> 04:44:24,070 that's where we focused on this project. 53 04:44:24,080 --> 04:44:25,590 Here's where we started back in March of 54 04:44:25,600 --> 04:44:28,070 this year. We were at overall unit test 55 04:44:28,080 --> 04:44:30,788 coverage of 11%. The MVVM layers were a 56 04:44:30,798 --> 04:44:32,628 bit lower. Repositories were at 11%, 57 04:44:32,638 --> 04:44:34,628 view models at 9, data sources at five, 58 04:44:34,638 --> 04:44:36,550 and combined we are about 9% for the 59 04:44:36,560 --> 04:44:39,190 MBVM slice of the app. 60 04:44:39,200 --> 04:44:41,190 So why does this matter now more than it 61 04:44:41,200 --> 04:44:43,750 used to? As agents and humans write more 62 04:44:43,760 --> 04:44:45,430 code and write it faster, writing code 63 04:44:45,440 --> 04:44:46,868 isn't the hard part anymore. Verifying 64 04:44:46,878 --> 04:44:49,030 it is. So when someone puts up a PR, how 65 04:44:49,040 --> 04:44:50,150 do you know that it's actually safe to 66 04:44:50,160 --> 04:44:51,830 ship? The bottleneck is really shifting 67 04:44:51,840 --> 04:44:53,990 from writing to verifying. And that's 68 04:44:54,000 --> 04:44:55,750 where unit tests come in. They make up 69 04:44:55,760 --> 04:44:57,190 the base of the testing pyramid. They're 70 04:44:57,200 --> 04:44:58,550 pretty cheap to write and fast to run. 71 04:44:58,560 --> 04:44:59,830 And they also act as guardrails for 72 04:44:59,840 --> 04:45:01,670 agents. So when an agent's change breaks 73 04:45:01,680 --> 04:45:03,510 a test in CI, it gets immediate feedback 74 04:45:03,520 --> 04:45:05,590 on what to fix. They also act as living 75 04:45:05,600 --> 04:45:06,788 documentation on what the code is 76 04:45:06,798 --> 04:45:08,390 supposed to do. And here's where we 77 04:45:08,400 --> 04:45:09,910 ended up 8 months later. Overall unit 78 04:45:09,920 --> 04:45:12,230 test coverage went from 11% to 24%, 79 04:45:12,240 --> 04:45:14,390 repositories went from 11 to 72, view 80 04:45:14,400 --> 04:45:16,708 models from 9 to 58, data sources from 5 81 04:45:16,718 --> 04:45:18,788 to 41, and the combined MVVM slice of 82 04:45:18,798 --> 04:45:21,750 the app went from 9% to 59%. So what do 83 04:45:21,760 --> 04:45:23,430 we actually end up building? On paper, 84 04:45:23,440 --> 04:45:25,030 it's a pretty simple loop. First you 85 04:45:25,040 --> 04:45:26,868 identify the files that need tests. Then 86 04:45:26,878 --> 04:45:28,550 you write those tests with an agent. You 87 04:45:28,560 --> 04:45:29,750 verify that the tests are good quality 88 04:45:29,760 --> 04:45:31,190 and they pass CI. Then you resolve the 89 04:45:31,200 --> 04:45:32,948 PR which means getting reviewed by an 90 04:45:32,958 --> 04:45:34,868 engineer merged. And finally iterate on 91 04:45:34,878 --> 04:45:35,990 the feedback so that the next batch of 92 04:45:36,000 --> 04:45:38,550 PRs is better than the last one. When 93 04:45:38,560 --> 04:45:40,150 production it's a little messier. This 94 04:45:40,160 --> 04:45:41,270 is what our system ended up looking 95 04:45:41,280 --> 04:45:42,788 like. I won't go through every box right 96 04:45:42,798 --> 04:45:44,070 now, but at a high level it's a few 97 04:45:44,080 --> 04:45:45,350 temporal workflows that coordinate 98 04:45:45,360 --> 04:45:47,590 through shared state stored in S3. And 99 04:45:47,600 --> 04:45:49,110 every box you see here maps back to one 100 04:45:49,120 --> 04:45:51,670 of the five steps I just talked about. 101 04:45:51,680 --> 04:45:52,788 Here are the building blocks that we 102 04:45:52,798 --> 04:45:54,470 ended up using. Temporal handles 103 04:45:54,480 --> 04:45:56,070 workflow orchestration and execution. 104 04:45:56,080 --> 04:45:58,470 Cloud code acts as the coding agent. S3 105 04:45:58,480 --> 04:45:59,910 holds the shared state that keeps our 106 04:45:59,920 --> 04:46:01,830 workflows in sync. And we use GitHub for 107 04:46:01,840 --> 04:46:04,390 actions, labels, and PRs. Data dogs acts 108 04:46:04,400 --> 04:46:05,670 as an observability layer where we keep 109 04:46:05,680 --> 04:46:08,470 an eye on the health of our pipelines. 110 04:46:08,480 --> 04:46:09,750 If you haven't used temporal before, 111 04:46:09,760 --> 04:46:11,030 it's basically a durable workflow 112 04:46:11,040 --> 04:46:12,550 execution engine. You write your 113 04:46:12,560 --> 04:46:14,070 workflows as normal Python code, and if 114 04:46:14,080 --> 04:46:15,430 a worker crashes, the workflow just 115 04:46:15,440 --> 04:46:17,350 picks up right where it left off. This 116 04:46:17,360 --> 04:46:19,110 fits agents really well. An agent 117 04:46:19,120 --> 04:46:20,948 session can run for seconds to hours and 118 04:46:20,958 --> 04:46:22,628 it can call out to tools like GitHub, 119 04:46:22,638 --> 04:46:25,350 CI, S3 and Claude. We also lean on a few 120 04:46:25,360 --> 04:46:27,190 built-in features of temporal. There are 121 04:46:27,200 --> 04:46:28,628 schedules which are basically crown-like 122 04:46:28,638 --> 04:46:29,990 schedules for our jobs. There are 123 04:46:30,000 --> 04:46:31,350 composible workflows so that we can fan 124 04:46:31,360 --> 04:46:33,110 out work in parallel. Then there's first 125 04:46:33,120 --> 04:46:34,628 class support for retries and timeouts 126 04:46:34,638 --> 04:46:36,310 and item potency so that we never kick 127 04:46:36,320 --> 04:46:38,868 off duplicate runs of the same file. 128 04:46:38,878 --> 04:46:40,470 To split up this work, we split this 129 04:46:40,480 --> 04:46:42,150 project into four milestones. The first 130 04:46:42,160 --> 04:46:44,150 was local validation. Basically, is this 131 04:46:44,160 --> 04:46:45,830 project even viable? The second was a 132 04:46:45,840 --> 04:46:47,190 scheduled backfield workflow to write 133 04:46:47,200 --> 04:46:49,110 tests for existing code. The third was a 134 04:46:49,120 --> 04:46:51,030 PR life cycle workflow to actually get 135 04:46:51,040 --> 04:46:52,948 those PRs merged. And the fourth was a 136 04:46:52,958 --> 04:46:54,708 label trigger so engineers can get tests 137 04:46:54,718 --> 04:46:56,390 for the new code that they're writing. 138 04:46:56,400 --> 04:46:57,750 Now, we'll go through each of these in 139 04:46:57,760 --> 04:46:59,670 order. Before building anything real, we 140 04:46:59,680 --> 04:47:00,788 wanted to do the least amount of work 141 04:47:00,798 --> 04:47:02,070 possible that would tell us whether an 142 04:47:02,080 --> 04:47:03,270 agent could actually do this job well 143 04:47:03,280 --> 04:47:05,190 and do it at scale. We started with a 144 04:47:05,200 --> 04:47:07,270 very simple local clot code setup where 145 04:47:07,280 --> 04:47:08,788 we had one cloud code command that acted 146 04:47:08,798 --> 04:47:10,948 as an orchestrator. It ran an Xcode 147 04:47:10,958 --> 04:47:12,550 build script to generate a code coverage 148 04:47:12,560 --> 04:47:14,470 report locally. Then it fed that report 149 04:47:14,480 --> 04:47:15,750 through a file selection script that we 150 04:47:15,760 --> 04:47:17,590 wrote and then it handed each selected 151 04:47:17,600 --> 04:47:20,150 file off of sub two sub aents. One wrote 152 04:47:20,160 --> 04:47:21,590 the tests and the other one opened the 153 04:47:21,600 --> 04:47:23,510 PR. We ran this in a loop and generated 154 04:47:23,520 --> 04:47:25,990 a bunch of PRs. We basically ran this in 155 04:47:26,000 --> 04:47:27,910 two batches. The first batch had 17 PRs 156 04:47:27,920 --> 04:47:29,270 and we merged eight of them. The second 157 04:47:29,280 --> 04:47:31,110 had 40 PRs and we merged 19. So that's 158 04:47:31,120 --> 04:47:32,948 roughly a 50% merge rate. But the goal 159 04:47:32,958 --> 04:47:34,230 here wasn't to merge a lot of PRs. It 160 04:47:34,240 --> 04:47:35,430 was to learn how the agents fail and to 161 04:47:35,440 --> 04:47:36,390 fix those paper cuts before 162 04:47:36,400 --> 04:47:38,788 productionizing anything. Here are some 163 04:47:38,798 --> 04:47:40,310 of the failure modes that we saw. Since 164 04:47:40,320 --> 04:47:41,670 we're a large legacy code base with both 165 04:47:41,680 --> 04:47:43,190 XC test and swift testing, the agents 166 04:47:43,200 --> 04:47:44,470 would sometimes get confused and mix the 167 04:47:44,480 --> 04:47:46,230 two in one file. We fixed that by giving 168 04:47:46,240 --> 04:47:47,670 the agents clear rules on when to use 169 04:47:47,680 --> 04:47:49,750 each. We also had a giant user client 170 04:47:49,760 --> 04:47:51,510 singleton class. Some code passed around 171 04:47:51,520 --> 04:47:52,708 the concrete version of this class which 172 04:47:52,718 --> 04:47:53,830 is really hard to mock whereas others 173 04:47:53,840 --> 04:47:55,350 passed around a protocolized version. So 174 04:47:55,360 --> 04:47:56,788 we migrated the entire code base over to 175 04:47:56,798 --> 04:47:58,628 the protocolized version and added lint 176 04:47:58,638 --> 04:48:00,390 rule to enforce it. This made the whole 177 04:48:00,400 --> 04:48:02,230 host of files easier to test for agents 178 04:48:02,240 --> 04:48:05,030 and for humans. We also ran to six 179 04:48:05,040 --> 04:48:06,868 concurrency related issues and agents 180 04:48:06,878 --> 04:48:08,230 writing timing based tests which are a 181 04:48:08,240 --> 04:48:09,990 big source of flakiness in CI. So we 182 04:48:10,000 --> 04:48:11,030 improved the state for both of these 183 04:48:11,040 --> 04:48:12,550 with better rules and examples in our 184 04:48:12,560 --> 04:48:14,390 testing skills. 185 04:48:14,400 --> 04:48:16,150 Now this brings us to milestone 2. Since 186 04:48:16,160 --> 04:48:17,910 we're so far behind on code coverage, 187 04:48:17,920 --> 04:48:19,190 the biggest leverage that we saw was 188 04:48:19,200 --> 04:48:20,470 backfilling tests for code that was 189 04:48:20,480 --> 04:48:22,150 already merged in. That's where the 190 04:48:22,160 --> 04:48:23,830 scheduled backfill workflow comes in. 191 04:48:23,840 --> 04:48:25,190 This is a system design diagram of our 192 04:48:25,200 --> 04:48:26,868 workflow as it runs in production. And 193 04:48:26,878 --> 04:48:28,708 it essentially does two jobs. The left 194 04:48:28,718 --> 04:48:30,470 half is to identify which is picking the 195 04:48:30,480 --> 04:48:32,150 files that need tests and the right half 196 04:48:32,160 --> 04:48:34,070 is right which is having the agents 197 04:48:34,080 --> 04:48:36,550 actually write those tests. Let's start 198 04:48:36,560 --> 04:48:38,550 with identify. Every time a PR merges to 199 04:48:38,560 --> 04:48:39,990 the main branch in the iOS repo, we kick 200 04:48:40,000 --> 04:48:41,670 off a GitHub access job that generates 201 04:48:41,680 --> 04:48:43,270 an Xcode coverage report and uploads it 202 04:48:43,280 --> 04:48:45,350 to S3 that becomes a source of truth for 203 04:48:45,360 --> 04:48:47,270 the latest code coverage. The backfield 204 04:48:47,280 --> 04:48:48,470 workflow then fetches that report and 205 04:48:48,480 --> 04:48:49,830 runs it through a selector script which 206 04:48:49,840 --> 04:48:52,310 picks the next set of files to test. 207 04:48:52,320 --> 04:48:53,750 Let's get a deeper a little deeper into 208 04:48:53,760 --> 04:48:55,510 the selector script. Its job is 209 04:48:55,520 --> 04:48:56,948 essentially to score every Swift file 210 04:48:56,958 --> 04:48:58,310 based on how confident we are that an 211 04:48:58,320 --> 04:49:00,070 agent can write a good test for it. The 212 04:49:00,080 --> 04:49:02,070 strongest signal is file type. So view 213 04:49:02,080 --> 04:49:03,510 models, repositories, and data sources 214 04:49:03,520 --> 04:49:05,910 get a 2.5x boost while legacy files get 215 04:49:05,920 --> 04:49:07,590 pushed down. We skipped UI code 216 04:49:07,600 --> 04:49:08,868 completely for now since snapshot tests 217 04:49:08,878 --> 04:49:10,868 are better fit there and larger files 218 04:49:10,878 --> 04:49:12,310 also get a penalty. So we prioritize 219 04:49:12,320 --> 04:49:13,750 code that we're confident that we can 220 04:49:13,760 --> 04:49:15,670 test well. 221 04:49:15,680 --> 04:49:17,030 The next step is to actually write the 222 04:49:17,040 --> 04:49:19,030 tests. We kick off one child workflow 223 04:49:19,040 --> 04:49:21,430 per file all running in parallel. In 224 04:49:21,440 --> 04:49:22,948 each one of these runs, the agent clones 225 04:49:22,958 --> 04:49:24,788 the iOS repository, reads the testing 226 04:49:24,798 --> 04:49:26,230 skill that we keep in the repo, which 227 04:49:26,240 --> 04:49:27,990 spells out how to write a good test. 228 04:49:28,000 --> 04:49:29,750 Then it writes the test plus any mocks 229 04:49:29,760 --> 04:49:31,430 and fakes it needs and opens a PR with a 230 04:49:31,440 --> 04:49:33,110 generated test label so we can identify 231 04:49:33,120 --> 04:49:35,590 it easily later. 232 04:49:35,600 --> 04:49:37,030 These agents run on Dualingo's cloud 233 04:49:37,040 --> 04:49:38,310 coding agent which is a platform that 234 04:49:38,320 --> 04:49:40,150 also runs on Nepor. The simplest way to 235 04:49:40,160 --> 04:49:41,430 think about it is as an abstraction on 236 04:49:41,440 --> 04:49:42,708 top of cloud code and codex running in 237 04:49:42,718 --> 04:49:44,550 the cloud. We can invoke it from slack, 238 04:49:44,560 --> 04:49:46,150 temporal or github. It talks to a bunch 239 04:49:46,160 --> 04:49:47,830 of our internal tools for MCP and it 240 04:49:47,840 --> 04:49:49,430 logs a bunch of events so we can see 241 04:49:49,440 --> 04:49:50,708 what each agent run is doing and how 242 04:49:50,718 --> 04:49:53,430 much it costs. We started running these 243 04:49:53,440 --> 04:49:55,030 on Linux machines and then later move 244 04:49:55,040 --> 04:49:56,390 them to MacRunners which will become 245 04:49:56,400 --> 04:49:58,868 important in a minute. Here's what a 246 04:49:58,878 --> 04:50:00,868 generated test PR looks like. Every PR 247 04:50:00,878 --> 04:50:02,150 description follows the same template 248 04:50:02,160 --> 04:50:03,990 and it says explicitly that the PR is AI 249 04:50:04,000 --> 04:50:05,750 generated. The reviewer has four 250 04:50:05,760 --> 04:50:07,190 options. They can approve it in the PR 251 04:50:07,200 --> 04:50:09,110 auto modes. They can push changes to it 252 04:50:09,120 --> 04:50:10,390 directly. They can reassign it to 253 04:50:10,400 --> 04:50:11,830 someone better suited or they can close 254 04:50:11,840 --> 04:50:13,110 it out with feedback on why it isn't 255 04:50:13,120 --> 04:50:14,708 mergeable. 256 04:50:14,718 --> 04:50:16,230 Now once we had the backfill running in 257 04:50:16,240 --> 04:50:17,350 production, we had a slightly new 258 04:50:17,360 --> 04:50:19,590 problem. We suddenly had tens of tens to 259 04:50:19,600 --> 04:50:20,948 hundreds of open PRs that nobody could 260 04:50:20,958 --> 04:50:22,708 manage by hand. So how do we get all of 261 04:50:22,718 --> 04:50:23,990 these to a mergeable state and in front 262 04:50:24,000 --> 04:50:25,990 of the right reviewer? 263 04:50:26,000 --> 04:50:27,510 That's where the PR life cycle workflow 264 04:50:27,520 --> 04:50:29,110 comes in. You can think of it as a state 265 04:50:29,120 --> 04:50:30,788 machine. It runs every couple of hours, 266 04:50:30,798 --> 04:50:32,470 looks through every open generated test 267 04:50:32,480 --> 04:50:35,110 PR and takes exactly one action on each. 268 04:50:35,120 --> 04:50:36,788 It covers the next two steps of our 269 04:50:36,798 --> 04:50:39,110 loop. Verify, which is does this PR pass 270 04:50:39,120 --> 04:50:40,868 CI and is it high quality and resolve, 271 04:50:40,878 --> 04:50:42,150 which is getting a human to review it 272 04:50:42,160 --> 04:50:44,390 and merge it. The logic for this 273 04:50:44,400 --> 04:50:45,990 workflow is a simple decision tree. If 274 04:50:46,000 --> 04:50:47,270 there are any merge conflicts or the PR 275 04:50:47,280 --> 04:50:48,390 has been open for about 3 weeks, we 276 04:50:48,400 --> 04:50:50,230 close it out and mark it as still. If CI 277 04:50:50,240 --> 04:50:51,430 is still running or a reviewer is 278 04:50:51,440 --> 04:50:53,110 already on it, we skip it. If CI is 279 04:50:53,120 --> 04:50:54,310 green and there's no reviewers, we find 280 04:50:54,320 --> 04:50:56,150 one and sign them. If CI is red, we try 281 04:50:56,160 --> 04:50:58,390 to fix it. And if the reviewer rejects 282 04:50:58,400 --> 04:51:00,070 the PR, the file goes into 30-day cool 283 04:51:00,080 --> 04:51:01,190 down. So we're not spamming engineers 284 04:51:01,200 --> 04:51:03,670 with the same PR again and again. 285 04:51:03,680 --> 04:51:05,430 When CI fails, the first thing we check 286 04:51:05,440 --> 04:51:07,430 is which job failed. If it's pre-comit, 287 04:51:07,440 --> 04:51:09,110 a GitHub action applies the fix. It's 288 04:51:09,120 --> 04:51:10,310 cheap, it's deterministic, and there's 289 04:51:10,320 --> 04:51:11,990 no agent needed. If it's a real test 290 04:51:12,000 --> 04:51:13,430 failure, we kick off another agent that 291 04:51:13,440 --> 04:51:14,788 reads the CI logs and pushes a fix to 292 04:51:14,798 --> 04:51:17,030 the PR. We cap that whole loop at five 293 04:51:17,040 --> 04:51:18,390 attemps. Pass that we close the PR 294 04:51:18,400 --> 04:51:20,868 rather than trying to fix it forever. 295 04:51:20,878 --> 04:51:22,310 Once CI is green, we need to find the 296 04:51:22,320 --> 04:51:24,150 right reviewer. What we do is we run get 297 04:51:24,160 --> 04:51:25,350 blame on the source file being tested. 298 04:51:25,360 --> 04:51:27,030 find the top contributor and assign it 299 04:51:27,040 --> 04:51:28,788 to them and turn on auto merge. If they 300 04:51:28,798 --> 04:51:30,390 approve it, the PR gets merged. If they 301 04:51:30,400 --> 04:51:31,830 reject it, the file goes into the 30-day 302 04:51:31,840 --> 04:51:33,830 cool down which I previously mentioned. 303 04:51:33,840 --> 04:51:35,910 This brings us to milestone four. Once 304 04:51:35,920 --> 04:51:37,350 all of this was back filling tests for 305 04:51:37,360 --> 04:51:38,788 existing code, we wanted to make it just 306 04:51:38,798 --> 04:51:39,990 as easy to get tests for the new code 307 04:51:40,000 --> 04:51:42,070 engineers were writing. Now, any 308 04:51:42,080 --> 04:51:43,590 engineer can add or write test label to 309 04:51:43,600 --> 04:51:45,430 their own PR. It hooks into the same 310 04:51:45,440 --> 04:51:46,788 temporal infrastructure we just walked 311 04:51:46,798 --> 04:51:48,150 through. The only real difference is 312 04:51:48,160 --> 04:51:49,510 selection. Instead of the selection 313 04:51:49,520 --> 04:51:50,708 script, we take the file straight from 314 04:51:50,718 --> 04:51:52,708 the engineer's PR. The agent writes the 315 04:51:52,718 --> 04:51:54,310 test and you get a draft SPR link back 316 04:51:54,320 --> 04:51:55,830 to your original one. Now let's talk 317 04:51:55,840 --> 04:51:56,948 about some of the challenges that we ran 318 04:51:56,958 --> 04:51:59,350 into. Our agents originally ran on Linux 319 04:51:59,360 --> 04:52:01,030 machines which don't have X code of Lint 320 04:52:01,040 --> 04:52:02,550 installed. So when an agent would open a 321 04:52:02,560 --> 04:52:04,708 PR after about 20 or 30 minutes CI would 322 04:52:04,718 --> 04:52:06,788 fail frequently on Lint. This waste the 323 04:52:06,798 --> 04:52:08,628 agents time and CI time as well which is 324 04:52:08,638 --> 04:52:10,550 an increasingly scarce resource. For a 325 04:52:10,560 --> 04:52:12,868 while 30.6% of our PRs were failing due 326 04:52:12,878 --> 04:52:14,788 to lint and then when we dug into it 62% 327 04:52:14,798 --> 04:52:15,910 of those failures came from just two 328 04:52:15,920 --> 04:52:18,550 lint rules. We fixed this in layers. The 329 04:52:18,560 --> 04:52:20,150 first was better examples in the prompt 330 04:52:20,160 --> 04:52:21,990 along with pitfalls to avoid. The second 331 04:52:22,000 --> 04:52:23,670 was a GitHub action that autofixes lint 332 04:52:23,680 --> 04:52:25,590 errors in CI. The third was an audit of 333 04:52:25,600 --> 04:52:27,430 the rules that couldn't be autofixed. We 334 04:52:27,440 --> 04:52:28,390 essentially removed the ones that 335 04:52:28,400 --> 04:52:29,910 weren't high signal. And this made life 336 04:52:29,920 --> 04:52:31,110 easier for both agents and human 337 04:52:31,120 --> 04:52:32,868 engineers. And finally, the real fix 338 04:52:32,878 --> 04:52:34,310 came when we moved the agents to Mac 339 04:52:34,320 --> 04:52:35,990 runners. So now the agents could build 340 04:52:36,000 --> 04:52:37,910 and lend the code locally before opening 341 04:52:37,920 --> 04:52:40,628 up PR. The second challenge was PR 342 04:52:40,638 --> 04:52:42,550 quality. When we assign a PR to a human, 343 04:52:42,560 --> 04:52:43,510 we want it to be something they'd 344 04:52:43,520 --> 04:52:45,350 actually feel comfortable proving. But 345 04:52:45,360 --> 04:52:46,708 agents sometimes take the shortest path 346 04:52:46,718 --> 04:52:48,788 to green CI. We saw agents editing 347 04:52:48,798 --> 04:52:49,990 production code to make a failing test 348 04:52:50,000 --> 04:52:51,430 pass. We also saw them modifying, 349 04:52:51,440 --> 04:52:52,948 deleting or silencing existing tests. 350 04:52:52,958 --> 04:52:54,230 And then we saw them adding long weight 351 04:52:54,240 --> 04:52:57,030 to tests which can make CI flicky. 352 04:52:57,040 --> 04:52:58,550 Again, there was no single fix here. So 353 04:52:58,560 --> 04:53:00,470 we built three layers. The first is the 354 04:53:00,480 --> 04:53:02,628 deterministic rule. If a PR touches any 355 04:53:02,638 --> 04:53:04,390 file that isn't a test, a mock, a fake 356 04:53:04,400 --> 04:53:06,390 or a stop, we close it out. The second 357 04:53:06,400 --> 04:53:07,990 builds in our AI code review. At 358 04:53:08,000 --> 04:53:09,510 Dualingo, we use Claude and Codex to 359 04:53:09,520 --> 04:53:11,590 review every PR and those bots often 360 04:53:11,600 --> 04:53:13,430 leave comments. So we built an agent 361 04:53:13,440 --> 04:53:15,270 that resolves any P 0 comments before a 362 04:53:15,280 --> 04:53:17,430 human ever looks at the PR. And the 363 04:53:17,440 --> 04:53:18,868 third is an LLM as a judge. It's a 364 04:53:18,878 --> 04:53:20,390 reviewer agent built just for these test 365 04:53:20,400 --> 04:53:22,550 PRs. It reads the PR diff and rejects 366 04:53:22,560 --> 04:53:23,990 production edits, changed or deleted 367 04:53:24,000 --> 04:53:26,070 tests, disabled tests, and long weights. 368 04:53:26,080 --> 04:53:27,430 And as you learn more, we keep adding 369 04:53:27,440 --> 04:53:30,070 additional checks to it. 370 04:53:30,080 --> 04:53:31,510 So far, the LLM as a judge has reviewed 371 04:53:31,520 --> 04:53:34,230 about,200 PRs and rejected 73 of them. 372 04:53:34,240 --> 04:53:35,590 Every rejection comes with a comment 373 04:53:35,600 --> 04:53:37,110 like this one saying exactly why the PR 374 04:53:37,120 --> 04:53:39,110 was rejected. We then read that feedback 375 04:53:39,120 --> 04:53:40,470 and fold it back into our prompts and 376 04:53:40,480 --> 04:53:41,990 skills so the same mistakes get less 377 04:53:42,000 --> 04:53:44,550 likely each time. So, let's talk about 378 04:53:44,560 --> 04:53:46,390 what all of this added up to. Since we 379 04:53:46,400 --> 04:53:47,750 started, the pipeline has merged more 380 04:53:47,760 --> 04:53:50,310 than 700 test PRs, adding about 225,000 381 04:53:50,320 --> 04:53:52,070 lines of test code. Every one of those 382 04:53:52,080 --> 04:53:53,510 PRs was reviewed and approved by a 383 04:53:53,520 --> 04:53:56,150 human. MVVM test coverage went from 9% 384 04:53:56,160 --> 04:53:58,070 to 59% and overall coverage went from 385 04:53:58,080 --> 04:54:00,788 11% to 24%. Most importantly, we now 386 04:54:00,798 --> 04:54:01,990 have a code base so engineers and agents 387 04:54:02,000 --> 04:54:03,030 can ship faster and with more 388 04:54:03,040 --> 04:54:05,270 confidence. Here's a few more 389 04:54:05,280 --> 04:54:06,788 interesting stats. Since March, the 390 04:54:06,798 --> 04:54:08,628 pipeline has written 34% of all new test 391 04:54:08,638 --> 04:54:10,788 files in the repo, 44% of all new test 392 04:54:10,798 --> 04:54:12,230 cases, and 10% of every strip line added 393 04:54:12,240 --> 04:54:13,910 to the codebase. And interestingly, that 394 04:54:13,920 --> 04:54:15,110 share has actually dripped in the last 395 04:54:15,120 --> 04:54:16,310 few weeks because engineers are now 396 04:54:16,320 --> 04:54:17,510 writing a lot more tests themselves, 397 04:54:17,520 --> 04:54:20,390 which is honestly the end goal. 398 04:54:20,400 --> 04:54:22,628 Um, and here's the coverage curve since 399 04:54:22,638 --> 04:54:23,910 April when the backfield started running 400 04:54:23,920 --> 04:54:25,590 on a schedule. You can see it bend 401 04:54:25,600 --> 04:54:26,868 slightly upward around June and July, 402 04:54:26,878 --> 04:54:28,070 which is when we really wrapped up the 403 04:54:28,080 --> 04:54:29,670 pipeline. 404 04:54:29,680 --> 04:54:30,868 So, what have we learned while doing 405 04:54:30,878 --> 04:54:32,550 this project? The first thing is that 406 04:54:32,560 --> 04:54:34,390 this pattern generalizes the loop we 407 04:54:34,400 --> 04:54:36,070 talked about identify, write, verify, 408 04:54:36,080 --> 04:54:37,590 resolve, iterate, it isn't specific to 409 04:54:37,600 --> 04:54:39,270 testing. Unit tests were just the lowest 410 04:54:39,280 --> 04:54:40,948 hanging fruit and any job that fits the 411 04:54:40,958 --> 04:54:41,750 shape can run on the same 412 04:54:41,760 --> 04:54:43,350 infrastructure. And that's exactly what 413 04:54:43,360 --> 04:54:44,868 we're starting to do as a company. We're 414 04:54:44,878 --> 04:54:46,708 applying the same loop to modernization 415 04:54:46,718 --> 04:54:48,390 to modernizing screens to MVBM and Swift 416 04:54:48,400 --> 04:54:50,628 UI, migrating code uh on Android to 417 04:54:50,638 --> 04:54:53,910 Corines and generating snapshot tests. 418 04:54:53,920 --> 04:54:55,670 So before I wrap up, I want to leave you 419 04:54:55,680 --> 04:54:57,270 with a thought experiment. There's a new 420 04:54:57,280 --> 04:54:58,390 class of fast general purpose 421 04:54:58,400 --> 04:55:00,150 classifiers like Jev. They take in some 422 04:55:00,160 --> 04:55:01,190 state and output an action in 423 04:55:01,200 --> 04:55:02,948 milliseconds. So what if we replace a 424 04:55:02,958 --> 04:55:04,470 handwritten decision tree in the VR life 425 04:55:04,480 --> 04:55:06,390 cycle workflow with one of these? PR 426 04:55:06,400 --> 04:55:08,150 state goes in, the classifier picks the 427 04:55:08,160 --> 04:55:09,590 next action and we execute it all in 428 04:55:09,600 --> 04:55:12,150 milliseconds. 429 04:55:12,160 --> 04:55:13,670 I also want to thank a few folks who 430 04:55:13,680 --> 04:55:15,430 worked on this with me. Jesse, Gordalup, 431 04:55:15,440 --> 04:55:16,708 Hunter, and Jiha. They built a lot of 432 04:55:16,718 --> 04:55:17,990 the platform pieces this project stands 433 04:55:18,000 --> 04:55:20,390 on and helped review a lot of this code. 434 04:55:20,400 --> 04:55:21,750 That's it for me. Thank you so much for 435 04:55:21,760 --> 04:55:23,030 making it this far. If you want to go 436 04:55:23,040 --> 04:55:23,910 deeper, we've written about this 437 04:55:23,920 --> 04:55:25,430 pipeline on the dual engineering blog. 438 04:55:25,440 --> 04:55:26,628 And I'd also love to hear from you if 439 04:55:26,638 --> 04:55:27,590 you're building something similar. You 440 04:55:27,600 --> 04:55:30,000 can find me on LinkedIn or next.