Duolingo's Test-Writing Agent Is a Pull-Request Lifecycle, Not a Prompt

4:42:54Hey everyone, thanks for tuning in.

4:42:56Today I'm going to walk you through how

4:42:57we use AI agents at Dualingo to write

4:42:59and ship more than 200,000,000 lines of

4:43:01unit tests on iOS codebase. I'll share

4:43:03what we built and what we learned along

4:43:04the way. Uh I'm Kush. I'm a staff

4:43:06engineer on the platform team here at

4:43:07Dualingo. My job is to build developer

4:43:09tooling and AI assisted workflows for

4:43:10the 100 plus engineers who contribute to

4:43:11our client apps. Before Duelingo, I

4:43:13spent close to a decade at Stripe and

4:43:14Uber working across a few teams there.

4:43:17Let's start with some background on the

4:43:18scale of mobile at Dualingo. Uh we ship

4:43:21our apps to the app store every week. Um

4:43:23we're also a very experiment-driven

4:43:25company. So at any given point of time,

4:43:26we have more than 600 AB tests running

4:43:28live. And our code bases are pretty

4:43:30mature. Both the iOS and Android apps

4:43:32have been around for more than 10 years

4:43:33at this point.

4:43:35Here's also the number of executable

4:43:36lines of app code in our iOS repository.

4:43:38So, as you can see, back in 2019, we

4:43:40were at about 200,000 lines, and today

4:43:42we're hit close to 2 million. And as AI

4:43:44assisted coding picks up, you can see

4:43:45that curve starting to bend upward. And

4:43:48it's not just the amount of code that's

4:43:49changing, it's also who's writing it.

4:43:51So, last year about 200 people

4:43:53contributed to the iOS repository. And

4:43:54this year, we've already crossed 250.

4:43:56And a lot of these folks aren't even iOS

4:43:58engineers. We have backend engineers,

4:43:59Android engineers, even PMs and

4:44:01designers all shipping code to the iOS

4:44:02app and that's mostly been unlocked by

4:44:04AI coding tools like cursor and cloud

4:44:06code. Um, another thing to note about

4:44:08our codebase is that it's a codebased in

4:44:09transition. So a few years ago we

4:44:11standardized on the MVVM architecture

4:44:12pattern. Today about half the code in

4:44:14our repository follows that pattern and

4:44:16the other half is still legacy code that

4:44:17we're in the process of migrating. Since

4:44:19the MVVM half is the easiest to test,

4:44:21that's where we focused on this project.

4:44:24Here's where we started back in March of

4:44:25this year. We were at overall unit test

4:44:28coverage of 11%. The MVVM layers were a

4:44:30bit lower. Repositories were at 11%,

4:44:32view models at 9, data sources at five,

4:44:34and combined we are about 9% for the

4:44:36MBVM slice of the app.

4:44:39So why does this matter now more than it

4:44:41used to? As agents and humans write more

4:44:43code and write it faster, writing code

4:44:45isn't the hard part anymore. Verifying

4:44:46it is. So when someone puts up a PR, how

4:44:49do you know that it's actually safe to

4:44:50ship? The bottleneck is really shifting

4:44:51from writing to verifying. And that's

4:44:54where unit tests come in. They make up

4:44:55the base of the testing pyramid. They're

4:44:57pretty cheap to write and fast to run.

4:44:58And they also act as guardrails for

4:44:59agents. So when an agent's change breaks

4:45:01a test in CI, it gets immediate feedback

4:45:03on what to fix. They also act as living

4:45:05documentation on what the code is

4:45:06supposed to do. And here's where we

4:45:08ended up 8 months later. Overall unit

4:45:09test coverage went from 11% to 24%,

4:45:12repositories went from 11 to 72, view

4:45:14models from 9 to 58, data sources from 5

4:45:16to 41, and the combined MVVM slice of

4:45:18the app went from 9% to 59%. So what do

4:45:21we actually end up building? On paper,

4:45:23it's a pretty simple loop. First you

4:45:25identify the files that need tests. Then

4:45:26you write those tests with an agent. You

4:45:28verify that the tests are good quality

4:45:29and they pass CI. Then you resolve the

4:45:31PR which means getting reviewed by an

4:45:32engineer merged. And finally iterate on

4:45:34the feedback so that the next batch of

4:45:36PRs is better than the last one. When

4:45:38production it's a little messier. This

4:45:40is what our system ended up looking

4:45:41like. I won't go through every box right

4:45:42now, but at a high level it's a few

4:45:44temporal workflows that coordinate

4:45:45through shared state stored in S3. And

4:45:47every box you see here maps back to one

4:45:49of the five steps I just talked about.

4:45:51Here are the building blocks that we

4:45:52ended up using. Temporal handles

4:45:54workflow orchestration and execution.

4:45:56Cloud code acts as the coding agent. S3

4:45:58holds the shared state that keeps our

4:45:59workflows in sync. And we use GitHub for

4:46:01actions, labels, and PRs. Data dogs acts

4:46:04as an observability layer where we keep

4:46:05an eye on the health of our pipelines.

4:46:08If you haven't used temporal before,

4:46:09it's basically a durable workflow

4:46:11execution engine. You write your

4:46:12workflows as normal Python code, and if

4:46:14a worker crashes, the workflow just

4:46:15picks up right where it left off. This

4:46:17fits agents really well. An agent

4:46:19session can run for seconds to hours and

4:46:20it can call out to tools like GitHub,

4:46:22CI, S3 and Claude. We also lean on a few

4:46:25built-in features of temporal. There are

4:46:27schedules which are basically crown-like

4:46:28schedules for our jobs. There are

4:46:30composible workflows so that we can fan

4:46:31out work in parallel. Then there's first

4:46:33class support for retries and timeouts

4:46:34and item potency so that we never kick

4:46:36off duplicate runs of the same file.

4:46:38To split up this work, we split this

4:46:40project into four milestones. The first

4:46:42was local validation. Basically, is this

4:46:44project even viable? The second was a

4:46:45scheduled backfield workflow to write

4:46:47tests for existing code. The third was a

4:46:49PR life cycle workflow to actually get

4:46:51those PRs merged. And the fourth was a

4:46:52label trigger so engineers can get tests

4:46:54for the new code that they're writing.

4:46:56Now, we'll go through each of these in

4:46:57order. Before building anything real, we

4:46:59wanted to do the least amount of work

4:47:00possible that would tell us whether an

4:47:02agent could actually do this job well

4:47:03and do it at scale. We started with a

4:47:05very simple local clot code setup where

4:47:07we had one cloud code command that acted

4:47:08as an orchestrator. It ran an Xcode

4:47:10build script to generate a code coverage

4:47:12report locally. Then it fed that report

4:47:14through a file selection script that we

4:47:15wrote and then it handed each selected

4:47:17file off of sub two sub aents. One wrote

4:47:20the tests and the other one opened the

4:47:21PR. We ran this in a loop and generated

4:47:23a bunch of PRs. We basically ran this in

4:47:26two batches. The first batch had 17 PRs

4:47:27and we merged eight of them. The second

4:47:29had 40 PRs and we merged 19. So that's

4:47:31roughly a 50% merge rate. But the goal

4:47:32here wasn't to merge a lot of PRs. It

4:47:34was to learn how the agents fail and to

4:47:35fix those paper cuts before

4:47:36productionizing anything. Here are some

4:47:38of the failure modes that we saw. Since

4:47:40we're a large legacy code base with both

4:47:41XC test and swift testing, the agents

4:47:43would sometimes get confused and mix the

4:47:44two in one file. We fixed that by giving

4:47:46the agents clear rules on when to use

4:47:47each. We also had a giant user client

4:47:49singleton class. Some code passed around

4:47:51the concrete version of this class which

4:47:52is really hard to mock whereas others

4:47:53passed around a protocolized version. So

4:47:55we migrated the entire code base over to

4:47:56the protocolized version and added lint

4:47:58rule to enforce it. This made the whole

4:48:00host of files easier to test for agents

4:48:02and for humans. We also ran to six

4:48:05concurrency related issues and agents

4:48:06writing timing based tests which are a

4:48:08big source of flakiness in CI. So we

4:48:10improved the state for both of these

4:48:11with better rules and examples in our

4:48:12testing skills.

4:48:14Now this brings us to milestone 2. Since

4:48:16we're so far behind on code coverage,

4:48:17the biggest leverage that we saw was

4:48:19backfilling tests for code that was

4:48:20already merged in. That's where the

4:48:22scheduled backfill workflow comes in.

4:48:23This is a system design diagram of our

4:48:25workflow as it runs in production. And

4:48:26it essentially does two jobs. The left

4:48:28half is to identify which is picking the

4:48:30files that need tests and the right half

4:48:32is right which is having the agents

4:48:34actually write those tests. Let's start

4:48:36with identify. Every time a PR merges to

4:48:38the main branch in the iOS repo, we kick

4:48:40off a GitHub access job that generates

4:48:41an Xcode coverage report and uploads it

4:48:43to S3 that becomes a source of truth for

4:48:45the latest code coverage. The backfield

4:48:47workflow then fetches that report and

4:48:48runs it through a selector script which

4:48:49picks the next set of files to test.

4:48:52Let's get a deeper a little deeper into

4:48:53the selector script. Its job is

4:48:55essentially to score every Swift file

4:48:56based on how confident we are that an

4:48:58agent can write a good test for it. The

4:49:00strongest signal is file type. So view

4:49:02models, repositories, and data sources

4:49:03get a 2.5x boost while legacy files get

4:49:05pushed down. We skipped UI code

4:49:07completely for now since snapshot tests

4:49:08are better fit there and larger files

4:49:10also get a penalty. So we prioritize

4:49:12code that we're confident that we can

4:49:13test well.

4:49:15The next step is to actually write the

4:49:17tests. We kick off one child workflow

4:49:19per file all running in parallel. In

4:49:21each one of these runs, the agent clones

4:49:22the iOS repository, reads the testing

4:49:24skill that we keep in the repo, which

4:49:26spells out how to write a good test.

4:49:28Then it writes the test plus any mocks

4:49:29and fakes it needs and opens a PR with a

4:49:31generated test label so we can identify

4:49:33it easily later.

4:49:35These agents run on Dualingo's cloud

4:49:37coding agent which is a platform that

4:49:38also runs on Nepor. The simplest way to

4:49:40think about it is as an abstraction on

4:49:41top of cloud code and codex running in

4:49:42the cloud. We can invoke it from slack,

4:49:44temporal or github. It talks to a bunch

4:49:46of our internal tools for MCP and it

4:49:47logs a bunch of events so we can see

4:49:49what each agent run is doing and how

4:49:50much it costs. We started running these

4:49:53on Linux machines and then later move

4:49:55them to MacRunners which will become

4:49:56important in a minute. Here's what a

4:49:58generated test PR looks like. Every PR

4:50:00description follows the same template

4:50:02and it says explicitly that the PR is AI

4:50:04generated. The reviewer has four

4:50:05options. They can approve it in the PR

4:50:07auto modes. They can push changes to it

4:50:09directly. They can reassign it to

4:50:10someone better suited or they can close

4:50:11it out with feedback on why it isn't

4:50:13mergeable.

4:50:14Now once we had the backfill running in

4:50:16production, we had a slightly new

4:50:17problem. We suddenly had tens of tens to

4:50:19hundreds of open PRs that nobody could

4:50:20manage by hand. So how do we get all of

4:50:22these to a mergeable state and in front

4:50:24of the right reviewer?

4:50:26That's where the PR life cycle workflow

4:50:27comes in. You can think of it as a state

4:50:29machine. It runs every couple of hours,

4:50:30looks through every open generated test

4:50:32PR and takes exactly one action on each.

4:50:35It covers the next two steps of our

4:50:36loop. Verify, which is does this PR pass

4:50:39CI and is it high quality and resolve,

4:50:40which is getting a human to review it

4:50:42and merge it. The logic for this

4:50:44workflow is a simple decision tree. If

4:50:46there are any merge conflicts or the PR

4:50:47has been open for about 3 weeks, we

4:50:48close it out and mark it as still. If CI

4:50:50is still running or a reviewer is

4:50:51already on it, we skip it. If CI is

4:50:53green and there's no reviewers, we find

4:50:54one and sign them. If CI is red, we try

4:50:56to fix it. And if the reviewer rejects

4:50:58the PR, the file goes into 30-day cool

4:51:00down. So we're not spamming engineers

4:51:01with the same PR again and again.

4:51:03When CI fails, the first thing we check

4:51:05is which job failed. If it's pre-comit,

4:51:07a GitHub action applies the fix. It's

4:51:09cheap, it's deterministic, and there's

4:51:10no agent needed. If it's a real test

4:51:12failure, we kick off another agent that

4:51:13reads the CI logs and pushes a fix to

4:51:14the PR. We cap that whole loop at five

4:51:17attemps. Pass that we close the PR

4:51:18rather than trying to fix it forever.

4:51:20Once CI is green, we need to find the

4:51:22right reviewer. What we do is we run get

4:51:24blame on the source file being tested.

4:51:25find the top contributor and assign it

4:51:27to them and turn on auto merge. If they

4:51:28approve it, the PR gets merged. If they

4:51:30reject it, the file goes into the 30-day

4:51:31cool down which I previously mentioned.

4:51:33This brings us to milestone four. Once

4:51:35all of this was back filling tests for

4:51:37existing code, we wanted to make it just

4:51:38as easy to get tests for the new code

4:51:40engineers were writing. Now, any

4:51:42engineer can add or write test label to

4:51:43their own PR. It hooks into the same

4:51:45temporal infrastructure we just walked

4:51:46through. The only real difference is

4:51:48selection. Instead of the selection

4:51:49script, we take the file straight from

4:51:50the engineer's PR. The agent writes the

4:51:52test and you get a draft SPR link back

4:51:54to your original one. Now let's talk

4:51:55about some of the challenges that we ran

4:51:56into. Our agents originally ran on Linux

4:51:59machines which don't have X code of Lint

4:52:01installed. So when an agent would open a

4:52:02PR after about 20 or 30 minutes CI would

4:52:04fail frequently on Lint. This waste the

4:52:06agents time and CI time as well which is

4:52:08an increasingly scarce resource. For a

4:52:10while 30.6% of our PRs were failing due

4:52:12to lint and then when we dug into it 62%

4:52:14of those failures came from just two

4:52:15lint rules. We fixed this in layers. The

4:52:18first was better examples in the prompt

4:52:20along with pitfalls to avoid. The second

4:52:22was a GitHub action that autofixes lint

4:52:23errors in CI. The third was an audit of

4:52:25the rules that couldn't be autofixed. We

4:52:27essentially removed the ones that

4:52:28weren't high signal. And this made life

4:52:29easier for both agents and human

4:52:31engineers. And finally, the real fix

4:52:32came when we moved the agents to Mac

4:52:34runners. So now the agents could build

4:52:36and lend the code locally before opening

4:52:37up PR. The second challenge was PR

4:52:40quality. When we assign a PR to a human,

4:52:42we want it to be something they'd

4:52:43actually feel comfortable proving. But

4:52:45agents sometimes take the shortest path

4:52:46to green CI. We saw agents editing

4:52:48production code to make a failing test

4:52:50pass. We also saw them modifying,

4:52:51deleting or silencing existing tests.

4:52:52And then we saw them adding long weight

4:52:54to tests which can make CI flicky.

4:52:57Again, there was no single fix here. So

4:52:58we built three layers. The first is the

4:53:00deterministic rule. If a PR touches any

4:53:02file that isn't a test, a mock, a fake

4:53:04or a stop, we close it out. The second

4:53:06builds in our AI code review. At

4:53:08Dualingo, we use Claude and Codex to

4:53:09review every PR and those bots often

4:53:11leave comments. So we built an agent

4:53:13that resolves any P 0 comments before a

4:53:15human ever looks at the PR. And the

4:53:17third is an LLM as a judge. It's a

4:53:18reviewer agent built just for these test

4:53:20PRs. It reads the PR diff and rejects

4:53:22production edits, changed or deleted

4:53:24tests, disabled tests, and long weights.

4:53:26And as you learn more, we keep adding

4:53:27additional checks to it.

4:53:30So far, the LLM as a judge has reviewed

4:53:31about,200 PRs and rejected 73 of them.

4:53:34Every rejection comes with a comment

4:53:35like this one saying exactly why the PR

4:53:37was rejected. We then read that feedback

4:53:39and fold it back into our prompts and

4:53:40skills so the same mistakes get less

4:53:42likely each time. So, let's talk about

4:53:44what all of this added up to. Since we

4:53:46started, the pipeline has merged more

4:53:47than 700 test PRs, adding about 225,000

4:53:50lines of test code. Every one of those

4:53:52PRs was reviewed and approved by a

4:53:53human. MVVM test coverage went from 9%

4:53:56to 59% and overall coverage went from

4:53:5811% to 24%. Most importantly, we now

4:54:00have a code base so engineers and agents

4:54:02can ship faster and with more

4:54:03confidence. Here's a few more

4:54:05interesting stats. Since March, the

4:54:06pipeline has written 34% of all new test

4:54:08files in the repo, 44% of all new test

4:54:10cases, and 10% of every strip line added

4:54:12to the codebase. And interestingly, that

4:54:13share has actually dripped in the last

4:54:15few weeks because engineers are now

4:54:16writing a lot more tests themselves,

4:54:17which is honestly the end goal.

4:54:20Um, and here's the coverage curve since

4:54:22April when the backfield started running

4:54:23on a schedule. You can see it bend

4:54:25slightly upward around June and July,

4:54:26which is when we really wrapped up the

4:54:28pipeline.

4:54:29So, what have we learned while doing

4:54:30this project? The first thing is that

4:54:32this pattern generalizes the loop we

4:54:34talked about identify, write, verify,

4:54:36resolve, iterate, it isn't specific to

4:54:37testing. Unit tests were just the lowest

4:54:39hanging fruit and any job that fits the

4:54:40shape can run on the same

4:54:41infrastructure. And that's exactly what

4:54:43we're starting to do as a company. We're

4:54:44applying the same loop to modernization

4:54:46to modernizing screens to MVBM and Swift

4:54:48UI, migrating code uh on Android to

4:54:50Corines and generating snapshot tests.

4:54:53So before I wrap up, I want to leave you

4:54:55with a thought experiment. There's a new

4:54:57class of fast general purpose

4:54:58classifiers like Jev. They take in some

4:55:00state and output an action in

4:55:01milliseconds. So what if we replace a

4:55:02handwritten decision tree in the VR life

4:55:04cycle workflow with one of these? PR

4:55:06state goes in, the classifier picks the

4:55:08next action and we execute it all in

4:55:09milliseconds.

4:55:12I also want to thank a few folks who

4:55:13worked on this with me. Jesse, Gordalup,

4:55:15Hunter, and Jiha. They built a lot of

4:55:16the platform pieces this project stands

4:55:18on and helped review a lot of this code.

4:55:20That's it for me. Thank you so much for

4:55:21making it this far. If you want to go

4:55:23deeper, we've written about this

4:55:23pipeline on the dual engineering blog.

4:55:25And I'd also love to hear from you if

4:55:26you're building something similar. You

4:55:27can find me on LinkedIn or next.