225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo

Duolingo shipped its mobile apps every week, with more than 600 A/B tests live at a time. Its iOS repository had grown from about 200,000 executable lines of app code in 2019 to nearly 2 million. About 200 people had contributed to the repository the previous year. More than 250 had already contributed in the year of the talk, as AI coding tools helped backend and Android engineers, product managers, and designers ship iOS changes. For Kush Agrawal, a staff engineer on Duolingo’s platform team, that put pressure on the reviewer of each pull request (PR). How could they tell whether a change was safe to ship?

In March, overall unit test coverage stood at 11%. About half the repository followed the Model–View–ViewModel (MVVM) architecture the team had standardized on. The other half was legacy code still being migrated. The MVVM layers were easier to test, yet their combined coverage was only about 9%. That made them a good place to start. A unit test could document expected behavior for the next contributor and give an agent immediate feedback when a later change broke that behavior in continuous integration (CI).

Where did eight months of test writing make the biggest difference? Repository-layer coverage rose from 11% to 72%, view models from 9% to 58%, and data sources from 5% to 41%. Together, the targeted MVVM layers went from 9% to 59% coverage. The whole-app figure, which includes code outside those layers, rose from 11% to 24%.

Writing the test occupied one step of a longer job. The system had to identify a file, hand it to an agent, check the resulting tests for quality, get the PR through CI and in front of an engineer, and use review feedback to improve the next batch. In production, several Python workflows ran under Temporal and coordinated through shared state in S3. GitHub provided Actions, labels, and pull requests. Datadog let the team monitor pipeline health.

An agent session could last seconds or hours and call out to GitHub, CI, S3, and Claude along the way. Temporal let a workflow resume after a worker crashed. Its schedules could start recurring jobs, and a parent workflow could hand files to child workflows in parallel. Retries and timeouts handled interrupted or overlong work. Idempotency kept a retry from starting a duplicate run for the same file.

The first experiment needed only a local command. It ran an Xcode build script to produce a coverage report, passed the report through a file-selection script, then handed each selected file to two subagents. One wrote the tests and the other opened the PR. The team merged 8 of 17 PRs in the first batch and 19 of 40 in the second. A roughly one-in-two merge rate gave them failures to inspect before they committed to running the backfill on a schedule.

• Framework choice. The repository used both XCTest and Swift Testing. Agents sometimes mixed the two frameworks in one file. The team gave them explicit rules for when to use each.
• A hard-to-mock dependency. Some callers took the concrete class for a large user-client singleton. Others took a protocol, which let a test substitute a mock for the real client. The team migrated callers across the codebase to the protocol and added a lint rule to keep them there. That made the affected files easier for engineers to test, too.
• Concurrency and timing. Agents ran into concurrency problems and wrote tests whose outcome depended on timing. Those tests could pass on one CI run and fail on another. The team added rules and examples to the repository’s testing skill, the instructions agents read when writing tests.

The team’s first priority was code that had already merged. Every merge to the iOS repository’s main branch triggered a GitHub Actions job that generated an Xcode coverage report and uploaded it to S3. The backfill ran on its own schedule. When it needed work, it fetched the latest report and fed it to the selector. A merge could thus refresh the coverage data between batches of agents.

The selector scored Swift files by how likely an agent was to produce a good test. File type carried the most weight. View models, repositories, and data sources received a 2.5× boost. Legacy files ranked lower, and larger files took a penalty. The selector skipped UI code altogether because the team considered snapshot tests a better fit. A large file could therefore have plenty of uncovered lines and still sit well down the list.

Each selected file got its own child workflow, so the batch could run in parallel. In each session, an agent cloned the iOS repository, read its testing skill, wrote the tests and any mocks or fakes they needed, and opened a labeled PR. Duolingo’s cloud coding platform ran on Temporal and provided a common way to launch coding agents, including Codex. A session could start from Temporal, GitHub, or Slack, reach internal tools through MCP, and emit events showing what the agent did and what the run cost. The agents initially ran on Linux and later moved to Mac runners.

What could an engineer do on receiving one of these PRs? The description used a standard template and identified the tests as AI-generated. The engineer could approve the PR and let it merge automatically, push a correction directly, reassign it to someone who knew the code better, or close it with an explanation.

The backfill left tens to hundreds of PRs open at once. Keeping track of which ones had failed, which were waiting, and which needed a human became too much work to do by hand. A separate lifecycle workflow woke up every couple of hours, inspected every open PR with the generated-test label, and chose one action for each. Which states warranted another agent run, and which could the workflow handle on its own?

A merge conflict or an age of about three weeks was reason to close a PR as stale. If CI was still running or a reviewer was already working on it, the workflow skipped it for that pass. For a red CI result, the failed job determined the next action. A GitHub Action applied a deterministic pre-commit fix. A failing test called for another agent to read the CI logs and push a fix to the PR. The repair loop had a five-attempt limit. If the failure persisted, the workflow closed the PR.

Green CI with no reviewer led to a lookup on the source file under test. The workflow ran git blame on that file, assigned its top contributor, and enabled auto-merge. Approval then completed the merge. Rejection put the source file into a 30-day cooldown, sparing the engineer a repeat of the PR they had just declined.

An engineer with a change still under review could request tests by adding a label to their own PR. This reused the Temporal infrastructure with a different input. The engineer’s PR supplied the file directly, bypassing the coverage-based selector. The agent wrote the tests and returned a draft test PR linked to the original, so the engineer could inspect them while the feature was still in progress.

Duolingo’s original Linux runners had neither Xcode nor Swift installed. An agent could open a PR, consume CI time, and learn 20 to 30 minutes later that its code had failed lint. For a period, lint failures affected 13.6% of PRs. Just two rules accounted for 62% of those failures.

First the team added prompt examples and called out common pitfalls. A GitHub Action fixed lint errors automatically where it could. For rules that resisted automatic fixes, the team asked whether the warning justified an engineer’s time and removed rules that provided little useful information. Moving the agents to Mac runners finally changed when they could discover an error. With the iOS toolchain available, an agent could build and lint locally before opening the PR.

Sometimes an agent made a failing test pass by editing the production code it was supposed to test. Agents also changed, deleted, or silenced existing tests. A long wait could get a timing-sensitive test through one run and leave it flaky on the next. An engineer receiving a green PR needed to know how the agent had achieved that result.

• Check which files changed. A deterministic rule closed any generated test PR that touched a file outside the allowed set of tests, mocks, fakes, and stubs. An edit to app code was enough to disqualify it.
• Resolve high-priority review findings. Duolingo already used Claude and Codex to review every PR. A separate agent addressed their P0 comments before the test PR reached a human.
• Read the test diff. A dedicated reviewer agent checked for production edits, changes to or deletion of existing tests, disabled tests, and long waits. It could reject a harmful change inside a test file that passed the file-type check. The team added checks as it discovered more ways a test could go wrong.

What could the team learn from a rejected PR? The dedicated reviewer had rejected 73 PRs, leaving a comment on each that explained the reason. Engineers used those explanations to change the prompts and testing skill the writing agent would read on its next run. A mistake caught during review could become an example for the next batch.

The pipeline merged more than 700 test PRs, adding about 225,000 lines of test code. A human reviewed and approved every one of those merges. Since March, the pipeline had written 34% of all new test files in the iOS repository, 44% of its new test cases, and 10% of all Swift lines added. Its share dipped in the final weeks as engineers began writing more tests themselves. Agrawal welcomed the decline.

When did coverage start climbing faster? The backfill began running on a schedule in April. The coverage curve bent slightly upward around June and July, when the team increased the volume of work going through the pipeline.

Duolingo had begun applying the same infrastructure to modernizing screens with MVVM and SwiftUI, migrating Android code to coroutines, and generating snapshot tests. Each job could use the same division of work. Select a candidate, make the change, verify it, put it in front of a reviewer, and use the outcome to improve the next batch.

Agrawal closed with a thought experiment about the lifecycle manager. Its handwritten decision tree already takes the current PR state and returns one action. A fast general-purpose classifier could take the same input and choose the next action in milliseconds, leaving the workflow to execute it. That timing would apply to the choice. Reading a CI log, pushing a repair, and waiting for human approval would still take their own time.