# 225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo

Kush Agrawal, Duolingo

[Talk page](https://ai.engineer/talks/D_4Rn6uLdQM-225-000-lines-tests-written-merged) · [Video](https://www.youtube.com/watch?v=D_4Rn6uLdQM)

## 225,000 Lines of Tests, Written and Merged by Agents

Engineers merged about half the pull requests from Duolingo’s first two batches of agent-written iOS tests. Getting to more than 700 human-approved merges took a way to choose testable files, repair failures, and catch agents that made a test pass by changing the app.

### A weekly release with 11% unit test coverage

[0:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=1s)

Duolingo shipped its mobile apps every week, with more than 600 A/B tests live at a time. Its iOS repository had grown from about 200,000 executable lines of app code in 2019 to nearly 2 million. About 200 people had contributed to the repository the previous year. More than 250 had already contributed in the year of the talk, as AI coding tools helped backend and Android engineers, product managers, and designers ship iOS changes. For Kush Agrawal, a staff engineer on Duolingo’s platform team, that put pressure on the reviewer of each pull request (PR). How could they tell whether a change was safe to ship?

In March, overall unit test coverage stood at 11%. About half the repository followed the Model–View–ViewModel (MVVM) architecture the team had standardized on. The other half was legacy code still being migrated. The MVVM layers were easier to test, yet their combined coverage was only about 9%. That made them a good place to start. A unit test could document expected behavior for the next contributor and give an agent immediate feedback when a later change broke that behavior in continuous integration (CI).

Where did eight months of test writing make the biggest difference? Repository-layer coverage rose from 11% to 72%, view models from 9% to 58%, and data sources from 5% to 41%. Together, the targeted MVVM layers went from 9% to 59% coverage. The whole-app figure, which includes code outside those layers, rose from 11% to 24%.

### Keeping a test PR moving after the agent writes it

[2:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=148s)

Writing the test occupied one step of a longer job. The system had to identify a file, hand it to an agent, check the resulting tests for quality, get the PR through CI and in front of an engineer, and use review feedback to improve the next batch. In production, several Python workflows ran under Temporal and coordinated through shared state in S3. GitHub provided Actions, labels, and pull requests. Datadog let the team monitor pipeline health.

An agent session could last seconds or hours and call out to GitHub, CI, S3, and **Claude** along the way. Temporal let a workflow resume after a worker crashed. Its schedules could start recurring jobs, and a parent workflow could hand files to child workflows in parallel. Retries and timeouts handled interrupted or overlong work. Idempotency kept a retry from starting a duplicate run for the same file.

### Two local batches exposed what made a file hard to test

[4:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=244s)

The first experiment needed only a local command. It ran an Xcode build script to produce a coverage report, passed the report through a file-selection script, then handed each selected file to two subagents. One wrote the tests and the other opened the PR. The team merged 8 of 17 PRs in the first batch and 19 of 40 in the second. A roughly one-in-two merge rate gave them failures to inspect before they committed to running the backfill on a schedule.

- **Framework choice.** The repository used both XCTest and Swift Testing. Agents sometimes mixed the two frameworks in one file. The team gave them explicit rules for when to use each.
- **A hard-to-mock dependency.** Some callers took the concrete class for a large user-client singleton. Others took a protocol, which let a test substitute a mock for the real client. The team migrated callers across the codebase to the protocol and added a lint rule to keep them there. That made the affected files easier for engineers to test, too.
- **Concurrency and timing.** Agents ran into concurrency problems and wrote tests whose outcome depended on timing. Those tests could pass on one CI run and fail on another. The team added rules and examples to the repository’s testing skill, the instructions agents read when writing tests.

### The scheduled backfill picked files an agent could test well

[5:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=321s)

The team’s first priority was code that had already merged. Every merge to the iOS repository’s main branch triggered a GitHub Actions job that generated an Xcode coverage report and uploaded it to S3. The backfill ran on its own schedule. When it needed work, it fetched the latest report and fed it to the selector. A merge could thus refresh the coverage data between batches of agents.

The selector scored Swift files by how likely an agent was to produce a good test. File type carried the most weight. View models, repositories, and data sources received a 2.5× boost. Legacy files ranked lower, and larger files took a penalty. The selector skipped UI code altogether because the team considered snapshot tests a better fit. A large file could therefore have plenty of uncovered lines and still sit well down the list.

Each selected file got its own child workflow, so the batch could run in parallel. In each session, an agent cloned the iOS repository, read its testing skill, wrote the tests and any mocks or fakes they needed, and opened a labeled PR. Duolingo’s cloud coding platform ran on Temporal and provided a common way to launch coding agents, including **Codex**. A session could start from Temporal, GitHub, or Slack, reach internal tools through MCP, and emit events showing what the agent did and what the run cost. The agents initially ran on Linux and later moved to Mac runners.

What could an engineer do on receiving one of these PRs? The description used a standard template and identified the tests as AI-generated. The engineer could approve the PR and let it merge automatically, push a correction directly, reassign it to someone who knew the code better, or close it with an explanation.

### One decision per open PR, every couple of hours

[7:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=441s)

The backfill left tens to hundreds of PRs open at once. Keeping track of which ones had failed, which were waiting, and which needed a human became too much work to do by hand. A separate lifecycle workflow woke up every couple of hours, inspected every open PR with the generated-test label, and chose one action for each. Which states warranted another agent run, and which could the workflow handle on its own?

A merge conflict or an age of about three weeks was reason to close a PR as stale. If CI was still running or a reviewer was already working on it, the workflow skipped it for that pass. For a red CI result, the failed job determined the next action. A GitHub Action applied a deterministic pre-commit fix. A failing test called for another agent to read the CI logs and push a fix to the PR. The repair loop had a five-attempt limit. If the failure persisted, the workflow closed the PR.

Green CI with no reviewer led to a lookup on the source file under test. The workflow ran `git blame` on that file, assigned its top contributor, and enabled auto-merge. Approval then completed the merge. Rejection put the source file into a 30-day cooldown, sparing the engineer a repeat of the PR they had just declined.

What happens on a visit to an open PR?
The failed job determines who attempts a repair. On the passing branch, the workflow assigns the PR and enables auto-merge. The engineer’s subsequent decision determines what happens to the change.
- Inspect open PR — Choose one action for this pass.
- Close as stale — A conflict or an age of about three weeks ends work on the PR.
- Skip this pass — CI or human review is in progress.
- Inspect failed job — The repair loop allows up to five attempts.
- Run automatic fix — A GitHub Action fixes the pre-commit failure.
- Run repair agent — The agent reads the CI logs and pushes a fix.
- Close unresolved PR — The failure persists after five attempts.
- Assign source contributor — Use git blame on the source file and enable auto-merge.
- Merge PR — Human approval triggers auto-merge.
- Defer source file — Rejection starts a 30-day cooldown for this file.
- Inspect open PR → Close as stale (Conflict or age limit)
- Inspect open PR → Skip this pass (CI running or reviewer assigned)
- Inspect open PR → Inspect failed job (CI red)
- Inspect failed job → Run automatic fix (Pre-commit failure, attempts left)
- Inspect failed job → Run repair agent (Test failure, attempts left)
- Inspect failed job → Close unresolved PR (Repair limit reached)
- Inspect open PR → Assign source contributor (CI green and no reviewer)
- Assign source contributor → Merge PR (Reviewer approves)
- Assign source contributor → Defer source file (Reviewer rejects)

### A label lets engineers request tests for new code

[8:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=520s)

An engineer with a change still under review could request tests by adding a label to their own PR. This reused the Temporal infrastructure with a different input. The engineer’s PR supplied the file directly, bypassing the coverage-based selector. The agent wrote the tests and returned a draft test PR linked to the original, so the engineer could inspect them while the feature was still in progress.

### A lint error took 20 to 30 minutes to reach the agent

[9:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=543s)

Duolingo’s original Linux runners had neither Xcode nor Swift installed. An agent could open a PR, consume CI time, and learn 20 to 30 minutes later that its code had failed lint. For a period, lint failures affected 13.6% of PRs. Just two rules accounted for 62% of those failures.

First the team added prompt examples and called out common pitfalls. A GitHub Action fixed lint errors automatically where it could. For rules that resisted automatic fixes, the team asked whether the warning justified an engineer’s time and removed rules that provided little useful information. Moving the agents to Mac runners finally changed when they could discover an error. With the iOS toolchain available, an agent could build and lint locally before opening the PR.

### How did the agent get CI to pass?

[9:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=587s)

Sometimes an agent made a failing test pass by editing the production code it was supposed to test. Agents also changed, deleted, or silenced existing tests. A long wait could get a timing-sensitive test through one run and leave it flaky on the next. An engineer receiving a green PR needed to know how the agent had achieved that result.

- **Check which files changed.** A deterministic rule closed any generated test PR that touched a file outside the allowed set of tests, mocks, fakes, and stubs. An edit to app code was enough to disqualify it.
- **Resolve high-priority review findings.** Duolingo already used Claude and Codex to review every PR. A separate agent addressed their P0 comments before the test PR reached a human.
- **Read the test diff.** A dedicated reviewer agent checked for production edits, changes to or deletion of existing tests, disabled tests, and long waits. It could reject a harmful change inside a test file that passed the file-type check. The team added checks as it discovered more ways a test could go wrong.

What could the team learn from a rejected PR? The dedicated reviewer had rejected 73 PRs, leaving a comment on each that explained the reason. Engineers used those explanations to change the prompts and testing skill the writing agent would read on its next run. A mistake caught during review could become an example for the next batch.

### What the pipeline contributed after March

[10:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=651s)

The pipeline merged more than 700 test PRs, adding about 225,000 lines of test code. A human reviewed and approved every one of those merges. Since March, the pipeline had written 34% of all new test files in the iOS repository, 44% of its new test cases, and 10% of all Swift lines added. Its share dipped in the final weeks as engineers began writing more tests themselves. Agrawal welcomed the decline.

When did coverage start climbing faster? The backfill began running on a schedule in April. The coverage curve bent slightly upward around June and July, when the team increased the volume of work going through the pipeline.

### Reusing the workflow for migrations

[11:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=696s)

Duolingo had begun applying the same infrastructure to modernizing screens with MVVM and SwiftUI, migrating Android code to coroutines, and generating snapshot tests. Each job could use the same division of work. Select a candidate, make the change, verify it, put it in front of a reviewer, and use the outcome to improve the next batch.

Agrawal closed with a thought experiment about the lifecycle manager. Its handwritten decision tree already takes the current PR state and returns one action. A fast general-purpose classifier could take the same input and choose the next action in milliseconds, leaving the workflow to execute it. That timing would apply to the choice. Reading a CI log, pushing a repair, and waiting for human approval would still take their own time.

## Key takeaways

- The selector ranked files by testability. A 2.5× boost for view models, repositories, and data sources directed work toward code where agents had a better chance of producing useful tests.
- Two recurring failures called for changes outside the prompt. A protocol let tests substitute a mock for the user-client singleton. Mac runners let agents build and lint before a PR consumed CI time.
- A test PR could pass CI by weakening an existing test. Checking the changed files caught edits to app code, reviewing the diff covered changes inside test files, and an engineer approved every merge.

## Slides

### How AI Agents Shipped over 200k Lines of Tests on a 1.8M Line codebase

[0:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=0s) [A green Duolingo slide introduces Kush Agrawal’s talk on using AI agents to ship over 200,000 lines of tests on a 1.8-million-line codebase.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d66845bd44a2d3d9d52f3fcb3b0eea371cd3fdaa5239242da9d5575b189b6c4.webp)

duolingo
How AI Agents Shipped over
200k Lines of Tests on a 1.8M
Line codebase
Kush Agrawal • Staff Software Engineer, iOS Platform • Duolingo

### Kush Agrawal

[0:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=11s) [Beside a portrait, the slide identifies Kush Agrawal as a Duolingo staff software engineer who builds developer tools and AI-assisted workflows for 100+ client engineers. It lists previous roles at Stripe and Uber.](https://assets.ai.engineer/sites/aiecode2025/sha256/5c806df5687cc628cc79594c8b0d3c11f43c29a719b7b841eec67119878f6cc3.webp)

Kush Agrawal
→ Staff Software Engineer, Platform at Duolingo
→ Developer tooling & AI-assisted workflows for
100+ Client engineers
→ Previously Stripe, Uber

### Duolingo Mobile at Scale

[0:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=25s) [Three columns show weekly App Store releases, 600+ concurrent experiments, and 10+ years of shipping.](https://assets.ai.engineer/sites/aiecode2025/sha256/568895604d487a72e13e00d79267459f9198e6f7355e932915630e2a7e7f7e36.webp)

Duolingo Mobile at Scale
weekly
App Store releases
600+
concurrent experiments
10+
years of shipping

### Executable lines of app code

[0:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=40s) [A green line rises from roughly 200,000 executable lines of iOS app code in 2019 to 1.86 million at the point labeled “now.”](https://assets.ai.engineer/sites/aiecode2025/sha256/32f644a07afb63fa6757901d8cc256e23a51291635f78a6c7f39221f0af02d57.webp)

Executable lines of app code
2M
1.86M
1M
0
2019
2020
2021
2022
2023
2024
2025
2026
now

### Distinct Committers per Year

[0:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=54s) [Annual bars show 66, 74, 97, 138, 143, 165, and 198 distinct committers from 2019 through 2025, followed by 259 in 2026 year to date.](https://assets.ai.engineer/sites/aiecode2025/sha256/7638eb781f10ee94f0ac999c795e83cafe75a8e4fd0e7b50449b9443d2aa682c.webp)

Distinct Committers per Year
2019: 66. 2020: 74. 2021: 97. 2022: 138. 2023: 143. 2024: 165. 2025: 198. 2026 YTD: 259.

### ~50% of the iOS Codebase is MVVM

[1:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=73s) [A bar splits the iOS codebase into roughly equal MVVM and legacy halves. Beneath the MVVM half, arrows connect View to ViewModel to Repository to DataSource.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4d27f866e388b51aa12826621dec4e0f8242bbb9ae6c5e056c4ee594ba1a435.webp)

~50% of the iOS Codebase is MVVM
50%
MVVM
View → ViewModel → Repository → DataSource
50%
legacy

### March 2026 unit test coverage

[1:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=90s) [Overall unit test coverage is 11%. The MVVM chart shows Repository at 11%, ViewModel at 9%, DataSource at 5%, and the three components combined at 9%.](https://assets.ai.engineer/sites/aiecode2025/sha256/4466728591c7ddc6e11128d32e0ae50a2df28ad3b703c829004e7c0542a8ae27.webp)

March 2026
11% overall unit test coverage
MVVM components
Repository 11%
ViewModel 9%
DataSource 5%
Combined 9%

### Verification becomes the bottleneck

[1:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=105s) [White text on green reads, “As agents and humans write more code, verification becomes the bottleneck.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d04d874a5a38862006f407ee3e89d2202de7febf0751532d214246211c933860.webp)

As agents and humans write more code,
verification becomes the bottleneck.

### Unit Tests as Guardrails

[1:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=119s) [Unit tests verify each change in seconds, let agents check their own work, and document expected behavior. The pyramid places unit tests at the base, integration tests in the middle, and UI/QA at the top.](https://assets.ai.engineer/sites/aiecode2025/sha256/c20e666dc4ed93407aafbb808044cdf515a2b95ec0c77911df322f33e45da268.webp)

Unit Tests as Guardrails
Fast verification
seconds, on every change
Agent guardrails
agents check their own work
Living documentation
what the code is supposed to do
UI / QA
integration
unit

### November 2026 unit test coverage

[2:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=133s) [Overall unit test coverage is 24%. From March to November, Repository coverage rose from 11% to 72%, ViewModel from 9% to 58%, and DataSource from 5% to 41%. Combined MVVM coverage rose from 9% to 59%.](https://assets.ai.engineer/sites/aiecode2025/sha256/a25b003568ed7a131c00cb8e08c4ffd59dc1372b1d51194f04296bb1e737bdc4.webp)

November 2026
24% overall unit test coverage
MVVM components
Repository 11→72%
ViewModel 9→58%
DataSource 5→41%
Combined 9→59%

### On Paper

[2:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=149s) [Five boxes run left to right in order: Identify, Write, Verify, Resolve, Iterate. An arrow from Iterate back to Identify reads “each run improves the next.”](https://assets.ai.engineer/sites/aiecode2025/sha256/6f04c89fb6de2d7d05acf9a9b088a1d46abaaae56068ed270080c9edad180d3a.webp)

On Paper
01 Identify → 02 Write → 03 Verify → 04 Resolve → 05 Iterate
each run improves the next

### In Production

[2:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=164s) [Schedules and pull request labels trigger Temporal workflows. A coding agent generates tests and opens pull requests. A lifecycle manager assigns reviewers, applies labels, and triggers fixes when CI fails. GitHub Actions uploads coverage to S3, which also stores workflow state.](https://assets.ai.engineer/sites/aiecode2025/sha256/d653351db579c8509cbf90719ec60ff0fb91068183c7ef446f853d1e919caaa4.webp)

In Production
Triggers: Cron schedules (1h / 2h), PR label
Orchestration (Temporal): Test Backfill, PR Lifecycle Manager, CI Auto-Heal
Code Generation: CodingAgent (Claude Code)
select files; generate tests; create PR; scan PRs; assign reviewer / apply labels; on failure; push fix; track state
GitHub: duolingo-ios PRs, CI / GitHub Actions
AWS S3: Coverage + State Store; upload coverage

### What We Built On

[2:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=177s) [Five cards show the tools and their roles. Temporal runs workflows, Claude Code acts as the coding agent, S3 shares state across workflows, GitHub handles Actions, labels and pull requests, and Datadog provides observability dashboards.](https://assets.ai.engineer/sites/aiecode2025/sha256/f788548abc986963c98dd5a01a59ab0a1b06f7bd8db7896fcb5f430d4e9e8ed5.webp)

What We Built On
Temporal
Workflow orchestration & execution
S3
Share state across workflows
Datadog
Observability dashboards
Claude Code
Coding Agent
GitHub
Actions, Labels and PRs

### Why Temporal

[3:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=194s) [The left column describes agent sessions that run for seconds to hours, call external tools, and handle waits and failures. The right column lists Temporal schedules, parallel child workflows, Nexus triggers, retries and timeouts, and idempotency to prevent duplicate runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b5851325a6b76af062c120f9f2eb9ce70e2fe9458597b56e6ba3276314299f3.webp)

Why Temporal
A durable workflow execution engine.
AGENT SESSIONS
Run for hours: Seconds to Hours
Call external tools: GitHub, CI, S3, Claude
Wait and fail safely: on humans, timeouts, flakes
FEATURES WE USE
Schedules: cron-like runs
Child workflows: parallel fan-out
Nexus: trigger from anywhere
Retries & timeouts: first-class
Idempotency: no duplicate runs

### Four Milestones

[3:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=225s) [Four numbered cards run left to right. Local validation tunes prompts, scheduled backfill reduces the backlog, the PR lifecycle workflow handles repair through merge, and a label triggers tests on demand. The two middle cards are highlighted in green.](https://assets.ai.engineer/sites/aiecode2025/sha256/54304ebd87b37f1d9114f7b8f79f3d5fb1e84b8d30834b321edf0a2a56a915f8.webp)

Four Milestones
1
Local validation
Run locally and tune
prompts
2
Scheduled
backfill workflow
Draw down the backlog
autonomously
3
PR lifecycle
workflow
Heal, Review, Assign,
Merge
4
Label trigger
Tests on demand

### Local Validation

[4:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=244s) [A flowchart runs from the Claude Code orchestrator through reading code coverage to a test selection script. The script branches to a write-tests agent and a create-test-pr agent. Both branches point to a pull request on GitHub.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9f6caf2efbc64b6913aa75c63bfe476b4dc52c4de66ffc729c97fbad1e5bb86.webp)

Local Validation
write-tests
agent
Claude Code Orchestrator
Read code
coverage
Test selection
script
create-test-pr
agent
PR on GitHub

### Local PR Batches

[4:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=271s) [Batch 1 merged 8 of 17 generated pull requests (47%). Batch 2 merged 19 of 40 (48%). Green bars show the two merge rates side by side.](https://assets.ai.engineer/sites/aiecode2025/sha256/7fbf7b45812c62659ee63c2443e20d1ce2916c951208348360c019879500d453.webp)

Local PR Batches
Batch 1: 17 Total Generated → 8 Merged. Merge Rate 47%. 8 out of 17 pull requests merged.
Batch 2: 40 Total Generated → 19 Merged. Merge Rate 48%. 19 out of 40 pull requests merged.

### Failure Modes

[4:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=284s) [Four rows pair failures with fixes. Mixing Swift Testing and XCTest prompted guidance on framework choice. A concrete client prompted a protocol and lint rule. Swift 6 Sendable errors prompted guidance and MainActor annotations on mocks. Missing access control and timing-based tests prompted updated rules and examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/bfa5f96b08328617e03c6e5591955f43e34fbb1f9fb87c5e0e671404925e2ca3.webp)

Failure Modes
FAILURE → FIX
Mixed Swift Testing & XCTest in one file → Give Agents clear instructions on when to pick each
Concrete UserClient, nothing injectable → Migrated the codebase to the protocol + lint rule
Swift 6 Sendable errors → Prompt guidance; mocks annotated @MainActor
Missing access control, timing based tests → Rules and updated examples in the skills

### Scheduled Backfill Workflow

[5:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=328s) [In the green Identify stage, a scheduled workflow ranks and filters files, then selects up to five. In the blue Write stage, parallel coding agents create test pull requests and record them in S3.](https://assets.ai.engineer/sites/aiecode2025/sha256/02586e95afa2d4da3bfc967929d199087527ca4024d44e14d2e8def7d24a1a85.webp)

Scheduled Backfill Workflow
Identify. Temporal Scheduler: Trigger (every ~2h). fetch_and_select_files activity: Fetch coverage from S3; coverage_selector.py (score & rank). S3: Coverage JSON. filter_already_tested_files activity: Filter out files with existing tests, open PRs, or 30-day cooldown. Files remaining? No → Exit (nothing to do); Yes → Select top N (max 5).
Write. IOSTestGenerationWorkflow ×N (child workflows). CodingAgent 1, CodingAgent 2, CodingAgent N: Clone → Test → PR. record_pr_created activity: Record PR in S3 state store. S3 State Store: files/ + prs/.

### Identify: Test coverage report

[5:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=342s) [After a pull request merges, GitHub Actions uploads an Xcode coverage report to S3. The Temporal backfill workflow fetches the report and uses a selector script to pick the top five files.](https://assets.ai.engineer/sites/aiecode2025/sha256/07de3b967dca5151f51d31198f380a1440bddc53c21a22d2fd3c559642ef9a18.webp)

Identify: Test coverage report
GitHub Actions: PR merge, XCode coverage report
upload → S3 → fetch
Temporal backfill: Selector script → Top 5 files

### Identify: Fetch and Score Files

[5:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=358s) [File type has the largest weight at 2.5×, with ViewModels, Repositories and DataSources listed. Coverage gap and module priority each carry 0.5×. Size carries a −0.3× penalty.](https://assets.ai.engineer/sites/aiecode2025/sha256/360f0976b4b26668f157618ffeddd99fabf075d91221428df72b269e297ff97b.webp)

Identify: Fetch and Score Files
File type
ViewModels, Repositories, DataSources
2.5x
Coverage gap
SECONDARY FACTOR
0.5x
Module priority
SECONDARY FACTOR
0.5x
Size
PENALTY DEDUCTION
-0.3x

### Write: Duolingo’s Cloud Coding Agent

[6:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=381s) [A Temporal workflow routes jobs to a Linux EC2 fleet or macOS runners. Both paths lead to Claude Code or Codex, which connects to GitHub and MCP servers. Separate branches show S3 sessions and cost and live events.](https://assets.ai.engineer/sites/aiecode2025/sha256/92d1011bacfac7a458fac8ade420be24e7e05bcb7555e45edb3c64d57f167456.webp)

Write: Duolingo’s Cloud Coding Agent
Cloud Coding Agent (Temporal): Coding agent workflow; S3 sessions; Cost + live events
Compute: Linux EC2 fleet, macOS runners
Agent: Claude Code / Codex; GitHub; MCP servers

### PR Lifecycle Workflow

[7:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=451s) [A scheduler finds open pull requests for generated tests. Arrows lead from a CI status check to verification actions in the yellow Verify section and reviewer assignment in the blue Resolve section.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e1381db62ee84f3e5c7bea89db1dacc9ac333f07d30dd98df2040bc8953267c.webp)

PR Lifecycle Workflow
Temporal Scheduler: Trigger (1h). fetch_generated_test_prs activity: Find open PRs with generated-test label. CI status? Green + unassigned, In progress, Green + assigned, Red.
Resolve: /assign (auto-assign reviewer); assign_reviewer activity: git blame → employee check; Enable auto-merge. mark_pr_gave_up activity: Give up (close PR + 30d cooldown); set cooldown; S3 State Store files/ + prs/.
Verify: process_pr activity. Skip. Skip. Max retries exceeded. Which job failed? Pre-commit → Add pre-commit label; Other CI → Add fix-ci label. External triggers: GHA: pre-commit run → commit & push fixes; CI auto-heal workflow.

### Decision Tree

[7:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=469s) [Five rows map PR conditions to actions. Close a stale or conflicting PR. Skip one with pending CI or an assigned reviewer. Assign a reviewer when CI passes and none is assigned. Automatically fix CI or pre-commit failures. Start a 30-day cooldown after reviewer rejection.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ae8a27063e196e3d35b6891eec866f312d2381b7e6982d4a5180124a33cd884.webp)

Decision Tree
1 Merge conflict or stale? → Close
2 CI pending or has a reviewer? → Skip
3 CI green, no reviewer? → Assign a reviewer
4 CI red or pre-commit failure? → Auto Fix CI
5 Rejected by the reviewer? → 30 day cool down

### Verify: Auto-Heal CI

[8:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=489s) [A CI failure branches by the job that failed. Pre-commit failures trigger pre-commit fixes. For unit test failures, an agent reads the logs and pushes a fix.](https://assets.ai.engineer/sites/aiecode2025/sha256/c144416d8ead2c2ee78488e575bf62417fd95bcbd59837ac8b55abab01a4b842.webp)

Verify: Auto-Heal CI
CI failure → Which job failed?
pre-commit → Apply pre-commit fixes
unit tests → Agent reads logs + pushes a fix

### Resolve: Reviewer Assignment

[8:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=506s) [After CI passes, the workflow uses git blame to identify the top contributor and assigns them as reviewer. Approval leads to a merge. Rejection starts a 30-day cooldown.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e2f68c7db08af10f28edb2e48ae1b94beca4f8dd4c078cc7af55a30f7fbc7be.webp)

Resolve: Reviewer Assignment
CI green → git blame → Top contributor → Assign
approved → Merge
rejected → 30-day cooldown

### Add a Label, Get a Draft PR of Tests

[8:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=528s) [Adding the “write tests” label to a pull request starts a Temporal workflow. It passes the PR diff and description to a coding agent, which writes tests for a draft PR linked to the original.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c9c6203e55bc432e9c3a2b1d2c983507c69ddbae6e8fe1c17740f90963b0e28.webp)

Add a Label, Get a Draft PR of Tests
Backfill workflow
Your PR
• write tests
add a label
>
Temporal
PR diff +
description
→
Coding agent
Writes tests
Draft test PR
Linked to your PR

## Full transcript

[0:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=1s) Hey everyone, thanks for tuning in.

[0:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=3s) Hey everyone, thanks for tuning in.

[0:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=3s) Hey everyone, thanks for tuning in. Today I'm going to walk you through how

[0:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=4s) Today I'm going to walk you through how

[0:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=4s) Today I'm going to walk you through how we use AI agents at Dualingo to write

[0:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=5s) we use AI agents at Dualingo to write

[0:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=6s) we use AI agents at Dualingo to write and ship more than 200,000 thousand

[0:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=7s) and ship more than 200,000 thousand

[0:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=7s) and ship more than 200,000 thousand lines of unit tests on iOS codebase.

[0:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=9s) lines of unit tests on iOS codebase.

[0:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=9s) lines of unit tests on iOS codebase. I'll share what we built and what we

[0:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=11s) I'll share what we built and what we

[0:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=11s) I'll share what we built and what we learned along the way. Uh I'm Kush. I'm

[0:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=13s) learned along the way. Uh I'm Kush. I'm

[0:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=13s) learned along the way. Uh I'm Kush. I'm a staff engineer on the platform team

[0:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=14s) a staff engineer on the platform team

[0:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=14s) a staff engineer on the platform team here at Dualingo. My job is to build

[0:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=15s) here at Dualingo. My job is to build

[0:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=15s) here at Dualingo. My job is to build developer tooling and AI assisted

[0:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=16s) developer tooling and AI assisted

[0:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=16s) developer tooling and AI assisted workflows for the 100 plus engineers who

[0:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=18s) workflows for the 100 plus engineers who

[0:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=18s) workflows for the 100 plus engineers who contribute to our client apps. Before

[0:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=19s) contribute to our client apps. Before

[0:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=19s) contribute to our client apps. Before Duelingo, I spent close to a decade at

[0:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=21s) Duelingo, I spent close to a decade at

[0:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=21s) Duelingo, I spent close to a decade at Stripe and Uber working across a few

[0:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=22s) Stripe and Uber working across a few

[0:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=22s) Stripe and Uber working across a few teams there. Let's start with some

[0:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=24s) teams there. Let's start with some

[0:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=24s) teams there. Let's start with some background on the scale of mobile at

[0:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=26s) background on the scale of mobile at

[0:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=26s) background on the scale of mobile at Dualingo. Uh we ship our apps to the app

[0:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=29s) Dualingo. Uh we ship our apps to the app

[0:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=29s) Dualingo. Uh we ship our apps to the app store every week. Um we're also a very

[0:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=31s) store every week. Um we're also a very

[0:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=31s) store every week. Um we're also a very experiment-driven company. So at any

[0:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=32s) experiment-driven company. So at any

[0:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=32s) experiment-driven company. So at any given point of time, we have more than

[0:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=34s) given point of time, we have more than

[0:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=34s) given point of time, we have more than 600 AB tests running live and our code

[0:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=36s) 600 AB tests running live and our code

[0:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=36s) 600 AB tests running live and our code bases are pretty mature. Both the iOS

[0:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=38s) bases are pretty mature. Both the iOS

[0:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=38s) bases are pretty mature. Both the iOS and Android apps have been around for

[0:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=39s) and Android apps have been around for

[0:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=39s) and Android apps have been around for more than 10 years at this point.

[0:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=42s) more than 10 years at this point.

[0:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=42s) more than 10 years at this point. Here's also the number of executable

[0:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=43s) Here's also the number of executable

[0:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=43s) Here's also the number of executable lines of app code in our iOS repository.

[0:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=45s) lines of app code in our iOS repository.

[0:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=45s) lines of app code in our iOS repository. So as you can see back in 2019, we were

[0:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=47s) So as you can see back in 2019, we were

[0:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=47s) So as you can see back in 2019, we were at about 200,000 lines and today we're

[0:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=49s) at about 200,000 lines and today we're

[0:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=49s) at about 200,000 lines and today we're hit close to 2 million. And as AI

[0:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=51s) hit close to 2 million. And as AI

[0:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=51s) hit close to 2 million. And as AI assisted coding picks up, you can see

[0:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=52s) assisted coding picks up, you can see

[0:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=52s) assisted coding picks up, you can see that curve starting to bend upward.

[0:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=55s) that curve starting to bend upward.

[0:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=55s) that curve starting to bend upward. And it's not just the amount of code

[0:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=56s) And it's not just the amount of code

[0:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=56s) And it's not just the amount of code that's changing, it's also who's writing

[0:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=57s) that's changing, it's also who's writing

[0:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=57s) that's changing, it's also who's writing it. So last year about 200 people

[1:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=60s) it. So last year about 200 people

[1:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=60s) it. So last year about 200 people contributed to the iOS repository. And

[1:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=61s) contributed to the iOS repository. And

[1:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=61s) contributed to the iOS repository. And this year we've already crossed 250. And

[1:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=63s) this year we've already crossed 250. And

[1:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=63s) this year we've already crossed 250. And a lot of these folks aren't even iOS

[1:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=64s) a lot of these folks aren't even iOS

[1:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=64s) a lot of these folks aren't even iOS engineers. We have backend engineers,

[1:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=66s) engineers. We have backend engineers,

[1:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=66s) engineers. We have backend engineers, Android engineers, even PMs and

[1:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=67s) Android engineers, even PMs and

[1:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=67s) Android engineers, even PMs and designers all shipping code to the iOS

[1:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=69s) designers all shipping code to the iOS

[1:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=69s) designers all shipping code to the iOS app. And that's mostly been unlocked by

[1:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=71s) app. And that's mostly been unlocked by

[1:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=71s) app. And that's mostly been unlocked by AI coding tools like cursor and cloud

[1:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=73s) AI coding tools like cursor and cloud

[1:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=73s) AI coding tools like cursor and cloud code. Um, another thing to note about

[1:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=75s) code. Um, another thing to note about

[1:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=75s) code. Um, another thing to note about our codebase is that it's a codebase in

[1:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=76s) our codebase is that it's a codebase in

[1:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=76s) our codebase is that it's a codebase in transition. So a few years ago, we

[1:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=78s) transition. So a few years ago, we

[1:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=78s) transition. So a few years ago, we standardized on the MVVM architecture

[1:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=79s) standardized on the MVVM architecture

[1:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=79s) standardized on the MVVM architecture pattern. Today about half the code in

[1:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=81s) pattern. Today about half the code in

[1:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=81s) pattern. Today about half the code in our repository follows that pattern and

[1:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=83s) our repository follows that pattern and

[1:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=83s) our repository follows that pattern and the other half is the legacy code that

[1:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=84s) the other half is the legacy code that

[1:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=84s) the other half is the legacy code that we're in the process of migrating. Since

[1:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=86s) we're in the process of migrating. Since

[1:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=86s) we're in the process of migrating. Since the MVVM half is the easiest to test,

[1:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=87s) the MVVM half is the easiest to test,

[1:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=88s) the MVVM half is the easiest to test, that's where we focused on this project.

[1:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=91s) that's where we focused on this project.

[1:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=91s) that's where we focused on this project. Here's where we started back in March of

[1:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=92s) Here's where we started back in March of

[1:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=92s) Here's where we started back in March of this year. We were at overall unit test

[1:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=95s) this year. We were at overall unit test

[1:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=95s) this year. We were at overall unit test coverage of 11%. The MVVM layers were a

[1:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=97s) coverage of 11%. The MVVM layers were a

[1:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=97s) coverage of 11%. The MVVM layers were a bit lower, repositories were at 11%,

[1:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=99s) bit lower, repositories were at 11%,

[1:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=99s) bit lower, repositories were at 11%, view models at 9, data sources at 5, and

[1:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=101s) view models at 9, data sources at 5, and

[1:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=101s) view models at 9, data sources at 5, and combined we are about 9% for the MVM

[1:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=103s) combined we are about 9% for the MVM

[1:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=103s) combined we are about 9% for the MVM slice of the app.

[1:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=106s) slice of the app.

[1:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=106s) slice of the app. So why does this matter now more than it

[1:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=108s) So why does this matter now more than it

[1:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=108s) So why does this matter now more than it used to? As agents and humans write more

[1:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=110s) used to? As agents and humans write more

[1:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=110s) used to? As agents and humans write more code and write it faster, writing code

[1:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=112s) code and write it faster, writing code

[1:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=112s) code and write it faster, writing code isn't the hard part anymore. Verifying

[1:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=113s) isn't the hard part anymore. Verifying

[1:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=113s) isn't the hard part anymore. Verifying it is. So when someone puts up a PR, how

[1:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=115s) it is. So when someone puts up a PR, how

[1:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=116s) it is. So when someone puts up a PR, how do you know that it's actually safe to

[1:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=117s) do you know that it's actually safe to

[1:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=117s) do you know that it's actually safe to ship? The bottleneck is really shifting

[1:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=118s) ship? The bottleneck is really shifting

[1:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=118s) ship? The bottleneck is really shifting from writing to verifying. And that's

[2:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=120s) from writing to verifying. And that's

[2:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=120s) from writing to verifying. And that's where unit tests come in. They make up

[2:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=122s) where unit tests come in. They make up

[2:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=122s) where unit tests come in. They make up the base of the testing pyramid. They're

[2:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=124s) the base of the testing pyramid. They're

[2:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=124s) the base of the testing pyramid. They're pretty cheap to write and pass to run.

[2:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=125s) pretty cheap to write and pass to run.

[2:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=125s) pretty cheap to write and pass to run. And they also act as guardrails for

[2:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=126s) And they also act as guardrails for

[2:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=126s) And they also act as guardrails for agents. So when an agent's change breaks

[2:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=128s) agents. So when an agent's change breaks

[2:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=128s) agents. So when an agent's change breaks a test in CI, it gets immediate feedback

[2:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=130s) a test in CI, it gets immediate feedback

[2:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=130s) a test in CI, it gets immediate feedback on what to fix. They also act as living

[2:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=132s) on what to fix. They also act as living

[2:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=132s) on what to fix. They also act as living documentation on what the code is

[2:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=133s) documentation on what the code is

[2:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=133s) documentation on what the code is supposed to do. And here's where we

[2:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=135s) supposed to do. And here's where we

[2:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=135s) supposed to do. And here's where we ended up 8 months later. Overall unit

[2:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=136s) ended up 8 months later. Overall unit

[2:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=136s) ended up 8 months later. Overall unit test coverage went from 11% to 24%.

[2:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=139s) test coverage went from 11% to 24%.

[2:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=139s) test coverage went from 11% to 24%. Repositories went from 11 to 72. View

[2:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=141s) Repositories went from 11 to 72. View

[2:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=141s) Repositories went from 11 to 72. View models from 9 to 58. Data sources from 5

[2:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=143s) models from 9 to 58. Data sources from 5

[2:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=143s) models from 9 to 58. Data sources from 5 to 41. And the combined MVM slice of the

[2:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=145s) to 41. And the combined MVM slice of the

[2:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=145s) to 41. And the combined MVM slice of the app went from 9% to 59%. So what did we

[2:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=148s) app went from 9% to 59%. So what did we

[2:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=148s) app went from 9% to 59%. So what did we actually end up building? On paper, it's

[2:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=150s) actually end up building? On paper, it's

[2:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=150s) actually end up building? On paper, it's a pretty simple loop. First, you

[2:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=151s) a pretty simple loop. First, you

[2:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=152s) a pretty simple loop. First, you identify the files that need tests. Then

[2:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=153s) identify the files that need tests. Then

[2:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=153s) identify the files that need tests. Then you write those tests with an agent. You

[2:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=155s) you write those tests with an agent. You

[2:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=155s) you write those tests with an agent. You verify that the tests are good quality

[2:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=156s) verify that the tests are good quality

[2:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=156s) verify that the tests are good quality and they pass CI. Then you resolve the

[2:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=158s) and they pass CI. Then you resolve the

[2:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=158s) and they pass CI. Then you resolve the PR, which means getting attributed by an

[2:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=159s) PR, which means getting attributed by an

[2:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=159s) PR, which means getting attributed by an engineer merged. And finally, it on the

[2:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=161s) engineer merged. And finally, it on the

[2:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=161s) engineer merged. And finally, it on the feedback so that the next batch of PRs

[2:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=163s) feedback so that the next batch of PRs

[2:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=163s) feedback so that the next batch of PRs is better than the last one. But in

[2:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=165s) is better than the last one. But in

[2:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=165s) is better than the last one. But in production, it's a little messier. This

[2:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=167s) production, it's a little messier. This

[2:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=167s) production, it's a little messier. This is what our system ended up looking

[2:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=168s) is what our system ended up looking

[2:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=168s) is what our system ended up looking like. I won't go through every box right

[2:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=169s) like. I won't go through every box right

[2:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=169s) like. I won't go through every box right now, but at a high level, it's a few

[2:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=170s) now, but at a high level, it's a few

[2:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=170s) now, but at a high level, it's a few temporal workflows that coordinate

[2:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=172s) temporal workflows that coordinate

[2:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=172s) temporal workflows that coordinate through shared state stored in S3. And

[2:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=174s) through shared state stored in S3. And

[2:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=174s) through shared state stored in S3. And every box you see here maps back to one

[2:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=176s) every box you see here maps back to one

[2:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=176s) every box you see here maps back to one of the five steps I just talked about.

[2:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=178s) of the five steps I just talked about.

[2:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=178s) of the five steps I just talked about. Here are the building blocks that we

[2:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=179s) Here are the building blocks that we

[2:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=179s) Here are the building blocks that we ended up using. Temporal handles

[3:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=181s) ended up using. Temporal handles

[3:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=181s) ended up using. Temporal handles workflow orchestration and execution.

[3:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=183s) workflow orchestration and execution.

[3:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=183s) workflow orchestration and execution. Cloud code acts as the coding agent. S3

[3:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=185s) Cloud code acts as the coding agent. S3

[3:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=185s) Cloud code acts as the coding agent. S3 holds a shared state that keeps our

[3:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=186s) holds a shared state that keeps our

[3:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=186s) holds a shared state that keeps our workflows in sync. And we use GitHub for

[3:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=188s) workflows in sync. And we use GitHub for

[3:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=188s) workflows in sync. And we use GitHub for actions, labels, and PRs. Data dogs acts

[3:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=191s) actions, labels, and PRs. Data dogs acts

[3:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=191s) actions, labels, and PRs. Data dogs acts as observability layer where we keep an

[3:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=192s) as observability layer where we keep an

[3:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=192s) as observability layer where we keep an eye on the health of our pipelines.

[3:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=195s) eye on the health of our pipelines.

[3:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=195s) eye on the health of our pipelines. If you haven't used empodoral before,

[3:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=196s) If you haven't used empodoral before,

[3:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=196s) If you haven't used empodoral before, it's basically a durable workflow

[3:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=197s) it's basically a durable workflow

[3:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=198s) it's basically a durable workflow execution engine. You write your

[3:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=199s) execution engine. You write your

[3:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=199s) execution engine. You write your workflows as normal Python code and if a

[3:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=201s) workflows as normal Python code and if a

[3:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=201s) workflows as normal Python code and if a worker crashes, the workflow just picks

[3:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=202s) worker crashes, the workflow just picks

[3:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=202s) worker crashes, the workflow just picks up right where it left off. This fits

[3:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=204s) up right where it left off. This fits

[3:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=204s) up right where it left off. This fits agents really well and agent session can

[3:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=206s) agents really well and agent session can

[3:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=206s) agents really well and agent session can run for seconds to hours and it can call

[3:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=208s) run for seconds to hours and it can call

[3:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=208s) run for seconds to hours and it can call out to tools like GitHub, CI, S3, and

[3:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=210s) out to tools like GitHub, CI, S3, and

[3:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=210s) out to tools like GitHub, CI, S3, and Claude. We also lean on a few built-in

[3:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=212s) Claude. We also lean on a few built-in

[3:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=212s) Claude. We also lean on a few built-in features of Temporal. There are

[3:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=214s) features of Temporal. There are

[3:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=214s) features of Temporal. There are schedules which are basically chronike

[3:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=215s) schedules which are basically chronike

[3:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=215s) schedules which are basically chronike schedules for our jobs. There are

[3:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=216s) schedules for our jobs. There are

[3:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=216s) schedules for our jobs. There are composible workflows so that we can fan

[3:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=218s) composible workflows so that we can fan

[3:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=218s) composible workflows so that we can fan out work in parallel. Then there's first

[3:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=220s) out work in parallel. Then there's first

[3:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=220s) out work in parallel. Then there's first class support for retries and timeouts

[3:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=221s) class support for retries and timeouts

[3:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=221s) class support for retries and timeouts and item potency so that we never kick

[3:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=223s) and item potency so that we never kick

[3:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=223s) and item potency so that we never kick off duplicate runs of the same file.

[3:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=225s) off duplicate runs of the same file.

[3:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=225s) off duplicate runs of the same file. To split up this work, we split this

[3:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=227s) To split up this work, we split this

[3:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=227s) To split up this work, we split this project into four milestones. The first

[3:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=229s) project into four milestones. The first

[3:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=229s) project into four milestones. The first was local validation. Basically, is this

[3:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=231s) was local validation. Basically, is this

[3:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=231s) was local validation. Basically, is this project even viable? The second was a

[3:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=232s) project even viable? The second was a

[3:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=232s) project even viable? The second was a scheduled backfield workflow to write

[3:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=234s) scheduled backfield workflow to write

[3:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=234s) scheduled backfield workflow to write tests for existing code. The third was a

[3:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=236s) tests for existing code. The third was a

[3:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=236s) tests for existing code. The third was a PR life cycle workflow to actually get

[3:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=238s) PR life cycle workflow to actually get

[3:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=238s) PR life cycle workflow to actually get those PRs merged. And the fourth was a

[3:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=239s) those PRs merged. And the fourth was a

[3:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=239s) those PRs merged. And the fourth was a label trigger so engineers can get tests

[4:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=241s) label trigger so engineers can get tests

[4:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=241s) label trigger so engineers can get tests for the new code that they're writing.

[4:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=243s) for the new code that they're writing.

[4:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=243s) for the new code that they're writing. Now, we'll go through each of these in

[4:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=244s) Now, we'll go through each of these in

[4:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=244s) Now, we'll go through each of these in order. Before building anything real, we

[4:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=246s) order. Before building anything real, we

[4:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=246s) order. Before building anything real, we wanted to do the least amount of work

[4:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=247s) wanted to do the least amount of work

[4:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=247s) wanted to do the least amount of work possible that would tell us whether an

[4:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=249s) possible that would tell us whether an

[4:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=249s) possible that would tell us whether an agent could actually do this job well

[4:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=250s) agent could actually do this job well

[4:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=250s) agent could actually do this job well and do it at scale. We started with a

[4:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=252s) and do it at scale. We started with a

[4:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=252s) and do it at scale. We started with a very simple local cloud code setup where

[4:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=254s) very simple local cloud code setup where

[4:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=254s) very simple local cloud code setup where we had one cloud code command that acted

[4:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=255s) we had one cloud code command that acted

[4:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=255s) we had one cloud code command that acted as an orchestrator. It ran an Xcode

[4:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=257s) as an orchestrator. It ran an Xcode

[4:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=258s) as an orchestrator. It ran an Xcode build script to generate a code coverage

[4:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=259s) build script to generate a code coverage

[4:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=259s) build script to generate a code coverage report locally. Then it fed that report

[4:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=261s) report locally. Then it fed that report

[4:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=261s) report locally. Then it fed that report through a file selection script that we

[4:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=262s) through a file selection script that we

[4:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=262s) through a file selection script that we wrote and then it handed each selected

[4:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=264s) wrote and then it handed each selected

[4:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=264s) wrote and then it handed each selected file off of sub two sub aents. One wrote

[4:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=267s) file off of sub two sub aents. One wrote

[4:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=267s) file off of sub two sub aents. One wrote the tests and the other one opened the

[4:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=268s) the tests and the other one opened the

[4:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=268s) the tests and the other one opened the PR. We ran this in a loop and generated

[4:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=270s) PR. We ran this in a loop and generated

[4:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=270s) PR. We ran this in a loop and generated a bunch of PRs. We basically ran this in

[4:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=272s) a bunch of PRs. We basically ran this in

[4:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=272s) a bunch of PRs. We basically ran this in two batches. The first patch had 17 PRs

[4:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=274s) two batches. The first patch had 17 PRs

[4:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=274s) two batches. The first patch had 17 PRs and we merged eight of them. The second

[4:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=276s) and we merged eight of them. The second

[4:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=276s) and we merged eight of them. The second had 40 PRs and we merged 19. So that's

[4:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=278s) had 40 PRs and we merged 19. So that's

[4:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=278s) had 40 PRs and we merged 19. So that's roughly a 50% merge rate. But the goal

[4:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=279s) roughly a 50% merge rate. But the goal

[4:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=279s) roughly a 50% merge rate. But the goal here wasn't to merge a lot of PRs. It

[4:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=281s) here wasn't to merge a lot of PRs. It

[4:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=281s) here wasn't to merge a lot of PRs. It was to learn how the agents fail and to

[4:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=282s) was to learn how the agents fail and to

[4:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=282s) was to learn how the agents fail and to fix those paper cuts before

[4:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=283s) fix those paper cuts before

[4:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=283s) fix those paper cuts before productionizing anything. Here are some

[4:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=285s) productionizing anything. Here are some

[4:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=285s) productionizing anything. Here are some of the failure modes that we saw. Since

[4:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=287s) of the failure modes that we saw. Since

[4:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=287s) of the failure modes that we saw. Since we're a large legacy code base with both

[4:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=288s) we're a large legacy code base with both

[4:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=288s) we're a large legacy code base with both XC test and swift testing, the agents

[4:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=290s) XC test and swift testing, the agents

[4:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=290s) XC test and swift testing, the agents would sometimes get confused and mix the

[4:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=291s) would sometimes get confused and mix the

[4:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=291s) would sometimes get confused and mix the two in one file. We fixed that by giving

[4:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=293s) two in one file. We fixed that by giving

[4:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=293s) two in one file. We fixed that by giving the agents clear rules on when to use

[4:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=294s) the agents clear rules on when to use

[4:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=294s) the agents clear rules on when to use each. We also had a giant user client

[4:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=296s) each. We also had a giant user client

[4:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=296s) each. We also had a giant user client singleton class. Some code passed around

[4:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=298s) singleton class. Some code passed around

[4:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=298s) singleton class. Some code passed around the concrete version of this class which

[4:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=299s) the concrete version of this class which

[4:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=299s) the concrete version of this class which is really hard to mock whereas others

[5:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=300s) is really hard to mock whereas others

[5:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=300s) is really hard to mock whereas others passed around a protocolized version. So

[5:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=302s) passed around a protocolized version. So

[5:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=302s) passed around a protocolized version. So we migrated the entire codebase over to

[5:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=303s) we migrated the entire codebase over to

[5:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=303s) we migrated the entire codebase over to the protocolized version and added lint

[5:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=305s) the protocolized version and added lint

[5:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=305s) the protocolized version and added lint rule to enforce it. This made the whole

[5:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=307s) rule to enforce it. This made the whole

[5:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=307s) rule to enforce it. This made the whole host of files easier to test for agents

[5:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=309s) host of files easier to test for agents

[5:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=309s) host of files easier to test for agents and for humans. We also ran into six

[5:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=312s) and for humans. We also ran into six

[5:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=312s) and for humans. We also ran into six concurrency related issues and agents

[5:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=313s) concurrency related issues and agents

[5:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=313s) concurrency related issues and agents writing timing based tests which are a

[5:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=315s) writing timing based tests which are a

[5:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=315s) writing timing based tests which are a big source of flakiness in CI. So we

[5:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=316s) big source of flakiness in CI. So we

[5:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=316s) big source of flakiness in CI. So we improved the state for both of these

[5:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=317s) improved the state for both of these

[5:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=318s) improved the state for both of these with better rules and examples in our

[5:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=319s) with better rules and examples in our

[5:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=319s) with better rules and examples in our testing skills.

[5:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=321s) testing skills.

[5:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=321s) testing skills. Now this brings us to milestone 2. Since

[5:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=323s) Now this brings us to milestone 2. Since

[5:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=323s) Now this brings us to milestone 2. Since we were so far behind on code coverage,

[5:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=324s) we were so far behind on code coverage,

[5:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=324s) we were so far behind on code coverage, the biggest leverage that we saw was

[5:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=326s) the biggest leverage that we saw was

[5:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=326s) the biggest leverage that we saw was backfilling tests for code that was

[5:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=327s) backfilling tests for code that was

[5:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=327s) backfilling tests for code that was already merged in. That's where the

[5:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=329s) already merged in. That's where the

[5:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=329s) already merged in. That's where the scheduled backfill workflow comes in.

[5:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=330s) scheduled backfill workflow comes in.

[5:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=330s) scheduled backfill workflow comes in. This is a system design diagram of the

[5:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=332s) This is a system design diagram of the

[5:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=332s) This is a system design diagram of the workflow as it runs in production and it

[5:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=334s) workflow as it runs in production and it

[5:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=334s) workflow as it runs in production and it essentially does two jobs. The left half

[5:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=335s) essentially does two jobs. The left half

[5:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=335s) essentially does two jobs. The left half is to identify which is picking the

[5:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=337s) is to identify which is picking the

[5:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=337s) is to identify which is picking the files that need tests and the right half

[5:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=339s) files that need tests and the right half

[5:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=339s) files that need tests and the right half is right which is having the agents

[5:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=341s) is right which is having the agents

[5:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=341s) is right which is having the agents actually write those tests. Let's start

[5:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=343s) actually write those tests. Let's start

[5:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=343s) actually write those tests. Let's start with identify. Every time a PR merges to

[5:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=345s) with identify. Every time a PR merges to

[5:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=345s) with identify. Every time a PR merges to the main branch in the iOS repo, we kick

[5:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=346s) the main branch in the iOS repo, we kick

[5:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=346s) the main branch in the iOS repo, we kick off a GitHub access job that generates

[5:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=348s) off a GitHub access job that generates

[5:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=348s) off a GitHub access job that generates an Xcode coverage report and uploads it

[5:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=350s) an Xcode coverage report and uploads it

[5:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=350s) an Xcode coverage report and uploads it to S3 that becomes a source of truth for

[5:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=352s) to S3 that becomes a source of truth for

[5:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=352s) to S3 that becomes a source of truth for the latest code coverage. The back fill

[5:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=354s) the latest code coverage. The back fill

[5:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=354s) the latest code coverage. The back fill workflow then fetches that report and

[5:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=355s) workflow then fetches that report and

[5:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=355s) workflow then fetches that report and runs it through a selector script which

[5:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=356s) runs it through a selector script which

[5:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=356s) runs it through a selector script which picks the next set of files to test.

[5:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=359s) picks the next set of files to test.

[5:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=359s) picks the next set of files to test. Let's get a deeper a little deeper into

[6:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=360s) Let's get a deeper a little deeper into

[6:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=360s) Let's get a deeper a little deeper into the selector script. Its job is

[6:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=362s) the selector script. Its job is

[6:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=362s) the selector script. Its job is essentially to score every Swift file

[6:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=363s) essentially to score every Swift file

[6:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=363s) essentially to score every Swift file based on how confident we are that an

[6:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=365s) based on how confident we are that an

[6:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=365s) based on how confident we are that an agent can write a good test for it. The

[6:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=367s) agent can write a good test for it. The

[6:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=367s) agent can write a good test for it. The strongest signal is file type. So view

[6:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=369s) strongest signal is file type. So view

[6:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=369s) strongest signal is file type. So view models, repositories, and data sources

[6:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=370s) models, repositories, and data sources

[6:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=370s) models, repositories, and data sources get a 2.5x boost while legacy files get

[6:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=372s) get a 2.5x boost while legacy files get

[6:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=372s) get a 2.5x boost while legacy files get pushed down. We skipped UI code

[6:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=374s) pushed down. We skipped UI code

[6:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=374s) pushed down. We skipped UI code completely for now since snapshot tests

[6:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=375s) completely for now since snapshot tests

[6:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=375s) completely for now since snapshot tests are better fit there. And larger files

[6:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=377s) are better fit there. And larger files

[6:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=377s) are better fit there. And larger files also get a penalty. So we prioritize

[6:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=379s) also get a penalty. So we prioritize

[6:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=379s) also get a penalty. So we prioritize code that we're confident that we can

[6:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=380s) code that we're confident that we can

[6:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=380s) code that we're confident that we can test well.

[6:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=382s) test well.

[6:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=382s) test well. The next step is to actually write the

[6:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=383s) The next step is to actually write the

[6:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=384s) The next step is to actually write the tests. We kick off one child workflow

[6:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=385s) tests. We kick off one child workflow

[6:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=386s) tests. We kick off one child workflow per file, all running in parallel. In

[6:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=388s) per file, all running in parallel. In

[6:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=388s) per file, all running in parallel. In each one of these runs, the agent clones

[6:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=389s) each one of these runs, the agent clones

[6:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=389s) each one of these runs, the agent clones the iOS repository, reads the testing

[6:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=391s) the iOS repository, reads the testing

[6:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=391s) the iOS repository, reads the testing skill that we keep in the repo, which

[6:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=393s) skill that we keep in the repo, which

[6:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=393s) skill that we keep in the repo, which spells out how to write a good test.

[6:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=394s) spells out how to write a good test.

[6:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=394s) spells out how to write a good test. Then it writes the test plus any mocks

[6:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=396s) Then it writes the test plus any mocks

[6:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=396s) Then it writes the test plus any mocks and fakes it needs and opens a PR with a

[6:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=398s) and fakes it needs and opens a PR with a

[6:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=398s) and fakes it needs and opens a PR with a generated test label so we can identify

[6:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=400s) generated test label so we can identify

[6:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=400s) generated test label so we can identify it easily later.

[6:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=402s) it easily later.

[6:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=402s) it easily later. These agents run on Dualingo's cloud

[6:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=403s) These agents run on Dualingo's cloud

[6:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=404s) These agents run on Dualingo's cloud coding agent which is a platform that

[6:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=405s) coding agent which is a platform that

[6:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=405s) coding agent which is a platform that also runs on Neporal. The simplest way

[6:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=406s) also runs on Neporal. The simplest way

[6:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=406s) also runs on Neporal. The simplest way to think about it is as an abstraction

[6:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=408s) to think about it is as an abstraction

[6:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=408s) to think about it is as an abstraction on top of cloud code and codex running

[6:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=409s) on top of cloud code and codex running

[6:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=409s) on top of cloud code and codex running in the cloud. We can invoke it from

[6:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=411s) in the cloud. We can invoke it from

[6:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=411s) in the cloud. We can invoke it from Slack, temporal or GitHub. It talks to a

[6:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=412s) Slack, temporal or GitHub. It talks to a

[6:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=412s) Slack, temporal or GitHub. It talks to a bunch of our internal tools for MCP and

[6:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=414s) bunch of our internal tools for MCP and

[6:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=414s) bunch of our internal tools for MCP and it logs a bunch of events so we can see

[6:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=416s) it logs a bunch of events so we can see

[6:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=416s) it logs a bunch of events so we can see what each agent run is doing and how

[6:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=417s) what each agent run is doing and how

[6:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=417s) what each agent run is doing and how much it costs. We started running these

[7:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=420s) much it costs. We started running these

[7:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=420s) much it costs. We started running these on Linux machines and then later move

[7:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=421s) on Linux machines and then later move

[7:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=422s) on Linux machines and then later move them to MacRunners which will become

[7:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=423s) them to MacRunners which will become

[7:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=423s) them to MacRunners which will become important in a minute. Here's what a

[7:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=425s) important in a minute. Here's what a

[7:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=425s) important in a minute. Here's what a generated test PR looks like. Every PR

[7:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=427s) generated test PR looks like. Every PR

[7:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=427s) generated test PR looks like. Every PR description follows the same template

[7:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=429s) description follows the same template

[7:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=429s) description follows the same template and it says explicitly that the PR is AI

[7:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=430s) and it says explicitly that the PR is AI

[7:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=430s) and it says explicitly that the PR is AI generated. The reviewer has four

[7:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=432s) generated. The reviewer has four

[7:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=432s) generated. The reviewer has four options. They can approve it and the PR

[7:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=434s) options. They can approve it and the PR

[7:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=434s) options. They can approve it and the PR automarges. They can push changes to it

[7:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=436s) automarges. They can push changes to it

[7:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=436s) automarges. They can push changes to it directly. They can reassign it to

[7:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=437s) directly. They can reassign it to

[7:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=437s) directly. They can reassign it to someone better suited or they can close

[7:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=438s) someone better suited or they can close

[7:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=438s) someone better suited or they can close it out with feedback on why it isn't

[7:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=439s) it out with feedback on why it isn't

[7:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=440s) it out with feedback on why it isn't mergeable.

[7:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=441s) mergeable.

[7:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=441s) mergeable. Now once we had the backfl running in

[7:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=443s) Now once we had the backfl running in

[7:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=443s) Now once we had the backfl running in production we had a slightly new

[7:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=444s) production we had a slightly new

[7:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=444s) production we had a slightly new problem. We suddenly had tens of tens to

[7:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=446s) problem. We suddenly had tens of tens to

[7:26](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=446s) problem. We suddenly had tens of tens to hundreds of open PRs that nobody could

[7:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=447s) hundreds of open PRs that nobody could

[7:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=447s) hundreds of open PRs that nobody could manage by hand. So how do we get all of

[7:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=449s) manage by hand. So how do we get all of

[7:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=449s) manage by hand. So how do we get all of these to a mergeable state and in front

[7:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=450s) these to a mergeable state and in front

[7:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=450s) these to a mergeable state and in front of the right reviewer

[7:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=452s) of the right reviewer

[7:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=452s) of the right reviewer that's where the PR life cycle workflow

[7:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=454s) that's where the PR life cycle workflow

[7:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=454s) that's where the PR life cycle workflow comes in. You can think of it as a state

[7:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=456s) comes in. You can think of it as a state

[7:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=456s) comes in. You can think of it as a state machine. It runs every couple of hours

[7:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=457s) machine. It runs every couple of hours

[7:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=457s) machine. It runs every couple of hours looks through every open generated test

[7:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=459s) looks through every open generated test

[7:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=459s) looks through every open generated test PR and takes exactly one action on each.

[7:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=462s) PR and takes exactly one action on each.

[7:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=462s) PR and takes exactly one action on each. It covers the next two steps of our

[7:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=463s) It covers the next two steps of our

[7:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=463s) It covers the next two steps of our loop. Verify which is does this PR pass

[7:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=465s) loop. Verify which is does this PR pass

[7:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=466s) loop. Verify which is does this PR pass CI and is it high quality and resolve

[7:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=467s) CI and is it high quality and resolve

[7:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=467s) CI and is it high quality and resolve which is getting a human to review it

[7:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=469s) which is getting a human to review it

[7:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=469s) which is getting a human to review it and monitor it. The logic for this

[7:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=471s) and monitor it. The logic for this

[7:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=471s) and monitor it. The logic for this workflow is a simple decision tree. If

[7:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=472s) workflow is a simple decision tree. If

[7:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=472s) workflow is a simple decision tree. If there are any merge conflicts or the PR

[7:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=474s) there are any merge conflicts or the PR

[7:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=474s) there are any merge conflicts or the PR has been open for about 3 weeks, we

[7:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=475s) has been open for about 3 weeks, we

[7:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=475s) has been open for about 3 weeks, we close it out and mark it as still. If CI

[7:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=477s) close it out and mark it as still. If CI

[7:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=477s) close it out and mark it as still. If CI is still running or a reviewer is

[7:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=478s) is still running or a reviewer is

[7:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=478s) is still running or a reviewer is already on it, we skip it. If CI is

[8:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=480s) already on it, we skip it. If CI is

[8:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=480s) already on it, we skip it. If CI is green and there's no reviewers, we find

[8:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=481s) green and there's no reviewers, we find

[8:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=481s) green and there's no reviewers, we find one assign them. If CI is red, we try to

[8:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=483s) one assign them. If CI is red, we try to

[8:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=483s) one assign them. If CI is red, we try to fix it. And if a reviewer rejects the

[8:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=485s) fix it. And if a reviewer rejects the

[8:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=485s) fix it. And if a reviewer rejects the PR, the file goes into 30-day cool down.

[8:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=487s) PR, the file goes into 30-day cool down.

[8:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=487s) PR, the file goes into 30-day cool down. So, we're not spamming engineers with

[8:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=488s) So, we're not spamming engineers with

[8:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=488s) So, we're not spamming engineers with the same PR again and again.

[8:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=490s) the same PR again and again.

[8:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=490s) the same PR again and again. When CI fails, the first thing we check

[8:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=492s) When CI fails, the first thing we check

[8:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=492s) When CI fails, the first thing we check is which job failed. If it's pre-commit,

[8:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=494s) is which job failed. If it's pre-commit,

[8:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=494s) is which job failed. If it's pre-commit, a GitHub action applies the fix. It's

[8:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=495s) a GitHub action applies the fix. It's

[8:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=496s) a GitHub action applies the fix. It's cheap, it's deterministic, and there's

[8:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=497s) cheap, it's deterministic, and there's

[8:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=497s) cheap, it's deterministic, and there's no agent needed. If it's a real test

[8:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=498s) no agent needed. If it's a real test

[8:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=498s) no agent needed. If it's a real test failure, we kick off another agent that

[8:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=500s) failure, we kick off another agent that

[8:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=500s) failure, we kick off another agent that reads the CI logs and pushes a fix to

[8:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=501s) reads the CI logs and pushes a fix to

[8:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=501s) reads the CI logs and pushes a fix to the PR. We cap that whole loop at five

[8:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=503s) the PR. We cap that whole loop at five

[8:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=503s) the PR. We cap that whole loop at five attemps. Past that we close the PR

[8:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=505s) attemps. Past that we close the PR

[8:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=505s) attemps. Past that we close the PR rather than trying to fix it forever.

[8:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=507s) rather than trying to fix it forever.

[8:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=507s) rather than trying to fix it forever. Once CI is green, we need to find the

[8:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=509s) Once CI is green, we need to find the

[8:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=509s) Once CI is green, we need to find the right reviewer. What we do is we run git

[8:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=511s) right reviewer. What we do is we run git

[8:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=511s) right reviewer. What we do is we run git blame on the source file being tested.

[8:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=512s) blame on the source file being tested.

[8:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=512s) blame on the source file being tested. Find the top contributor and assign it

[8:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=514s) Find the top contributor and assign it

[8:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=514s) Find the top contributor and assign it to them and turn on auto merge. If they

[8:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=515s) to them and turn on auto merge. If they

[8:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=515s) to them and turn on auto merge. If they approve it, the PR gets merged. If they

[8:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=517s) approve it, the PR gets merged. If they

[8:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=517s) approve it, the PR gets merged. If they rejected, the file goes into the 30-day

[8:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=518s) rejected, the file goes into the 30-day

[8:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=518s) rejected, the file goes into the 30-day cool down, which I previously mentioned.

[8:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=520s) cool down, which I previously mentioned.

[8:40](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=520s) cool down, which I previously mentioned. This brings us to milestone four. Once

[8:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=522s) This brings us to milestone four. Once

[8:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=522s) This brings us to milestone four. Once all of this was back filling tests for

[8:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=524s) all of this was back filling tests for

[8:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=524s) all of this was back filling tests for existing code, we wanted to make it just

[8:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=525s) existing code, we wanted to make it just

[8:45](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=525s) existing code, we wanted to make it just as easy to get tests for the new code

[8:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=527s) as easy to get tests for the new code

[8:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=527s) as easy to get tests for the new code engineers were writing. Now any engineer

[8:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=529s) engineers were writing. Now any engineer

[8:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=529s) engineers were writing. Now any engineer can add or write test label to their own

[8:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=530s) can add or write test label to their own

[8:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=530s) can add or write test label to their own PR, it hooks into the same temporal

[8:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=532s) PR, it hooks into the same temporal

[8:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=532s) PR, it hooks into the same temporal infrastructure we just walked through.

[8:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=534s) infrastructure we just walked through.

[8:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=534s) infrastructure we just walked through. The only real difference is selection.

[8:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=535s) The only real difference is selection.

[8:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=535s) The only real difference is selection. Instead of the selection script, we take

[8:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=537s) Instead of the selection script, we take

[8:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=537s) Instead of the selection script, we take the file straight from the engineer's

[8:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=538s) the file straight from the engineer's

[8:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=538s) the file straight from the engineer's PR. The agent writes the test and you

[9:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=540s) PR. The agent writes the test and you

[9:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=540s) PR. The agent writes the test and you get a draft SPR link back to your

[9:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=541s) get a draft SPR link back to your

[9:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=541s) get a draft SPR link back to your original one. Now let's talk about some

[9:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=543s) original one. Now let's talk about some

[9:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=543s) original one. Now let's talk about some of the challenges that we ran into. Our

[9:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=544s) of the challenges that we ran into. Our

[9:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=544s) of the challenges that we ran into. Our agents originally ran on Linux machines

[9:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=546s) agents originally ran on Linux machines

[9:06](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=546s) agents originally ran on Linux machines which don't have X code or Swift

[9:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=547s) which don't have X code or Swift

[9:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=547s) which don't have X code or Swift installed. So when an agent would open a

[9:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=549s) installed. So when an agent would open a

[9:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=549s) installed. So when an agent would open a PR after about 20 or 30 minutes CI would

[9:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=551s) PR after about 20 or 30 minutes CI would

[9:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=551s) PR after about 20 or 30 minutes CI would fail frequently on lint. This wastes the

[9:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=553s) fail frequently on lint. This wastes the

[9:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=553s) fail frequently on lint. This wastes the agents time and CI time as well which is

[9:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=555s) agents time and CI time as well which is

[9:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=555s) agents time and CI time as well which is an increasingly scarce resource. For a

[9:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=557s) an increasingly scarce resource. For a

[9:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=557s) an increasingly scarce resource. For a while 30 13.6% of our PRs were failing

[9:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=559s) while 30 13.6% of our PRs were failing

[9:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=559s) while 30 13.6% of our PRs were failing due to lint and then when we dug into it

[9:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=561s) due to lint and then when we dug into it

[9:21](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=561s) due to lint and then when we dug into it 62% of those failures came from just to

[9:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=562s) 62% of those failures came from just to

[9:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=562s) 62% of those failures came from just to lint rules. We fixed this in layers. The

[9:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=565s) lint rules. We fixed this in layers. The

[9:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=565s) lint rules. We fixed this in layers. The first was better examples in the prompt

[9:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=567s) first was better examples in the prompt

[9:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=567s) first was better examples in the prompt along with pitfalls to avoid. The second

[9:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=568s) along with pitfalls to avoid. The second

[9:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=568s) along with pitfalls to avoid. The second was a GitHub action that auto fixes lint

[9:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=570s) was a GitHub action that auto fixes lint

[9:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=570s) was a GitHub action that auto fixes lint errors in CI. The third was an audit of

[9:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=572s) errors in CI. The third was an audit of

[9:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=572s) errors in CI. The third was an audit of the rules that couldn't be autofixed. We

[9:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=574s) the rules that couldn't be autofixed. We

[9:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=574s) the rules that couldn't be autofixed. We essentially removed the ones that

[9:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=575s) essentially removed the ones that

[9:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=575s) essentially removed the ones that weren't high signal and this made life

[9:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=576s) weren't high signal and this made life

[9:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=576s) weren't high signal and this made life easier for both agents and human

[9:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=578s) easier for both agents and human

[9:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=578s) easier for both agents and human engineers. And finally, the real fix

[9:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=579s) engineers. And finally, the real fix

[9:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=579s) engineers. And finally, the real fix came when we moved the agents to run on

[9:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=581s) came when we moved the agents to run on

[9:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=581s) came when we moved the agents to run on Mac runners. So now the agents could

[9:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=582s) Mac runners. So now the agents could

[9:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=582s) Mac runners. So now the agents could build and lend the code locally before

[9:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=584s) build and lend the code locally before

[9:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=584s) build and lend the code locally before ever opening a PR. The second challenge

[9:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=587s) ever opening a PR. The second challenge

[9:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=587s) ever opening a PR. The second challenge was PR quality. When we assign a PR to a

[9:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=589s) was PR quality. When we assign a PR to a

[9:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=589s) was PR quality. When we assign a PR to a human, we wanted to be something they'd

[9:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=590s) human, we wanted to be something they'd

[9:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=590s) human, we wanted to be something they'd actually feel comfortable proving. But

[9:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=592s) actually feel comfortable proving. But

[9:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=592s) actually feel comfortable proving. But agents sometimes take the shortest paths

[9:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=593s) agents sometimes take the shortest paths

[9:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=593s) agents sometimes take the shortest paths to green CI. We saw agents editing

[9:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=595s) to green CI. We saw agents editing

[9:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=595s) to green CI. We saw agents editing production code to make a failing test

[9:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=596s) production code to make a failing test

[9:56](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=596s) production code to make a failing test pass. We also saw them modifying,

[9:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=598s) pass. We also saw them modifying,

[9:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=598s) pass. We also saw them modifying, deleting, or silencing existing tests.

[9:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=599s) deleting, or silencing existing tests.

[9:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=599s) deleting, or silencing existing tests. And then we saw them adding long weight

[10:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=601s) And then we saw them adding long weight

[10:01](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=601s) And then we saw them adding long weight to tests which can make CI flaky.

[10:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=603s) to tests which can make CI flaky.

[10:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=603s) to tests which can make CI flaky. Again, there was no single fix here, so

[10:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=605s) Again, there was no single fix here, so

[10:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=605s) Again, there was no single fix here, so we built three layers. The first is the

[10:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=607s) we built three layers. The first is the

[10:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=607s) we built three layers. The first is the deterministic rule. If a PR touches any

[10:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=609s) deterministic rule. If a PR touches any

[10:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=609s) deterministic rule. If a PR touches any file that isn't a test, a mock, a fake,

[10:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=611s) file that isn't a test, a mock, a fake,

[10:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=611s) file that isn't a test, a mock, a fake, or a stub, we close it out. The second

[10:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=613s) or a stub, we close it out. The second

[10:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=613s) or a stub, we close it out. The second builds on our AI code review. At

[10:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=614s) builds on our AI code review. At

[10:14](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=614s) builds on our AI code review. At Dualingo, we use Claude and Codex to

[10:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=616s) Dualingo, we use Claude and Codex to

[10:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=616s) Dualingo, we use Claude and Codex to review every PR. And those B bots often

[10:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=618s) review every PR. And those B bots often

[10:18](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=618s) review every PR. And those B bots often leave comments. So, we built an agent

[10:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=620s) leave comments. So, we built an agent

[10:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=620s) leave comments. So, we built an agent that resolves any P 0 comments before a

[10:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=622s) that resolves any P 0 comments before a

[10:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=622s) that resolves any P 0 comments before a human ever looks at the PR. And the

[10:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=624s) human ever looks at the PR. And the

[10:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=624s) human ever looks at the PR. And the third is an LLM as a judge. It's a

[10:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=625s) third is an LLM as a judge. It's a

[10:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=625s) third is an LLM as a judge. It's a reviewer agent built just for these test

[10:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=627s) reviewer agent built just for these test

[10:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=627s) reviewer agent built just for these test PRs. It reads the PR diff and rejects

[10:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=629s) PRs. It reads the PR diff and rejects

[10:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=629s) PRs. It reads the PR diff and rejects production edits, changed or deleted

[10:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=630s) production edits, changed or deleted

[10:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=630s) production edits, changed or deleted tests, disabled tests, and long weights.

[10:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=633s) tests, disabled tests, and long weights.

[10:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=633s) tests, disabled tests, and long weights. And as you learn more, we keep adding

[10:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=634s) And as you learn more, we keep adding

[10:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=634s) And as you learn more, we keep adding additional checks to it.

[10:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=636s) additional checks to it.

[10:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=636s) additional checks to it. So far, the LM as a judge has reviewed

[10:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=638s) So far, the LM as a judge has reviewed

[10:38](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=638s) So far, the LM as a judge has reviewed about,200 PRs and rejected 73 of them.

[10:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=641s) about,200 PRs and rejected 73 of them.

[10:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=641s) about,200 PRs and rejected 73 of them. Every rejection comes with a comment

[10:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=642s) Every rejection comes with a comment

[10:42](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=642s) Every rejection comes with a comment like this one, saying exactly why the PR

[10:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=644s) like this one, saying exactly why the PR

[10:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=644s) like this one, saying exactly why the PR was rejected. We then read that feedback

[10:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=646s) was rejected. We then read that feedback

[10:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=646s) was rejected. We then read that feedback and fold it back into our prompts and

[10:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=647s) and fold it back into our prompts and

[10:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=647s) and fold it back into our prompts and skills, so the same mistakes get less

[10:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=648s) skills, so the same mistakes get less

[10:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=648s) skills, so the same mistakes get less likely each time. So, let's talk about

[10:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=651s) likely each time. So, let's talk about

[10:51](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=651s) likely each time. So, let's talk about what all of this added up to. Since we

[10:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=653s) what all of this added up to. Since we

[10:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=653s) what all of this added up to. Since we started, the pipeline has merged more

[10:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=654s) started, the pipeline has merged more

[10:54](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=654s) started, the pipeline has merged more than 700 test PRs, adding about 225,000

[10:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=657s) than 700 test PRs, adding about 225,000

[10:57](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=657s) than 700 test PRs, adding about 225,000 lines of test code. Every one of those

[10:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=659s) lines of test code. Every one of those

[10:59](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=659s) lines of test code. Every one of those PRs was reviewed and approved by a

[11:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=660s) PRs was reviewed and approved by a

[11:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=660s) PRs was reviewed and approved by a human. MVVM test coverage went from 9%

[11:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=663s) human. MVVM test coverage went from 9%

[11:03](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=663s) human. MVVM test coverage went from 9% to 59% and overall coverage went from

[11:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=664s) to 59% and overall coverage went from

[11:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=664s) to 59% and overall coverage went from 11% to 24%. Most importantly, we now

[11:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=667s) 11% to 24%. Most importantly, we now

[11:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=667s) 11% to 24%. Most importantly, we now have a code base so engineers and agents

[11:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=668s) have a code base so engineers and agents

[11:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=668s) have a code base so engineers and agents can ship faster and with more

[11:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=669s) can ship faster and with more

[11:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=670s) can ship faster and with more confidence. Here's a few more

[11:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=672s) confidence. Here's a few more

[11:12](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=672s) confidence. Here's a few more interesting stats. Since March, the

[11:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=673s) interesting stats. Since March, the

[11:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=673s) interesting stats. Since March, the pipeline has written 34% of all new test

[11:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=675s) pipeline has written 34% of all new test

[11:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=675s) pipeline has written 34% of all new test files in the repo, 44% of all new test

[11:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=677s) files in the repo, 44% of all new test

[11:17](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=677s) files in the repo, 44% of all new test cases, and 10% of every swift line added

[11:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=679s) cases, and 10% of every swift line added

[11:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=679s) cases, and 10% of every swift line added to the codebase. And interestingly, that

[11:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=680s) to the codebase. And interestingly, that

[11:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=680s) to the codebase. And interestingly, that share has actually dripped in the last

[11:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=682s) share has actually dripped in the last

[11:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=682s) share has actually dripped in the last few weeks because engineers are now

[11:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=683s) few weeks because engineers are now

[11:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=683s) few weeks because engineers are now writing a lot more tests themselves,

[11:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=684s) writing a lot more tests themselves,

[11:24](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=684s) writing a lot more tests themselves, which is honestly the end goal.

[11:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=687s) which is honestly the end goal.

[11:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=687s) which is honestly the end goal. Um, and here's the coverage curve since

[11:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=689s) Um, and here's the coverage curve since

[11:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=689s) Um, and here's the coverage curve since April when the backfield started running

[11:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=690s) April when the backfield started running

[11:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=690s) April when the backfield started running on a schedule. You can see it bent

[11:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=692s) on a schedule. You can see it bent

[11:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=692s) on a schedule. You can see it bent slightly upward around June and July,

[11:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=693s) slightly upward around June and July,

[11:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=693s) slightly upward around June and July, which is when we really ramped up the

[11:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=695s) which is when we really ramped up the

[11:35](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=695s) which is when we really ramped up the pipeline.

[11:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=696s) pipeline.

[11:36](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=696s) pipeline. So, what have we learned while doing

[11:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=697s) So, what have we learned while doing

[11:37](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=697s) So, what have we learned while doing this project? The first thing is that

[11:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=699s) this project? The first thing is that

[11:39](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=699s) this project? The first thing is that this pattern generalizes the loop we

[11:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=701s) this pattern generalizes the loop we

[11:41](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=701s) this pattern generalizes the loop we talked about identify, write, verify,

[11:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=703s) talked about identify, write, verify,

[11:43](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=703s) talked about identify, write, verify, resolve, iterate. It isn't specific to

[11:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=704s) resolve, iterate. It isn't specific to

[11:44](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=704s) resolve, iterate. It isn't specific to testing. Unit tests were just the lowest

[11:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=706s) testing. Unit tests were just the lowest

[11:46](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=706s) testing. Unit tests were just the lowest hanging fruit and any job that fits the

[11:47](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=707s) hanging fruit and any job that fits the

[11:48](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=708s) hanging fruit and any job that fits the shape can run on the same infrastructure

[11:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=709s) shape can run on the same infrastructure

[11:49](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=709s) shape can run on the same infrastructure and that's exactly what we're starting

[11:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=710s) and that's exactly what we're starting

[11:50](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=710s) and that's exactly what we're starting to do at the company. We're applying the

[11:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=712s) to do at the company. We're applying the

[11:52](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=712s) to do at the company. We're applying the same loop to modernization to

[11:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=713s) same loop to modernization to

[11:53](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=713s) same loop to modernization to modernizing screens to MVM and Swift UI,

[11:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=715s) modernizing screens to MVM and Swift UI,

[11:55](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=715s) modernizing screens to MVM and Swift UI, migrating code uh on Android to Corines

[11:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=718s) migrating code uh on Android to Corines

[11:58](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=718s) migrating code uh on Android to Corines and generating snapshot tests.

[12:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=720s) and generating snapshot tests.

[12:00](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=720s) and generating snapshot tests. So before I wrap up, I want to leave you

[12:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=722s) So before I wrap up, I want to leave you

[12:02](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=722s) So before I wrap up, I want to leave you with a thought experiment. There's a new

[12:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=724s) with a thought experiment. There's a new

[12:04](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=724s) with a thought experiment. There's a new class of fast general purpose

[12:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=725s) class of fast general purpose

[12:05](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=725s) class of fast general purpose classifiers like Jev. They taken some

[12:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=727s) classifiers like Jev. They taken some

[12:07](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=727s) classifiers like Jev. They taken some state and output an action in

[12:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=728s) state and output an action in

[12:08](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=728s) state and output an action in milliseconds. So what if we replace a

[12:09](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=729s) milliseconds. So what if we replace a

[12:10](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=730s) milliseconds. So what if we replace a handwritten decision tree in the PR life

[12:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=731s) handwritten decision tree in the PR life

[12:11](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=731s) handwritten decision tree in the PR life cycle workflow with one of these? PR

[12:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=733s) cycle workflow with one of these? PR

[12:13](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=733s) cycle workflow with one of these? PR state goes in, the classifier picks the

[12:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=735s) state goes in, the classifier picks the

[12:15](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=735s) state goes in, the classifier picks the next action and we execute it all in

[12:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=736s) next action and we execute it all in

[12:16](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=736s) next action and we execute it all in many seconds.

[12:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=739s) many seconds.

[12:19](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=739s) many seconds. I also want to thank a few folks who

[12:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=740s) I also want to thank a few folks who

[12:20](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=740s) I also want to thank a few folks who worked on this with me, Jesse, Gordalup,

[12:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=742s) worked on this with me, Jesse, Gordalup,

[12:22](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=742s) worked on this with me, Jesse, Gordalup, Hunter and Jiho. They built a lot of the

[12:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=743s) Hunter and Jiho. They built a lot of the

[12:23](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=743s) Hunter and Jiho. They built a lot of the platform pieces this project stands on

[12:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=745s) platform pieces this project stands on

[12:25](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=745s) platform pieces this project stands on and helped review a lot of this code.

[12:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=747s) and helped review a lot of this code.

[12:27](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=747s) and helped review a lot of this code. That's it for me. Thank you so much for

[12:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=748s) That's it for me. Thank you so much for

[12:28](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=748s) That's it for me. Thank you so much for making it this far. If you want to go

[12:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=749s) making it this far. If you want to go

[12:29](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=749s) making it this far. If you want to go deeper, we've written about this

[12:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=750s) deeper, we've written about this

[12:30](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=750s) deeper, we've written about this pipeline on the dualing engineering

[12:31](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=751s) pipeline on the dualing engineering

[12:32](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=752s) pipeline on the dualing engineering blog. And I'd also love to hear from you

[12:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=753s) blog. And I'd also love to hear from you

[12:33](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=753s) blog. And I'd also love to hear from you if you're building something similar.

[12:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=754s) if you're building something similar.

[12:34](https://www.youtube.com/watch?v=D_4Rn6uLdQM&t=754s) if you're building something similar. You can find me on LinkedIn or next