[
  {
    "startMs": 1199,
    "text": "Hey everyone, thanks for tuning in."
  },
  {
    "startMs": 3030,
    "text": "Hey everyone, thanks for tuning in."
  },
  {
    "startMs": 3040,
    "text": "Hey everyone, thanks for tuning in. Today I'm going to walk you through how"
  },
  {
    "startMs": 4150,
    "text": "Today I'm going to walk you through how"
  },
  {
    "startMs": 4160,
    "text": "Today I'm going to walk you through how we use AI agents at Dualingo to write"
  },
  {
    "startMs": 5990,
    "text": "we use AI agents at Dualingo to write"
  },
  {
    "startMs": 6000,
    "text": "we use AI agents at Dualingo to write and ship more than 200,000 thousand"
  },
  {
    "startMs": 7749,
    "text": "and ship more than 200,000 thousand"
  },
  {
    "startMs": 7759,
    "text": "and ship more than 200,000 thousand lines of unit tests on iOS codebase."
  },
  {
    "startMs": 9589,
    "text": "lines of unit tests on iOS codebase."
  },
  {
    "startMs": 9599,
    "text": "lines of unit tests on iOS codebase. I'll share what we built and what we"
  },
  {
    "startMs": 11030,
    "text": "I'll share what we built and what we"
  },
  {
    "startMs": 11040,
    "text": "I'll share what we built and what we learned along the way. Uh I'm Kush. I'm"
  },
  {
    "startMs": 13110,
    "text": "learned along the way. Uh I'm Kush. I'm"
  },
  {
    "startMs": 13120,
    "text": "learned along the way. Uh I'm Kush. I'm a staff engineer on the platform team"
  },
  {
    "startMs": 14390,
    "text": "a staff engineer on the platform team"
  },
  {
    "startMs": 14400,
    "text": "a staff engineer on the platform team here at Dualingo. My job is to build"
  },
  {
    "startMs": 15589,
    "text": "here at Dualingo. My job is to build"
  },
  {
    "startMs": 15599,
    "text": "here at Dualingo. My job is to build developer tooling and AI assisted"
  },
  {
    "startMs": 16870,
    "text": "developer tooling and AI assisted"
  },
  {
    "startMs": 16880,
    "text": "developer tooling and AI assisted workflows for the 100 plus engineers who"
  },
  {
    "startMs": 18310,
    "text": "workflows for the 100 plus engineers who"
  },
  {
    "startMs": 18320,
    "text": "workflows for the 100 plus engineers who contribute to our client apps. Before"
  },
  {
    "startMs": 19830,
    "text": "contribute to our client apps. Before"
  },
  {
    "startMs": 19840,
    "text": "contribute to our client apps. Before Duelingo, I spent close to a decade at"
  },
  {
    "startMs": 21349,
    "text": "Duelingo, I spent close to a decade at"
  },
  {
    "startMs": 21359,
    "text": "Duelingo, I spent close to a decade at Stripe and Uber working across a few"
  },
  {
    "startMs": 22710,
    "text": "Stripe and Uber working across a few"
  },
  {
    "startMs": 22720,
    "text": "Stripe and Uber working across a few teams there. Let's start with some"
  },
  {
    "startMs": 24870,
    "text": "teams there. Let's start with some"
  },
  {
    "startMs": 24880,
    "text": "teams there. Let's start with some background on the scale of mobile at"
  },
  {
    "startMs": 26630,
    "text": "background on the scale of mobile at"
  },
  {
    "startMs": 26640,
    "text": "background on the scale of mobile at Dualingo. Uh we ship our apps to the app"
  },
  {
    "startMs": 29029,
    "text": "Dualingo. Uh we ship our apps to the app"
  },
  {
    "startMs": 29039,
    "text": "Dualingo. Uh we ship our apps to the app store every week. Um we're also a very"
  },
  {
    "startMs": 31429,
    "text": "store every week. Um we're also a very"
  },
  {
    "startMs": 31439,
    "text": "store every week. Um we're also a very experiment-driven company. So at any"
  },
  {
    "startMs": 32870,
    "text": "experiment-driven company. So at any"
  },
  {
    "startMs": 32880,
    "text": "experiment-driven company. So at any given point of time, we have more than"
  },
  {
    "startMs": 34069,
    "text": "given point of time, we have more than"
  },
  {
    "startMs": 34079,
    "text": "given point of time, we have more than 600 AB tests running live and our code"
  },
  {
    "startMs": 36709,
    "text": "600 AB tests running live and our code"
  },
  {
    "startMs": 36719,
    "text": "600 AB tests running live and our code bases are pretty mature. Both the iOS"
  },
  {
    "startMs": 38389,
    "text": "bases are pretty mature. Both the iOS"
  },
  {
    "startMs": 38399,
    "text": "bases are pretty mature. Both the iOS and Android apps have been around for"
  },
  {
    "startMs": 39510,
    "text": "and Android apps have been around for"
  },
  {
    "startMs": 39520,
    "text": "and Android apps have been around for more than 10 years at this point."
  },
  {
    "startMs": 42069,
    "text": "more than 10 years at this point."
  },
  {
    "startMs": 42079,
    "text": "more than 10 years at this point. Here's also the number of executable"
  },
  {
    "startMs": 43670,
    "text": "Here's also the number of executable"
  },
  {
    "startMs": 43680,
    "text": "Here's also the number of executable lines of app code in our iOS repository."
  },
  {
    "startMs": 45670,
    "text": "lines of app code in our iOS repository."
  },
  {
    "startMs": 45680,
    "text": "lines of app code in our iOS repository. So as you can see back in 2019, we were"
  },
  {
    "startMs": 47430,
    "text": "So as you can see back in 2019, we were"
  },
  {
    "startMs": 47440,
    "text": "So as you can see back in 2019, we were at about 200,000 lines and today we're"
  },
  {
    "startMs": 49270,
    "text": "at about 200,000 lines and today we're"
  },
  {
    "startMs": 49280,
    "text": "at about 200,000 lines and today we're hit close to 2 million. And as AI"
  },
  {
    "startMs": 51510,
    "text": "hit close to 2 million. And as AI"
  },
  {
    "startMs": 51520,
    "text": "hit close to 2 million. And as AI assisted coding picks up, you can see"
  },
  {
    "startMs": 52790,
    "text": "assisted coding picks up, you can see"
  },
  {
    "startMs": 52800,
    "text": "assisted coding picks up, you can see that curve starting to bend upward."
  },
  {
    "startMs": 55189,
    "text": "that curve starting to bend upward."
  },
  {
    "startMs": 55199,
    "text": "that curve starting to bend upward. And it's not just the amount of code"
  },
  {
    "startMs": 56310,
    "text": "And it's not just the amount of code"
  },
  {
    "startMs": 56320,
    "text": "And it's not just the amount of code that's changing, it's also who's writing"
  },
  {
    "startMs": 57910,
    "text": "that's changing, it's also who's writing"
  },
  {
    "startMs": 57920,
    "text": "that's changing, it's also who's writing it. So last year about 200 people"
  },
  {
    "startMs": 60069,
    "text": "it. So last year about 200 people"
  },
  {
    "startMs": 60079,
    "text": "it. So last year about 200 people contributed to the iOS repository. And"
  },
  {
    "startMs": 61510,
    "text": "contributed to the iOS repository. And"
  },
  {
    "startMs": 61520,
    "text": "contributed to the iOS repository. And this year we've already crossed 250. And"
  },
  {
    "startMs": 63510,
    "text": "this year we've already crossed 250. And"
  },
  {
    "startMs": 63520,
    "text": "this year we've already crossed 250. And a lot of these folks aren't even iOS"
  },
  {
    "startMs": 64950,
    "text": "a lot of these folks aren't even iOS"
  },
  {
    "startMs": 64960,
    "text": "a lot of these folks aren't even iOS engineers. We have backend engineers,"
  },
  {
    "startMs": 66390,
    "text": "engineers. We have backend engineers,"
  },
  {
    "startMs": 66400,
    "text": "engineers. We have backend engineers, Android engineers, even PMs and"
  },
  {
    "startMs": 67910,
    "text": "Android engineers, even PMs and"
  },
  {
    "startMs": 67920,
    "text": "Android engineers, even PMs and designers all shipping code to the iOS"
  },
  {
    "startMs": 69350,
    "text": "designers all shipping code to the iOS"
  },
  {
    "startMs": 69360,
    "text": "designers all shipping code to the iOS app. And that's mostly been unlocked by"
  },
  {
    "startMs": 71590,
    "text": "app. And that's mostly been unlocked by"
  },
  {
    "startMs": 71600,
    "text": "app. And that's mostly been unlocked by AI coding tools like cursor and cloud"
  },
  {
    "startMs": 73030,
    "text": "AI coding tools like cursor and cloud"
  },
  {
    "startMs": 73040,
    "text": "AI coding tools like cursor and cloud code. Um, another thing to note about"
  },
  {
    "startMs": 75190,
    "text": "code. Um, another thing to note about"
  },
  {
    "startMs": 75200,
    "text": "code. Um, another thing to note about our codebase is that it's a codebase in"
  },
  {
    "startMs": 76630,
    "text": "our codebase is that it's a codebase in"
  },
  {
    "startMs": 76640,
    "text": "our codebase is that it's a codebase in transition. So a few years ago, we"
  },
  {
    "startMs": 78310,
    "text": "transition. So a few years ago, we"
  },
  {
    "startMs": 78320,
    "text": "transition. So a few years ago, we standardized on the MVVM architecture"
  },
  {
    "startMs": 79749,
    "text": "standardized on the MVVM architecture"
  },
  {
    "startMs": 79759,
    "text": "standardized on the MVVM architecture pattern. Today about half the code in"
  },
  {
    "startMs": 81590,
    "text": "pattern. Today about half the code in"
  },
  {
    "startMs": 81600,
    "text": "pattern. Today about half the code in our repository follows that pattern and"
  },
  {
    "startMs": 83030,
    "text": "our repository follows that pattern and"
  },
  {
    "startMs": 83040,
    "text": "our repository follows that pattern and the other half is the legacy code that"
  },
  {
    "startMs": 84390,
    "text": "the other half is the legacy code that"
  },
  {
    "startMs": 84400,
    "text": "the other half is the legacy code that we're in the process of migrating. Since"
  },
  {
    "startMs": 86310,
    "text": "we're in the process of migrating. Since"
  },
  {
    "startMs": 86320,
    "text": "we're in the process of migrating. Since the MVVM half is the easiest to test,"
  },
  {
    "startMs": 87990,
    "text": "the MVVM half is the easiest to test,"
  },
  {
    "startMs": 88000,
    "text": "the MVVM half is the easiest to test, that's where we focused on this project."
  },
  {
    "startMs": 91030,
    "text": "that's where we focused on this project."
  },
  {
    "startMs": 91040,
    "text": "that's where we focused on this project. Here's where we started back in March of"
  },
  {
    "startMs": 92469,
    "text": "Here's where we started back in March of"
  },
  {
    "startMs": 92479,
    "text": "Here's where we started back in March of this year. We were at overall unit test"
  },
  {
    "startMs": 95030,
    "text": "this year. We were at overall unit test"
  },
  {
    "startMs": 95040,
    "text": "this year. We were at overall unit test coverage of 11%. The MVVM layers were a"
  },
  {
    "startMs": 97670,
    "text": "coverage of 11%. The MVVM layers were a"
  },
  {
    "startMs": 97680,
    "text": "coverage of 11%. The MVVM layers were a bit lower, repositories were at 11%,"
  },
  {
    "startMs": 99590,
    "text": "bit lower, repositories were at 11%,"
  },
  {
    "startMs": 99600,
    "text": "bit lower, repositories were at 11%, view models at 9, data sources at 5, and"
  },
  {
    "startMs": 101830,
    "text": "view models at 9, data sources at 5, and"
  },
  {
    "startMs": 101840,
    "text": "view models at 9, data sources at 5, and combined we are about 9% for the MVM"
  },
  {
    "startMs": 103910,
    "text": "combined we are about 9% for the MVM"
  },
  {
    "startMs": 103920,
    "text": "combined we are about 9% for the MVM slice of the app."
  },
  {
    "startMs": 106149,
    "text": "slice of the app."
  },
  {
    "startMs": 106159,
    "text": "slice of the app. So why does this matter now more than it"
  },
  {
    "startMs": 108149,
    "text": "So why does this matter now more than it"
  },
  {
    "startMs": 108159,
    "text": "So why does this matter now more than it used to? As agents and humans write more"
  },
  {
    "startMs": 110630,
    "text": "used to? As agents and humans write more"
  },
  {
    "startMs": 110640,
    "text": "used to? As agents and humans write more code and write it faster, writing code"
  },
  {
    "startMs": 112310,
    "text": "code and write it faster, writing code"
  },
  {
    "startMs": 112320,
    "text": "code and write it faster, writing code isn't the hard part anymore. Verifying"
  },
  {
    "startMs": 113830,
    "text": "isn't the hard part anymore. Verifying"
  },
  {
    "startMs": 113840,
    "text": "isn't the hard part anymore. Verifying it is. So when someone puts up a PR, how"
  },
  {
    "startMs": 115990,
    "text": "it is. So when someone puts up a PR, how"
  },
  {
    "startMs": 116000,
    "text": "it is. So when someone puts up a PR, how do you know that it's actually safe to"
  },
  {
    "startMs": 117109,
    "text": "do you know that it's actually safe to"
  },
  {
    "startMs": 117119,
    "text": "do you know that it's actually safe to ship? The bottleneck is really shifting"
  },
  {
    "startMs": 118789,
    "text": "ship? The bottleneck is really shifting"
  },
  {
    "startMs": 118799,
    "text": "ship? The bottleneck is really shifting from writing to verifying. And that's"
  },
  {
    "startMs": 120870,
    "text": "from writing to verifying. And that's"
  },
  {
    "startMs": 120880,
    "text": "from writing to verifying. And that's where unit tests come in. They make up"
  },
  {
    "startMs": 122709,
    "text": "where unit tests come in. They make up"
  },
  {
    "startMs": 122719,
    "text": "where unit tests come in. They make up the base of the testing pyramid. They're"
  },
  {
    "startMs": 124069,
    "text": "the base of the testing pyramid. They're"
  },
  {
    "startMs": 124079,
    "text": "the base of the testing pyramid. They're pretty cheap to write and pass to run."
  },
  {
    "startMs": 125510,
    "text": "pretty cheap to write and pass to run."
  },
  {
    "startMs": 125520,
    "text": "pretty cheap to write and pass to run. And they also act as guardrails for"
  },
  {
    "startMs": 126709,
    "text": "And they also act as guardrails for"
  },
  {
    "startMs": 126719,
    "text": "And they also act as guardrails for agents. So when an agent's change breaks"
  },
  {
    "startMs": 128630,
    "text": "agents. So when an agent's change breaks"
  },
  {
    "startMs": 128640,
    "text": "agents. So when an agent's change breaks a test in CI, it gets immediate feedback"
  },
  {
    "startMs": 130469,
    "text": "a test in CI, it gets immediate feedback"
  },
  {
    "startMs": 130479,
    "text": "a test in CI, it gets immediate feedback on what to fix. They also act as living"
  },
  {
    "startMs": 132550,
    "text": "on what to fix. They also act as living"
  },
  {
    "startMs": 132560,
    "text": "on what to fix. They also act as living documentation on what the code is"
  },
  {
    "startMs": 133750,
    "text": "documentation on what the code is"
  },
  {
    "startMs": 133760,
    "text": "documentation on what the code is supposed to do. And here's where we"
  },
  {
    "startMs": 135270,
    "text": "supposed to do. And here's where we"
  },
  {
    "startMs": 135280,
    "text": "supposed to do. And here's where we ended up 8 months later. Overall unit"
  },
  {
    "startMs": 136869,
    "text": "ended up 8 months later. Overall unit"
  },
  {
    "startMs": 136879,
    "text": "ended up 8 months later. Overall unit test coverage went from 11% to 24%."
  },
  {
    "startMs": 139190,
    "text": "test coverage went from 11% to 24%."
  },
  {
    "startMs": 139200,
    "text": "test coverage went from 11% to 24%. Repositories went from 11 to 72. View"
  },
  {
    "startMs": 141350,
    "text": "Repositories went from 11 to 72. View"
  },
  {
    "startMs": 141360,
    "text": "Repositories went from 11 to 72. View models from 9 to 58. Data sources from 5"
  },
  {
    "startMs": 143670,
    "text": "models from 9 to 58. Data sources from 5"
  },
  {
    "startMs": 143680,
    "text": "models from 9 to 58. Data sources from 5 to 41. And the combined MVM slice of the"
  },
  {
    "startMs": 145830,
    "text": "to 41. And the combined MVM slice of the"
  },
  {
    "startMs": 145840,
    "text": "to 41. And the combined MVM slice of the app went from 9% to 59%. So what did we"
  },
  {
    "startMs": 148790,
    "text": "app went from 9% to 59%. So what did we"
  },
  {
    "startMs": 148800,
    "text": "app went from 9% to 59%. So what did we actually end up building? On paper, it's"
  },
  {
    "startMs": 150550,
    "text": "actually end up building? On paper, it's"
  },
  {
    "startMs": 150560,
    "text": "actually end up building? On paper, it's a pretty simple loop. First, you"
  },
  {
    "startMs": 151990,
    "text": "a pretty simple loop. First, you"
  },
  {
    "startMs": 152000,
    "text": "a pretty simple loop. First, you identify the files that need tests. Then"
  },
  {
    "startMs": 153750,
    "text": "identify the files that need tests. Then"
  },
  {
    "startMs": 153760,
    "text": "identify the files that need tests. Then you write those tests with an agent. You"
  },
  {
    "startMs": 155430,
    "text": "you write those tests with an agent. You"
  },
  {
    "startMs": 155440,
    "text": "you write those tests with an agent. You verify that the tests are good quality"
  },
  {
    "startMs": 156710,
    "text": "verify that the tests are good quality"
  },
  {
    "startMs": 156720,
    "text": "verify that the tests are good quality and they pass CI. Then you resolve the"
  },
  {
    "startMs": 158150,
    "text": "and they pass CI. Then you resolve the"
  },
  {
    "startMs": 158160,
    "text": "and they pass CI. Then you resolve the PR, which means getting attributed by an"
  },
  {
    "startMs": 159830,
    "text": "PR, which means getting attributed by an"
  },
  {
    "startMs": 159840,
    "text": "PR, which means getting attributed by an engineer merged. And finally, it on the"
  },
  {
    "startMs": 161910,
    "text": "engineer merged. And finally, it on the"
  },
  {
    "startMs": 161920,
    "text": "engineer merged. And finally, it on the feedback so that the next batch of PRs"
  },
  {
    "startMs": 163270,
    "text": "feedback so that the next batch of PRs"
  },
  {
    "startMs": 163280,
    "text": "feedback so that the next batch of PRs is better than the last one. But in"
  },
  {
    "startMs": 165589,
    "text": "is better than the last one. But in"
  },
  {
    "startMs": 165599,
    "text": "is better than the last one. But in production, it's a little messier. This"
  },
  {
    "startMs": 167030,
    "text": "production, it's a little messier. This"
  },
  {
    "startMs": 167040,
    "text": "production, it's a little messier. This is what our system ended up looking"
  },
  {
    "startMs": 168229,
    "text": "is what our system ended up looking"
  },
  {
    "startMs": 168239,
    "text": "is what our system ended up looking like. I won't go through every box right"
  },
  {
    "startMs": 169750,
    "text": "like. I won't go through every box right"
  },
  {
    "startMs": 169760,
    "text": "like. I won't go through every box right now, but at a high level, it's a few"
  },
  {
    "startMs": 170949,
    "text": "now, but at a high level, it's a few"
  },
  {
    "startMs": 170959,
    "text": "now, but at a high level, it's a few temporal workflows that coordinate"
  },
  {
    "startMs": 172229,
    "text": "temporal workflows that coordinate"
  },
  {
    "startMs": 172239,
    "text": "temporal workflows that coordinate through shared state stored in S3. And"
  },
  {
    "startMs": 174550,
    "text": "through shared state stored in S3. And"
  },
  {
    "startMs": 174560,
    "text": "through shared state stored in S3. And every box you see here maps back to one"
  },
  {
    "startMs": 176070,
    "text": "every box you see here maps back to one"
  },
  {
    "startMs": 176080,
    "text": "every box you see here maps back to one of the five steps I just talked about."
  },
  {
    "startMs": 178550,
    "text": "of the five steps I just talked about."
  },
  {
    "startMs": 178560,
    "text": "of the five steps I just talked about. Here are the building blocks that we"
  },
  {
    "startMs": 179670,
    "text": "Here are the building blocks that we"
  },
  {
    "startMs": 179680,
    "text": "Here are the building blocks that we ended up using. Temporal handles"
  },
  {
    "startMs": 181430,
    "text": "ended up using. Temporal handles"
  },
  {
    "startMs": 181440,
    "text": "ended up using. Temporal handles workflow orchestration and execution."
  },
  {
    "startMs": 183030,
    "text": "workflow orchestration and execution."
  },
  {
    "startMs": 183040,
    "text": "workflow orchestration and execution. Cloud code acts as the coding agent. S3"
  },
  {
    "startMs": 185350,
    "text": "Cloud code acts as the coding agent. S3"
  },
  {
    "startMs": 185360,
    "text": "Cloud code acts as the coding agent. S3 holds a shared state that keeps our"
  },
  {
    "startMs": 186869,
    "text": "holds a shared state that keeps our"
  },
  {
    "startMs": 186879,
    "text": "holds a shared state that keeps our workflows in sync. And we use GitHub for"
  },
  {
    "startMs": 188790,
    "text": "workflows in sync. And we use GitHub for"
  },
  {
    "startMs": 188800,
    "text": "workflows in sync. And we use GitHub for actions, labels, and PRs. Data dogs acts"
  },
  {
    "startMs": 191350,
    "text": "actions, labels, and PRs. Data dogs acts"
  },
  {
    "startMs": 191360,
    "text": "actions, labels, and PRs. Data dogs acts as observability layer where we keep an"
  },
  {
    "startMs": 192710,
    "text": "as observability layer where we keep an"
  },
  {
    "startMs": 192720,
    "text": "as observability layer where we keep an eye on the health of our pipelines."
  },
  {
    "startMs": 195430,
    "text": "eye on the health of our pipelines."
  },
  {
    "startMs": 195440,
    "text": "eye on the health of our pipelines. If you haven't used empodoral before,"
  },
  {
    "startMs": 196710,
    "text": "If you haven't used empodoral before,"
  },
  {
    "startMs": 196720,
    "text": "If you haven't used empodoral before, it's basically a durable workflow"
  },
  {
    "startMs": 197990,
    "text": "it's basically a durable workflow"
  },
  {
    "startMs": 198000,
    "text": "it's basically a durable workflow execution engine. You write your"
  },
  {
    "startMs": 199509,
    "text": "execution engine. You write your"
  },
  {
    "startMs": 199519,
    "text": "execution engine. You write your workflows as normal Python code and if a"
  },
  {
    "startMs": 201110,
    "text": "workflows as normal Python code and if a"
  },
  {
    "startMs": 201120,
    "text": "workflows as normal Python code and if a worker crashes, the workflow just picks"
  },
  {
    "startMs": 202550,
    "text": "worker crashes, the workflow just picks"
  },
  {
    "startMs": 202560,
    "text": "worker crashes, the workflow just picks up right where it left off. This fits"
  },
  {
    "startMs": 204630,
    "text": "up right where it left off. This fits"
  },
  {
    "startMs": 204640,
    "text": "up right where it left off. This fits agents really well and agent session can"
  },
  {
    "startMs": 206470,
    "text": "agents really well and agent session can"
  },
  {
    "startMs": 206480,
    "text": "agents really well and agent session can run for seconds to hours and it can call"
  },
  {
    "startMs": 208390,
    "text": "run for seconds to hours and it can call"
  },
  {
    "startMs": 208400,
    "text": "run for seconds to hours and it can call out to tools like GitHub, CI, S3, and"
  },
  {
    "startMs": 210470,
    "text": "out to tools like GitHub, CI, S3, and"
  },
  {
    "startMs": 210480,
    "text": "out to tools like GitHub, CI, S3, and Claude. We also lean on a few built-in"
  },
  {
    "startMs": 212630,
    "text": "Claude. We also lean on a few built-in"
  },
  {
    "startMs": 212640,
    "text": "Claude. We also lean on a few built-in features of Temporal. There are"
  },
  {
    "startMs": 214149,
    "text": "features of Temporal. There are"
  },
  {
    "startMs": 214159,
    "text": "features of Temporal. There are schedules which are basically chronike"
  },
  {
    "startMs": 215589,
    "text": "schedules which are basically chronike"
  },
  {
    "startMs": 215599,
    "text": "schedules which are basically chronike schedules for our jobs. There are"
  },
  {
    "startMs": 216949,
    "text": "schedules for our jobs. There are"
  },
  {
    "startMs": 216959,
    "text": "schedules for our jobs. There are composible workflows so that we can fan"
  },
  {
    "startMs": 218309,
    "text": "composible workflows so that we can fan"
  },
  {
    "startMs": 218319,
    "text": "composible workflows so that we can fan out work in parallel. Then there's first"
  },
  {
    "startMs": 220070,
    "text": "out work in parallel. Then there's first"
  },
  {
    "startMs": 220080,
    "text": "out work in parallel. Then there's first class support for retries and timeouts"
  },
  {
    "startMs": 221589,
    "text": "class support for retries and timeouts"
  },
  {
    "startMs": 221599,
    "text": "class support for retries and timeouts and item potency so that we never kick"
  },
  {
    "startMs": 223190,
    "text": "and item potency so that we never kick"
  },
  {
    "startMs": 223200,
    "text": "and item potency so that we never kick off duplicate runs of the same file."
  },
  {
    "startMs": 225830,
    "text": "off duplicate runs of the same file."
  },
  {
    "startMs": 225840,
    "text": "off duplicate runs of the same file. To split up this work, we split this"
  },
  {
    "startMs": 227430,
    "text": "To split up this work, we split this"
  },
  {
    "startMs": 227440,
    "text": "To split up this work, we split this project into four milestones. The first"
  },
  {
    "startMs": 229030,
    "text": "project into four milestones. The first"
  },
  {
    "startMs": 229040,
    "text": "project into four milestones. The first was local validation. Basically, is this"
  },
  {
    "startMs": 231110,
    "text": "was local validation. Basically, is this"
  },
  {
    "startMs": 231120,
    "text": "was local validation. Basically, is this project even viable? The second was a"
  },
  {
    "startMs": 232869,
    "text": "project even viable? The second was a"
  },
  {
    "startMs": 232879,
    "text": "project even viable? The second was a scheduled backfield workflow to write"
  },
  {
    "startMs": 234229,
    "text": "scheduled backfield workflow to write"
  },
  {
    "startMs": 234239,
    "text": "scheduled backfield workflow to write tests for existing code. The third was a"
  },
  {
    "startMs": 236149,
    "text": "tests for existing code. The third was a"
  },
  {
    "startMs": 236159,
    "text": "tests for existing code. The third was a PR life cycle workflow to actually get"
  },
  {
    "startMs": 238070,
    "text": "PR life cycle workflow to actually get"
  },
  {
    "startMs": 238080,
    "text": "PR life cycle workflow to actually get those PRs merged. And the fourth was a"
  },
  {
    "startMs": 239910,
    "text": "those PRs merged. And the fourth was a"
  },
  {
    "startMs": 239920,
    "text": "those PRs merged. And the fourth was a label trigger so engineers can get tests"
  },
  {
    "startMs": 241750,
    "text": "label trigger so engineers can get tests"
  },
  {
    "startMs": 241760,
    "text": "label trigger so engineers can get tests for the new code that they're writing."
  },
  {
    "startMs": 243350,
    "text": "for the new code that they're writing."
  },
  {
    "startMs": 243360,
    "text": "for the new code that they're writing. Now, we'll go through each of these in"
  },
  {
    "startMs": 244789,
    "text": "Now, we'll go through each of these in"
  },
  {
    "startMs": 244799,
    "text": "Now, we'll go through each of these in order. Before building anything real, we"
  },
  {
    "startMs": 246630,
    "text": "order. Before building anything real, we"
  },
  {
    "startMs": 246640,
    "text": "order. Before building anything real, we wanted to do the least amount of work"
  },
  {
    "startMs": 247830,
    "text": "wanted to do the least amount of work"
  },
  {
    "startMs": 247840,
    "text": "wanted to do the least amount of work possible that would tell us whether an"
  },
  {
    "startMs": 249110,
    "text": "possible that would tell us whether an"
  },
  {
    "startMs": 249120,
    "text": "possible that would tell us whether an agent could actually do this job well"
  },
  {
    "startMs": 250229,
    "text": "agent could actually do this job well"
  },
  {
    "startMs": 250239,
    "text": "agent could actually do this job well and do it at scale. We started with a"
  },
  {
    "startMs": 252149,
    "text": "and do it at scale. We started with a"
  },
  {
    "startMs": 252159,
    "text": "and do it at scale. We started with a very simple local cloud code setup where"
  },
  {
    "startMs": 254309,
    "text": "very simple local cloud code setup where"
  },
  {
    "startMs": 254319,
    "text": "very simple local cloud code setup where we had one cloud code command that acted"
  },
  {
    "startMs": 255830,
    "text": "we had one cloud code command that acted"
  },
  {
    "startMs": 255840,
    "text": "we had one cloud code command that acted as an orchestrator. It ran an Xcode"
  },
  {
    "startMs": 257990,
    "text": "as an orchestrator. It ran an Xcode"
  },
  {
    "startMs": 258000,
    "text": "as an orchestrator. It ran an Xcode build script to generate a code coverage"
  },
  {
    "startMs": 259590,
    "text": "build script to generate a code coverage"
  },
  {
    "startMs": 259600,
    "text": "build script to generate a code coverage report locally. Then it fed that report"
  },
  {
    "startMs": 261430,
    "text": "report locally. Then it fed that report"
  },
  {
    "startMs": 261440,
    "text": "report locally. Then it fed that report through a file selection script that we"
  },
  {
    "startMs": 262710,
    "text": "through a file selection script that we"
  },
  {
    "startMs": 262720,
    "text": "through a file selection script that we wrote and then it handed each selected"
  },
  {
    "startMs": 264550,
    "text": "wrote and then it handed each selected"
  },
  {
    "startMs": 264560,
    "text": "wrote and then it handed each selected file off of sub two sub aents. One wrote"
  },
  {
    "startMs": 267110,
    "text": "file off of sub two sub aents. One wrote"
  },
  {
    "startMs": 267120,
    "text": "file off of sub two sub aents. One wrote the tests and the other one opened the"
  },
  {
    "startMs": 268469,
    "text": "the tests and the other one opened the"
  },
  {
    "startMs": 268479,
    "text": "the tests and the other one opened the PR. We ran this in a loop and generated"
  },
  {
    "startMs": 270390,
    "text": "PR. We ran this in a loop and generated"
  },
  {
    "startMs": 270400,
    "text": "PR. We ran this in a loop and generated a bunch of PRs. We basically ran this in"
  },
  {
    "startMs": 272870,
    "text": "a bunch of PRs. We basically ran this in"
  },
  {
    "startMs": 272880,
    "text": "a bunch of PRs. We basically ran this in two batches. The first patch had 17 PRs"
  },
  {
    "startMs": 274870,
    "text": "two batches. The first patch had 17 PRs"
  },
  {
    "startMs": 274880,
    "text": "two batches. The first patch had 17 PRs and we merged eight of them. The second"
  },
  {
    "startMs": 276230,
    "text": "and we merged eight of them. The second"
  },
  {
    "startMs": 276240,
    "text": "and we merged eight of them. The second had 40 PRs and we merged 19. So that's"
  },
  {
    "startMs": 278070,
    "text": "had 40 PRs and we merged 19. So that's"
  },
  {
    "startMs": 278080,
    "text": "had 40 PRs and we merged 19. So that's roughly a 50% merge rate. But the goal"
  },
  {
    "startMs": 279830,
    "text": "roughly a 50% merge rate. But the goal"
  },
  {
    "startMs": 279840,
    "text": "roughly a 50% merge rate. But the goal here wasn't to merge a lot of PRs. It"
  },
  {
    "startMs": 281110,
    "text": "here wasn't to merge a lot of PRs. It"
  },
  {
    "startMs": 281120,
    "text": "here wasn't to merge a lot of PRs. It was to learn how the agents fail and to"
  },
  {
    "startMs": 282390,
    "text": "was to learn how the agents fail and to"
  },
  {
    "startMs": 282400,
    "text": "was to learn how the agents fail and to fix those paper cuts before"
  },
  {
    "startMs": 283270,
    "text": "fix those paper cuts before"
  },
  {
    "startMs": 283280,
    "text": "fix those paper cuts before productionizing anything. Here are some"
  },
  {
    "startMs": 285749,
    "text": "productionizing anything. Here are some"
  },
  {
    "startMs": 285759,
    "text": "productionizing anything. Here are some of the failure modes that we saw. Since"
  },
  {
    "startMs": 287270,
    "text": "of the failure modes that we saw. Since"
  },
  {
    "startMs": 287280,
    "text": "of the failure modes that we saw. Since we're a large legacy code base with both"
  },
  {
    "startMs": 288710,
    "text": "we're a large legacy code base with both"
  },
  {
    "startMs": 288720,
    "text": "we're a large legacy code base with both XC test and swift testing, the agents"
  },
  {
    "startMs": 290230,
    "text": "XC test and swift testing, the agents"
  },
  {
    "startMs": 290240,
    "text": "XC test and swift testing, the agents would sometimes get confused and mix the"
  },
  {
    "startMs": 291510,
    "text": "would sometimes get confused and mix the"
  },
  {
    "startMs": 291520,
    "text": "would sometimes get confused and mix the two in one file. We fixed that by giving"
  },
  {
    "startMs": 293189,
    "text": "two in one file. We fixed that by giving"
  },
  {
    "startMs": 293199,
    "text": "two in one file. We fixed that by giving the agents clear rules on when to use"
  },
  {
    "startMs": 294710,
    "text": "the agents clear rules on when to use"
  },
  {
    "startMs": 294720,
    "text": "the agents clear rules on when to use each. We also had a giant user client"
  },
  {
    "startMs": 296790,
    "text": "each. We also had a giant user client"
  },
  {
    "startMs": 296800,
    "text": "each. We also had a giant user client singleton class. Some code passed around"
  },
  {
    "startMs": 298550,
    "text": "singleton class. Some code passed around"
  },
  {
    "startMs": 298560,
    "text": "singleton class. Some code passed around the concrete version of this class which"
  },
  {
    "startMs": 299670,
    "text": "the concrete version of this class which"
  },
  {
    "startMs": 299680,
    "text": "the concrete version of this class which is really hard to mock whereas others"
  },
  {
    "startMs": 300790,
    "text": "is really hard to mock whereas others"
  },
  {
    "startMs": 300800,
    "text": "is really hard to mock whereas others passed around a protocolized version. So"
  },
  {
    "startMs": 302390,
    "text": "passed around a protocolized version. So"
  },
  {
    "startMs": 302400,
    "text": "passed around a protocolized version. So we migrated the entire codebase over to"
  },
  {
    "startMs": 303830,
    "text": "we migrated the entire codebase over to"
  },
  {
    "startMs": 303840,
    "text": "we migrated the entire codebase over to the protocolized version and added lint"
  },
  {
    "startMs": 305670,
    "text": "the protocolized version and added lint"
  },
  {
    "startMs": 305680,
    "text": "the protocolized version and added lint rule to enforce it. This made the whole"
  },
  {
    "startMs": 307430,
    "text": "rule to enforce it. This made the whole"
  },
  {
    "startMs": 307440,
    "text": "rule to enforce it. This made the whole host of files easier to test for agents"
  },
  {
    "startMs": 309189,
    "text": "host of files easier to test for agents"
  },
  {
    "startMs": 309199,
    "text": "host of files easier to test for agents and for humans. We also ran into six"
  },
  {
    "startMs": 312070,
    "text": "and for humans. We also ran into six"
  },
  {
    "startMs": 312080,
    "text": "and for humans. We also ran into six concurrency related issues and agents"
  },
  {
    "startMs": 313909,
    "text": "concurrency related issues and agents"
  },
  {
    "startMs": 313919,
    "text": "concurrency related issues and agents writing timing based tests which are a"
  },
  {
    "startMs": 315189,
    "text": "writing timing based tests which are a"
  },
  {
    "startMs": 315199,
    "text": "writing timing based tests which are a big source of flakiness in CI. So we"
  },
  {
    "startMs": 316950,
    "text": "big source of flakiness in CI. So we"
  },
  {
    "startMs": 316960,
    "text": "big source of flakiness in CI. So we improved the state for both of these"
  },
  {
    "startMs": 317990,
    "text": "improved the state for both of these"
  },
  {
    "startMs": 318000,
    "text": "improved the state for both of these with better rules and examples in our"
  },
  {
    "startMs": 319430,
    "text": "with better rules and examples in our"
  },
  {
    "startMs": 319440,
    "text": "with better rules and examples in our testing skills."
  },
  {
    "startMs": 321270,
    "text": "testing skills."
  },
  {
    "startMs": 321280,
    "text": "testing skills. Now this brings us to milestone 2. Since"
  },
  {
    "startMs": 323110,
    "text": "Now this brings us to milestone 2. Since"
  },
  {
    "startMs": 323120,
    "text": "Now this brings us to milestone 2. Since we were so far behind on code coverage,"
  },
  {
    "startMs": 324790,
    "text": "we were so far behind on code coverage,"
  },
  {
    "startMs": 324800,
    "text": "we were so far behind on code coverage, the biggest leverage that we saw was"
  },
  {
    "startMs": 326070,
    "text": "the biggest leverage that we saw was"
  },
  {
    "startMs": 326080,
    "text": "the biggest leverage that we saw was backfilling tests for code that was"
  },
  {
    "startMs": 327350,
    "text": "backfilling tests for code that was"
  },
  {
    "startMs": 327360,
    "text": "backfilling tests for code that was already merged in. That's where the"
  },
  {
    "startMs": 329110,
    "text": "already merged in. That's where the"
  },
  {
    "startMs": 329120,
    "text": "already merged in. That's where the scheduled backfill workflow comes in."
  },
  {
    "startMs": 330790,
    "text": "scheduled backfill workflow comes in."
  },
  {
    "startMs": 330800,
    "text": "scheduled backfill workflow comes in. This is a system design diagram of the"
  },
  {
    "startMs": 332150,
    "text": "This is a system design diagram of the"
  },
  {
    "startMs": 332160,
    "text": "This is a system design diagram of the workflow as it runs in production and it"
  },
  {
    "startMs": 334070,
    "text": "workflow as it runs in production and it"
  },
  {
    "startMs": 334080,
    "text": "workflow as it runs in production and it essentially does two jobs. The left half"
  },
  {
    "startMs": 335830,
    "text": "essentially does two jobs. The left half"
  },
  {
    "startMs": 335840,
    "text": "essentially does two jobs. The left half is to identify which is picking the"
  },
  {
    "startMs": 337350,
    "text": "is to identify which is picking the"
  },
  {
    "startMs": 337360,
    "text": "is to identify which is picking the files that need tests and the right half"
  },
  {
    "startMs": 339110,
    "text": "files that need tests and the right half"
  },
  {
    "startMs": 339120,
    "text": "files that need tests and the right half is right which is having the agents"
  },
  {
    "startMs": 341029,
    "text": "is right which is having the agents"
  },
  {
    "startMs": 341039,
    "text": "is right which is having the agents actually write those tests. Let's start"
  },
  {
    "startMs": 343510,
    "text": "actually write those tests. Let's start"
  },
  {
    "startMs": 343520,
    "text": "actually write those tests. Let's start with identify. Every time a PR merges to"
  },
  {
    "startMs": 345510,
    "text": "with identify. Every time a PR merges to"
  },
  {
    "startMs": 345520,
    "text": "with identify. Every time a PR merges to the main branch in the iOS repo, we kick"
  },
  {
    "startMs": 346950,
    "text": "the main branch in the iOS repo, we kick"
  },
  {
    "startMs": 346960,
    "text": "the main branch in the iOS repo, we kick off a GitHub access job that generates"
  },
  {
    "startMs": 348550,
    "text": "off a GitHub access job that generates"
  },
  {
    "startMs": 348560,
    "text": "off a GitHub access job that generates an Xcode coverage report and uploads it"
  },
  {
    "startMs": 350230,
    "text": "an Xcode coverage report and uploads it"
  },
  {
    "startMs": 350240,
    "text": "an Xcode coverage report and uploads it to S3 that becomes a source of truth for"
  },
  {
    "startMs": 352310,
    "text": "to S3 that becomes a source of truth for"
  },
  {
    "startMs": 352320,
    "text": "to S3 that becomes a source of truth for the latest code coverage. The back fill"
  },
  {
    "startMs": 354150,
    "text": "the latest code coverage. The back fill"
  },
  {
    "startMs": 354160,
    "text": "the latest code coverage. The back fill workflow then fetches that report and"
  },
  {
    "startMs": 355430,
    "text": "workflow then fetches that report and"
  },
  {
    "startMs": 355440,
    "text": "workflow then fetches that report and runs it through a selector script which"
  },
  {
    "startMs": 356790,
    "text": "runs it through a selector script which"
  },
  {
    "startMs": 356800,
    "text": "runs it through a selector script which picks the next set of files to test."
  },
  {
    "startMs": 359270,
    "text": "picks the next set of files to test."
  },
  {
    "startMs": 359280,
    "text": "picks the next set of files to test. Let's get a deeper a little deeper into"
  },
  {
    "startMs": 360629,
    "text": "Let's get a deeper a little deeper into"
  },
  {
    "startMs": 360639,
    "text": "Let's get a deeper a little deeper into the selector script. Its job is"
  },
  {
    "startMs": 362469,
    "text": "the selector script. Its job is"
  },
  {
    "startMs": 362479,
    "text": "the selector script. Its job is essentially to score every Swift file"
  },
  {
    "startMs": 363830,
    "text": "essentially to score every Swift file"
  },
  {
    "startMs": 363840,
    "text": "essentially to score every Swift file based on how confident we are that an"
  },
  {
    "startMs": 365270,
    "text": "based on how confident we are that an"
  },
  {
    "startMs": 365280,
    "text": "based on how confident we are that an agent can write a good test for it. The"
  },
  {
    "startMs": 367029,
    "text": "agent can write a good test for it. The"
  },
  {
    "startMs": 367039,
    "text": "agent can write a good test for it. The strongest signal is file type. So view"
  },
  {
    "startMs": 369029,
    "text": "strongest signal is file type. So view"
  },
  {
    "startMs": 369039,
    "text": "strongest signal is file type. So view models, repositories, and data sources"
  },
  {
    "startMs": 370469,
    "text": "models, repositories, and data sources"
  },
  {
    "startMs": 370479,
    "text": "models, repositories, and data sources get a 2.5x boost while legacy files get"
  },
  {
    "startMs": 372790,
    "text": "get a 2.5x boost while legacy files get"
  },
  {
    "startMs": 372800,
    "text": "get a 2.5x boost while legacy files get pushed down. We skipped UI code"
  },
  {
    "startMs": 374550,
    "text": "pushed down. We skipped UI code"
  },
  {
    "startMs": 374560,
    "text": "pushed down. We skipped UI code completely for now since snapshot tests"
  },
  {
    "startMs": 375830,
    "text": "completely for now since snapshot tests"
  },
  {
    "startMs": 375840,
    "text": "completely for now since snapshot tests are better fit there. And larger files"
  },
  {
    "startMs": 377749,
    "text": "are better fit there. And larger files"
  },
  {
    "startMs": 377759,
    "text": "are better fit there. And larger files also get a penalty. So we prioritize"
  },
  {
    "startMs": 379270,
    "text": "also get a penalty. So we prioritize"
  },
  {
    "startMs": 379280,
    "text": "also get a penalty. So we prioritize code that we're confident that we can"
  },
  {
    "startMs": 380629,
    "text": "code that we're confident that we can"
  },
  {
    "startMs": 380639,
    "text": "code that we're confident that we can test well."
  },
  {
    "startMs": 382629,
    "text": "test well."
  },
  {
    "startMs": 382639,
    "text": "test well. The next step is to actually write the"
  },
  {
    "startMs": 383990,
    "text": "The next step is to actually write the"
  },
  {
    "startMs": 384000,
    "text": "The next step is to actually write the tests. We kick off one child workflow"
  },
  {
    "startMs": 385990,
    "text": "tests. We kick off one child workflow"
  },
  {
    "startMs": 386000,
    "text": "tests. We kick off one child workflow per file, all running in parallel. In"
  },
  {
    "startMs": 388390,
    "text": "per file, all running in parallel. In"
  },
  {
    "startMs": 388400,
    "text": "per file, all running in parallel. In each one of these runs, the agent clones"
  },
  {
    "startMs": 389909,
    "text": "each one of these runs, the agent clones"
  },
  {
    "startMs": 389919,
    "text": "each one of these runs, the agent clones the iOS repository, reads the testing"
  },
  {
    "startMs": 391749,
    "text": "the iOS repository, reads the testing"
  },
  {
    "startMs": 391759,
    "text": "the iOS repository, reads the testing skill that we keep in the repo, which"
  },
  {
    "startMs": 393189,
    "text": "skill that we keep in the repo, which"
  },
  {
    "startMs": 393199,
    "text": "skill that we keep in the repo, which spells out how to write a good test."
  },
  {
    "startMs": 394950,
    "text": "spells out how to write a good test."
  },
  {
    "startMs": 394960,
    "text": "spells out how to write a good test. Then it writes the test plus any mocks"
  },
  {
    "startMs": 396710,
    "text": "Then it writes the test plus any mocks"
  },
  {
    "startMs": 396720,
    "text": "Then it writes the test plus any mocks and fakes it needs and opens a PR with a"
  },
  {
    "startMs": 398390,
    "text": "and fakes it needs and opens a PR with a"
  },
  {
    "startMs": 398400,
    "text": "and fakes it needs and opens a PR with a generated test label so we can identify"
  },
  {
    "startMs": 400070,
    "text": "generated test label so we can identify"
  },
  {
    "startMs": 400080,
    "text": "generated test label so we can identify it easily later."
  },
  {
    "startMs": 402550,
    "text": "it easily later."
  },
  {
    "startMs": 402560,
    "text": "it easily later. These agents run on Dualingo's cloud"
  },
  {
    "startMs": 403990,
    "text": "These agents run on Dualingo's cloud"
  },
  {
    "startMs": 404000,
    "text": "These agents run on Dualingo's cloud coding agent which is a platform that"
  },
  {
    "startMs": 405270,
    "text": "coding agent which is a platform that"
  },
  {
    "startMs": 405280,
    "text": "coding agent which is a platform that also runs on Neporal. The simplest way"
  },
  {
    "startMs": 406950,
    "text": "also runs on Neporal. The simplest way"
  },
  {
    "startMs": 406960,
    "text": "also runs on Neporal. The simplest way to think about it is as an abstraction"
  },
  {
    "startMs": 408230,
    "text": "to think about it is as an abstraction"
  },
  {
    "startMs": 408240,
    "text": "to think about it is as an abstraction on top of cloud code and codex running"
  },
  {
    "startMs": 409510,
    "text": "on top of cloud code and codex running"
  },
  {
    "startMs": 409520,
    "text": "on top of cloud code and codex running in the cloud. We can invoke it from"
  },
  {
    "startMs": 411189,
    "text": "in the cloud. We can invoke it from"
  },
  {
    "startMs": 411199,
    "text": "in the cloud. We can invoke it from Slack, temporal or GitHub. It talks to a"
  },
  {
    "startMs": 412950,
    "text": "Slack, temporal or GitHub. It talks to a"
  },
  {
    "startMs": 412960,
    "text": "Slack, temporal or GitHub. It talks to a bunch of our internal tools for MCP and"
  },
  {
    "startMs": 414629,
    "text": "bunch of our internal tools for MCP and"
  },
  {
    "startMs": 414639,
    "text": "bunch of our internal tools for MCP and it logs a bunch of events so we can see"
  },
  {
    "startMs": 416390,
    "text": "it logs a bunch of events so we can see"
  },
  {
    "startMs": 416400,
    "text": "it logs a bunch of events so we can see what each agent run is doing and how"
  },
  {
    "startMs": 417670,
    "text": "what each agent run is doing and how"
  },
  {
    "startMs": 417680,
    "text": "what each agent run is doing and how much it costs. We started running these"
  },
  {
    "startMs": 420309,
    "text": "much it costs. We started running these"
  },
  {
    "startMs": 420319,
    "text": "much it costs. We started running these on Linux machines and then later move"
  },
  {
    "startMs": 421990,
    "text": "on Linux machines and then later move"
  },
  {
    "startMs": 422000,
    "text": "on Linux machines and then later move them to MacRunners which will become"
  },
  {
    "startMs": 423350,
    "text": "them to MacRunners which will become"
  },
  {
    "startMs": 423360,
    "text": "them to MacRunners which will become important in a minute. Here's what a"
  },
  {
    "startMs": 425830,
    "text": "important in a minute. Here's what a"
  },
  {
    "startMs": 425840,
    "text": "important in a minute. Here's what a generated test PR looks like. Every PR"
  },
  {
    "startMs": 427749,
    "text": "generated test PR looks like. Every PR"
  },
  {
    "startMs": 427759,
    "text": "generated test PR looks like. Every PR description follows the same template"
  },
  {
    "startMs": 429029,
    "text": "description follows the same template"
  },
  {
    "startMs": 429039,
    "text": "description follows the same template and it says explicitly that the PR is AI"
  },
  {
    "startMs": 430950,
    "text": "and it says explicitly that the PR is AI"
  },
  {
    "startMs": 430960,
    "text": "and it says explicitly that the PR is AI generated. The reviewer has four"
  },
  {
    "startMs": 432629,
    "text": "generated. The reviewer has four"
  },
  {
    "startMs": 432639,
    "text": "generated. The reviewer has four options. They can approve it and the PR"
  },
  {
    "startMs": 434150,
    "text": "options. They can approve it and the PR"
  },
  {
    "startMs": 434160,
    "text": "options. They can approve it and the PR automarges. They can push changes to it"
  },
  {
    "startMs": 436070,
    "text": "automarges. They can push changes to it"
  },
  {
    "startMs": 436080,
    "text": "automarges. They can push changes to it directly. They can reassign it to"
  },
  {
    "startMs": 437350,
    "text": "directly. They can reassign it to"
  },
  {
    "startMs": 437360,
    "text": "directly. They can reassign it to someone better suited or they can close"
  },
  {
    "startMs": 438790,
    "text": "someone better suited or they can close"
  },
  {
    "startMs": 438800,
    "text": "someone better suited or they can close it out with feedback on why it isn't"
  },
  {
    "startMs": 439990,
    "text": "it out with feedback on why it isn't"
  },
  {
    "startMs": 440000,
    "text": "it out with feedback on why it isn't mergeable."
  },
  {
    "startMs": 441670,
    "text": "mergeable."
  },
  {
    "startMs": 441680,
    "text": "mergeable. Now once we had the backfl running in"
  },
  {
    "startMs": 443189,
    "text": "Now once we had the backfl running in"
  },
  {
    "startMs": 443199,
    "text": "Now once we had the backfl running in production we had a slightly new"
  },
  {
    "startMs": 444230,
    "text": "production we had a slightly new"
  },
  {
    "startMs": 444240,
    "text": "production we had a slightly new problem. We suddenly had tens of tens to"
  },
  {
    "startMs": 446469,
    "text": "problem. We suddenly had tens of tens to"
  },
  {
    "startMs": 446479,
    "text": "problem. We suddenly had tens of tens to hundreds of open PRs that nobody could"
  },
  {
    "startMs": 447830,
    "text": "hundreds of open PRs that nobody could"
  },
  {
    "startMs": 447840,
    "text": "hundreds of open PRs that nobody could manage by hand. So how do we get all of"
  },
  {
    "startMs": 449670,
    "text": "manage by hand. So how do we get all of"
  },
  {
    "startMs": 449680,
    "text": "manage by hand. So how do we get all of these to a mergeable state and in front"
  },
  {
    "startMs": 450870,
    "text": "these to a mergeable state and in front"
  },
  {
    "startMs": 450880,
    "text": "these to a mergeable state and in front of the right reviewer"
  },
  {
    "startMs": 452950,
    "text": "of the right reviewer"
  },
  {
    "startMs": 452960,
    "text": "of the right reviewer that's where the PR life cycle workflow"
  },
  {
    "startMs": 454469,
    "text": "that's where the PR life cycle workflow"
  },
  {
    "startMs": 454479,
    "text": "that's where the PR life cycle workflow comes in. You can think of it as a state"
  },
  {
    "startMs": 456070,
    "text": "comes in. You can think of it as a state"
  },
  {
    "startMs": 456080,
    "text": "comes in. You can think of it as a state machine. It runs every couple of hours"
  },
  {
    "startMs": 457749,
    "text": "machine. It runs every couple of hours"
  },
  {
    "startMs": 457759,
    "text": "machine. It runs every couple of hours looks through every open generated test"
  },
  {
    "startMs": 459350,
    "text": "looks through every open generated test"
  },
  {
    "startMs": 459360,
    "text": "looks through every open generated test PR and takes exactly one action on each."
  },
  {
    "startMs": 462070,
    "text": "PR and takes exactly one action on each."
  },
  {
    "startMs": 462080,
    "text": "PR and takes exactly one action on each. It covers the next two steps of our"
  },
  {
    "startMs": 463749,
    "text": "It covers the next two steps of our"
  },
  {
    "startMs": 463759,
    "text": "It covers the next two steps of our loop. Verify which is does this PR pass"
  },
  {
    "startMs": 465990,
    "text": "loop. Verify which is does this PR pass"
  },
  {
    "startMs": 466000,
    "text": "loop. Verify which is does this PR pass CI and is it high quality and resolve"
  },
  {
    "startMs": 467830,
    "text": "CI and is it high quality and resolve"
  },
  {
    "startMs": 467840,
    "text": "CI and is it high quality and resolve which is getting a human to review it"
  },
  {
    "startMs": 469029,
    "text": "which is getting a human to review it"
  },
  {
    "startMs": 469039,
    "text": "which is getting a human to review it and monitor it. The logic for this"
  },
  {
    "startMs": 471350,
    "text": "and monitor it. The logic for this"
  },
  {
    "startMs": 471360,
    "text": "and monitor it. The logic for this workflow is a simple decision tree. If"
  },
  {
    "startMs": 472950,
    "text": "workflow is a simple decision tree. If"
  },
  {
    "startMs": 472960,
    "text": "workflow is a simple decision tree. If there are any merge conflicts or the PR"
  },
  {
    "startMs": 474230,
    "text": "there are any merge conflicts or the PR"
  },
  {
    "startMs": 474240,
    "text": "there are any merge conflicts or the PR has been open for about 3 weeks, we"
  },
  {
    "startMs": 475350,
    "text": "has been open for about 3 weeks, we"
  },
  {
    "startMs": 475360,
    "text": "has been open for about 3 weeks, we close it out and mark it as still. If CI"
  },
  {
    "startMs": 477189,
    "text": "close it out and mark it as still. If CI"
  },
  {
    "startMs": 477199,
    "text": "close it out and mark it as still. If CI is still running or a reviewer is"
  },
  {
    "startMs": 478390,
    "text": "is still running or a reviewer is"
  },
  {
    "startMs": 478400,
    "text": "is still running or a reviewer is already on it, we skip it. If CI is"
  },
  {
    "startMs": 480070,
    "text": "already on it, we skip it. If CI is"
  },
  {
    "startMs": 480080,
    "text": "already on it, we skip it. If CI is green and there's no reviewers, we find"
  },
  {
    "startMs": 481270,
    "text": "green and there's no reviewers, we find"
  },
  {
    "startMs": 481280,
    "text": "green and there's no reviewers, we find one assign them. If CI is red, we try to"
  },
  {
    "startMs": 483270,
    "text": "one assign them. If CI is red, we try to"
  },
  {
    "startMs": 483280,
    "text": "one assign them. If CI is red, we try to fix it. And if a reviewer rejects the"
  },
  {
    "startMs": 485430,
    "text": "fix it. And if a reviewer rejects the"
  },
  {
    "startMs": 485440,
    "text": "fix it. And if a reviewer rejects the PR, the file goes into 30-day cool down."
  },
  {
    "startMs": 487189,
    "text": "PR, the file goes into 30-day cool down."
  },
  {
    "startMs": 487199,
    "text": "PR, the file goes into 30-day cool down. So, we're not spamming engineers with"
  },
  {
    "startMs": 488230,
    "text": "So, we're not spamming engineers with"
  },
  {
    "startMs": 488240,
    "text": "So, we're not spamming engineers with the same PR again and again."
  },
  {
    "startMs": 490629,
    "text": "the same PR again and again."
  },
  {
    "startMs": 490639,
    "text": "the same PR again and again. When CI fails, the first thing we check"
  },
  {
    "startMs": 492390,
    "text": "When CI fails, the first thing we check"
  },
  {
    "startMs": 492400,
    "text": "When CI fails, the first thing we check is which job failed. If it's pre-commit,"
  },
  {
    "startMs": 494309,
    "text": "is which job failed. If it's pre-commit,"
  },
  {
    "startMs": 494319,
    "text": "is which job failed. If it's pre-commit, a GitHub action applies the fix. It's"
  },
  {
    "startMs": 495990,
    "text": "a GitHub action applies the fix. It's"
  },
  {
    "startMs": 496000,
    "text": "a GitHub action applies the fix. It's cheap, it's deterministic, and there's"
  },
  {
    "startMs": 497189,
    "text": "cheap, it's deterministic, and there's"
  },
  {
    "startMs": 497199,
    "text": "cheap, it's deterministic, and there's no agent needed. If it's a real test"
  },
  {
    "startMs": 498869,
    "text": "no agent needed. If it's a real test"
  },
  {
    "startMs": 498879,
    "text": "no agent needed. If it's a real test failure, we kick off another agent that"
  },
  {
    "startMs": 500309,
    "text": "failure, we kick off another agent that"
  },
  {
    "startMs": 500319,
    "text": "failure, we kick off another agent that reads the CI logs and pushes a fix to"
  },
  {
    "startMs": 501749,
    "text": "reads the CI logs and pushes a fix to"
  },
  {
    "startMs": 501759,
    "text": "reads the CI logs and pushes a fix to the PR. We cap that whole loop at five"
  },
  {
    "startMs": 503909,
    "text": "the PR. We cap that whole loop at five"
  },
  {
    "startMs": 503919,
    "text": "the PR. We cap that whole loop at five attemps. Past that we close the PR"
  },
  {
    "startMs": 505350,
    "text": "attemps. Past that we close the PR"
  },
  {
    "startMs": 505360,
    "text": "attemps. Past that we close the PR rather than trying to fix it forever."
  },
  {
    "startMs": 507830,
    "text": "rather than trying to fix it forever."
  },
  {
    "startMs": 507840,
    "text": "rather than trying to fix it forever. Once CI is green, we need to find the"
  },
  {
    "startMs": 509270,
    "text": "Once CI is green, we need to find the"
  },
  {
    "startMs": 509280,
    "text": "Once CI is green, we need to find the right reviewer. What we do is we run git"
  },
  {
    "startMs": 511110,
    "text": "right reviewer. What we do is we run git"
  },
  {
    "startMs": 511120,
    "text": "right reviewer. What we do is we run git blame on the source file being tested."
  },
  {
    "startMs": 512389,
    "text": "blame on the source file being tested."
  },
  {
    "startMs": 512399,
    "text": "blame on the source file being tested. Find the top contributor and assign it"
  },
  {
    "startMs": 514070,
    "text": "Find the top contributor and assign it"
  },
  {
    "startMs": 514080,
    "text": "Find the top contributor and assign it to them and turn on auto merge. If they"
  },
  {
    "startMs": 515829,
    "text": "to them and turn on auto merge. If they"
  },
  {
    "startMs": 515839,
    "text": "to them and turn on auto merge. If they approve it, the PR gets merged. If they"
  },
  {
    "startMs": 517350,
    "text": "approve it, the PR gets merged. If they"
  },
  {
    "startMs": 517360,
    "text": "approve it, the PR gets merged. If they rejected, the file goes into the 30-day"
  },
  {
    "startMs": 518790,
    "text": "rejected, the file goes into the 30-day"
  },
  {
    "startMs": 518800,
    "text": "rejected, the file goes into the 30-day cool down, which I previously mentioned."
  },
  {
    "startMs": 520870,
    "text": "cool down, which I previously mentioned."
  },
  {
    "startMs": 520880,
    "text": "cool down, which I previously mentioned. This brings us to milestone four. Once"
  },
  {
    "startMs": 522949,
    "text": "This brings us to milestone four. Once"
  },
  {
    "startMs": 522959,
    "text": "This brings us to milestone four. Once all of this was back filling tests for"
  },
  {
    "startMs": 524310,
    "text": "all of this was back filling tests for"
  },
  {
    "startMs": 524320,
    "text": "all of this was back filling tests for existing code, we wanted to make it just"
  },
  {
    "startMs": 525750,
    "text": "existing code, we wanted to make it just"
  },
  {
    "startMs": 525760,
    "text": "existing code, we wanted to make it just as easy to get tests for the new code"
  },
  {
    "startMs": 527030,
    "text": "as easy to get tests for the new code"
  },
  {
    "startMs": 527040,
    "text": "as easy to get tests for the new code engineers were writing. Now any engineer"
  },
  {
    "startMs": 529430,
    "text": "engineers were writing. Now any engineer"
  },
  {
    "startMs": 529440,
    "text": "engineers were writing. Now any engineer can add or write test label to their own"
  },
  {
    "startMs": 530870,
    "text": "can add or write test label to their own"
  },
  {
    "startMs": 530880,
    "text": "can add or write test label to their own PR, it hooks into the same temporal"
  },
  {
    "startMs": 532870,
    "text": "PR, it hooks into the same temporal"
  },
  {
    "startMs": 532880,
    "text": "PR, it hooks into the same temporal infrastructure we just walked through."
  },
  {
    "startMs": 534150,
    "text": "infrastructure we just walked through."
  },
  {
    "startMs": 534160,
    "text": "infrastructure we just walked through. The only real difference is selection."
  },
  {
    "startMs": 535670,
    "text": "The only real difference is selection."
  },
  {
    "startMs": 535680,
    "text": "The only real difference is selection. Instead of the selection script, we take"
  },
  {
    "startMs": 537110,
    "text": "Instead of the selection script, we take"
  },
  {
    "startMs": 537120,
    "text": "Instead of the selection script, we take the file straight from the engineer's"
  },
  {
    "startMs": 538230,
    "text": "the file straight from the engineer's"
  },
  {
    "startMs": 538240,
    "text": "the file straight from the engineer's PR. The agent writes the test and you"
  },
  {
    "startMs": 540150,
    "text": "PR. The agent writes the test and you"
  },
  {
    "startMs": 540160,
    "text": "PR. The agent writes the test and you get a draft SPR link back to your"
  },
  {
    "startMs": 541509,
    "text": "get a draft SPR link back to your"
  },
  {
    "startMs": 541519,
    "text": "get a draft SPR link back to your original one. Now let's talk about some"
  },
  {
    "startMs": 543030,
    "text": "original one. Now let's talk about some"
  },
  {
    "startMs": 543040,
    "text": "original one. Now let's talk about some of the challenges that we ran into. Our"
  },
  {
    "startMs": 544870,
    "text": "of the challenges that we ran into. Our"
  },
  {
    "startMs": 544880,
    "text": "of the challenges that we ran into. Our agents originally ran on Linux machines"
  },
  {
    "startMs": 546550,
    "text": "agents originally ran on Linux machines"
  },
  {
    "startMs": 546560,
    "text": "agents originally ran on Linux machines which don't have X code or Swift"
  },
  {
    "startMs": 547910,
    "text": "which don't have X code or Swift"
  },
  {
    "startMs": 547920,
    "text": "which don't have X code or Swift installed. So when an agent would open a"
  },
  {
    "startMs": 549509,
    "text": "installed. So when an agent would open a"
  },
  {
    "startMs": 549519,
    "text": "installed. So when an agent would open a PR after about 20 or 30 minutes CI would"
  },
  {
    "startMs": 551670,
    "text": "PR after about 20 or 30 minutes CI would"
  },
  {
    "startMs": 551680,
    "text": "PR after about 20 or 30 minutes CI would fail frequently on lint. This wastes the"
  },
  {
    "startMs": 553750,
    "text": "fail frequently on lint. This wastes the"
  },
  {
    "startMs": 553760,
    "text": "fail frequently on lint. This wastes the agents time and CI time as well which is"
  },
  {
    "startMs": 555509,
    "text": "agents time and CI time as well which is"
  },
  {
    "startMs": 555519,
    "text": "agents time and CI time as well which is an increasingly scarce resource. For a"
  },
  {
    "startMs": 557509,
    "text": "an increasingly scarce resource. For a"
  },
  {
    "startMs": 557519,
    "text": "an increasingly scarce resource. For a while 30 13.6% of our PRs were failing"
  },
  {
    "startMs": 559670,
    "text": "while 30 13.6% of our PRs were failing"
  },
  {
    "startMs": 559680,
    "text": "while 30 13.6% of our PRs were failing due to lint and then when we dug into it"
  },
  {
    "startMs": 561269,
    "text": "due to lint and then when we dug into it"
  },
  {
    "startMs": 561279,
    "text": "due to lint and then when we dug into it 62% of those failures came from just to"
  },
  {
    "startMs": 562870,
    "text": "62% of those failures came from just to"
  },
  {
    "startMs": 562880,
    "text": "62% of those failures came from just to lint rules. We fixed this in layers. The"
  },
  {
    "startMs": 565430,
    "text": "lint rules. We fixed this in layers. The"
  },
  {
    "startMs": 565440,
    "text": "lint rules. We fixed this in layers. The first was better examples in the prompt"
  },
  {
    "startMs": 567030,
    "text": "first was better examples in the prompt"
  },
  {
    "startMs": 567040,
    "text": "first was better examples in the prompt along with pitfalls to avoid. The second"
  },
  {
    "startMs": 568949,
    "text": "along with pitfalls to avoid. The second"
  },
  {
    "startMs": 568959,
    "text": "along with pitfalls to avoid. The second was a GitHub action that auto fixes lint"
  },
  {
    "startMs": 570630,
    "text": "was a GitHub action that auto fixes lint"
  },
  {
    "startMs": 570640,
    "text": "was a GitHub action that auto fixes lint errors in CI. The third was an audit of"
  },
  {
    "startMs": 572550,
    "text": "errors in CI. The third was an audit of"
  },
  {
    "startMs": 572560,
    "text": "errors in CI. The third was an audit of the rules that couldn't be autofixed. We"
  },
  {
    "startMs": 574310,
    "text": "the rules that couldn't be autofixed. We"
  },
  {
    "startMs": 574320,
    "text": "the rules that couldn't be autofixed. We essentially removed the ones that"
  },
  {
    "startMs": 575350,
    "text": "essentially removed the ones that"
  },
  {
    "startMs": 575360,
    "text": "essentially removed the ones that weren't high signal and this made life"
  },
  {
    "startMs": 576870,
    "text": "weren't high signal and this made life"
  },
  {
    "startMs": 576880,
    "text": "weren't high signal and this made life easier for both agents and human"
  },
  {
    "startMs": 578070,
    "text": "easier for both agents and human"
  },
  {
    "startMs": 578080,
    "text": "easier for both agents and human engineers. And finally, the real fix"
  },
  {
    "startMs": 579750,
    "text": "engineers. And finally, the real fix"
  },
  {
    "startMs": 579760,
    "text": "engineers. And finally, the real fix came when we moved the agents to run on"
  },
  {
    "startMs": 581030,
    "text": "came when we moved the agents to run on"
  },
  {
    "startMs": 581040,
    "text": "came when we moved the agents to run on Mac runners. So now the agents could"
  },
  {
    "startMs": 582790,
    "text": "Mac runners. So now the agents could"
  },
  {
    "startMs": 582800,
    "text": "Mac runners. So now the agents could build and lend the code locally before"
  },
  {
    "startMs": 584389,
    "text": "build and lend the code locally before"
  },
  {
    "startMs": 584399,
    "text": "build and lend the code locally before ever opening a PR. The second challenge"
  },
  {
    "startMs": 587110,
    "text": "ever opening a PR. The second challenge"
  },
  {
    "startMs": 587120,
    "text": "ever opening a PR. The second challenge was PR quality. When we assign a PR to a"
  },
  {
    "startMs": 589269,
    "text": "was PR quality. When we assign a PR to a"
  },
  {
    "startMs": 589279,
    "text": "was PR quality. When we assign a PR to a human, we wanted to be something they'd"
  },
  {
    "startMs": 590470,
    "text": "human, we wanted to be something they'd"
  },
  {
    "startMs": 590480,
    "text": "human, we wanted to be something they'd actually feel comfortable proving. But"
  },
  {
    "startMs": 592230,
    "text": "actually feel comfortable proving. But"
  },
  {
    "startMs": 592240,
    "text": "actually feel comfortable proving. But agents sometimes take the shortest paths"
  },
  {
    "startMs": 593670,
    "text": "agents sometimes take the shortest paths"
  },
  {
    "startMs": 593680,
    "text": "agents sometimes take the shortest paths to green CI. We saw agents editing"
  },
  {
    "startMs": 595750,
    "text": "to green CI. We saw agents editing"
  },
  {
    "startMs": 595760,
    "text": "to green CI. We saw agents editing production code to make a failing test"
  },
  {
    "startMs": 596949,
    "text": "production code to make a failing test"
  },
  {
    "startMs": 596959,
    "text": "production code to make a failing test pass. We also saw them modifying,"
  },
  {
    "startMs": 598310,
    "text": "pass. We also saw them modifying,"
  },
  {
    "startMs": 598320,
    "text": "pass. We also saw them modifying, deleting, or silencing existing tests."
  },
  {
    "startMs": 599910,
    "text": "deleting, or silencing existing tests."
  },
  {
    "startMs": 599920,
    "text": "deleting, or silencing existing tests. And then we saw them adding long weight"
  },
  {
    "startMs": 601190,
    "text": "And then we saw them adding long weight"
  },
  {
    "startMs": 601200,
    "text": "And then we saw them adding long weight to tests which can make CI flaky."
  },
  {
    "startMs": 603910,
    "text": "to tests which can make CI flaky."
  },
  {
    "startMs": 603920,
    "text": "to tests which can make CI flaky. Again, there was no single fix here, so"
  },
  {
    "startMs": 605509,
    "text": "Again, there was no single fix here, so"
  },
  {
    "startMs": 605519,
    "text": "Again, there was no single fix here, so we built three layers. The first is the"
  },
  {
    "startMs": 607430,
    "text": "we built three layers. The first is the"
  },
  {
    "startMs": 607440,
    "text": "we built three layers. The first is the deterministic rule. If a PR touches any"
  },
  {
    "startMs": 609509,
    "text": "deterministic rule. If a PR touches any"
  },
  {
    "startMs": 609519,
    "text": "deterministic rule. If a PR touches any file that isn't a test, a mock, a fake,"
  },
  {
    "startMs": 611269,
    "text": "file that isn't a test, a mock, a fake,"
  },
  {
    "startMs": 611279,
    "text": "file that isn't a test, a mock, a fake, or a stub, we close it out. The second"
  },
  {
    "startMs": 613350,
    "text": "or a stub, we close it out. The second"
  },
  {
    "startMs": 613360,
    "text": "or a stub, we close it out. The second builds on our AI code review. At"
  },
  {
    "startMs": 614949,
    "text": "builds on our AI code review. At"
  },
  {
    "startMs": 614959,
    "text": "builds on our AI code review. At Dualingo, we use Claude and Codex to"
  },
  {
    "startMs": 616470,
    "text": "Dualingo, we use Claude and Codex to"
  },
  {
    "startMs": 616480,
    "text": "Dualingo, we use Claude and Codex to review every PR. And those B bots often"
  },
  {
    "startMs": 618550,
    "text": "review every PR. And those B bots often"
  },
  {
    "startMs": 618560,
    "text": "review every PR. And those B bots often leave comments. So, we built an agent"
  },
  {
    "startMs": 620389,
    "text": "leave comments. So, we built an agent"
  },
  {
    "startMs": 620399,
    "text": "leave comments. So, we built an agent that resolves any P 0 comments before a"
  },
  {
    "startMs": 622150,
    "text": "that resolves any P 0 comments before a"
  },
  {
    "startMs": 622160,
    "text": "that resolves any P 0 comments before a human ever looks at the PR. And the"
  },
  {
    "startMs": 624310,
    "text": "human ever looks at the PR. And the"
  },
  {
    "startMs": 624320,
    "text": "human ever looks at the PR. And the third is an LLM as a judge. It's a"
  },
  {
    "startMs": 625829,
    "text": "third is an LLM as a judge. It's a"
  },
  {
    "startMs": 625839,
    "text": "third is an LLM as a judge. It's a reviewer agent built just for these test"
  },
  {
    "startMs": 627269,
    "text": "reviewer agent built just for these test"
  },
  {
    "startMs": 627279,
    "text": "reviewer agent built just for these test PRs. It reads the PR diff and rejects"
  },
  {
    "startMs": 629430,
    "text": "PRs. It reads the PR diff and rejects"
  },
  {
    "startMs": 629440,
    "text": "PRs. It reads the PR diff and rejects production edits, changed or deleted"
  },
  {
    "startMs": 630870,
    "text": "production edits, changed or deleted"
  },
  {
    "startMs": 630880,
    "text": "production edits, changed or deleted tests, disabled tests, and long weights."
  },
  {
    "startMs": 633030,
    "text": "tests, disabled tests, and long weights."
  },
  {
    "startMs": 633040,
    "text": "tests, disabled tests, and long weights. And as you learn more, we keep adding"
  },
  {
    "startMs": 634389,
    "text": "And as you learn more, we keep adding"
  },
  {
    "startMs": 634399,
    "text": "And as you learn more, we keep adding additional checks to it."
  },
  {
    "startMs": 636949,
    "text": "additional checks to it."
  },
  {
    "startMs": 636959,
    "text": "additional checks to it. So far, the LM as a judge has reviewed"
  },
  {
    "startMs": 638470,
    "text": "So far, the LM as a judge has reviewed"
  },
  {
    "startMs": 638480,
    "text": "So far, the LM as a judge has reviewed about,200 PRs and rejected 73 of them."
  },
  {
    "startMs": 641190,
    "text": "about,200 PRs and rejected 73 of them."
  },
  {
    "startMs": 641200,
    "text": "about,200 PRs and rejected 73 of them. Every rejection comes with a comment"
  },
  {
    "startMs": 642470,
    "text": "Every rejection comes with a comment"
  },
  {
    "startMs": 642480,
    "text": "Every rejection comes with a comment like this one, saying exactly why the PR"
  },
  {
    "startMs": 644069,
    "text": "like this one, saying exactly why the PR"
  },
  {
    "startMs": 644079,
    "text": "like this one, saying exactly why the PR was rejected. We then read that feedback"
  },
  {
    "startMs": 646069,
    "text": "was rejected. We then read that feedback"
  },
  {
    "startMs": 646079,
    "text": "was rejected. We then read that feedback and fold it back into our prompts and"
  },
  {
    "startMs": 647430,
    "text": "and fold it back into our prompts and"
  },
  {
    "startMs": 647440,
    "text": "and fold it back into our prompts and skills, so the same mistakes get less"
  },
  {
    "startMs": 648949,
    "text": "skills, so the same mistakes get less"
  },
  {
    "startMs": 648959,
    "text": "skills, so the same mistakes get less likely each time. So, let's talk about"
  },
  {
    "startMs": 651430,
    "text": "likely each time. So, let's talk about"
  },
  {
    "startMs": 651440,
    "text": "likely each time. So, let's talk about what all of this added up to. Since we"
  },
  {
    "startMs": 653350,
    "text": "what all of this added up to. Since we"
  },
  {
    "startMs": 653360,
    "text": "what all of this added up to. Since we started, the pipeline has merged more"
  },
  {
    "startMs": 654710,
    "text": "started, the pipeline has merged more"
  },
  {
    "startMs": 654720,
    "text": "started, the pipeline has merged more than 700 test PRs, adding about 225,000"
  },
  {
    "startMs": 657269,
    "text": "than 700 test PRs, adding about 225,000"
  },
  {
    "startMs": 657279,
    "text": "than 700 test PRs, adding about 225,000 lines of test code. Every one of those"
  },
  {
    "startMs": 659030,
    "text": "lines of test code. Every one of those"
  },
  {
    "startMs": 659040,
    "text": "lines of test code. Every one of those PRs was reviewed and approved by a"
  },
  {
    "startMs": 660470,
    "text": "PRs was reviewed and approved by a"
  },
  {
    "startMs": 660480,
    "text": "PRs was reviewed and approved by a human. MVVM test coverage went from 9%"
  },
  {
    "startMs": 663030,
    "text": "human. MVVM test coverage went from 9%"
  },
  {
    "startMs": 663040,
    "text": "human. MVVM test coverage went from 9% to 59% and overall coverage went from"
  },
  {
    "startMs": 664949,
    "text": "to 59% and overall coverage went from"
  },
  {
    "startMs": 664959,
    "text": "to 59% and overall coverage went from 11% to 24%. Most importantly, we now"
  },
  {
    "startMs": 667670,
    "text": "11% to 24%. Most importantly, we now"
  },
  {
    "startMs": 667680,
    "text": "11% to 24%. Most importantly, we now have a code base so engineers and agents"
  },
  {
    "startMs": 668949,
    "text": "have a code base so engineers and agents"
  },
  {
    "startMs": 668959,
    "text": "have a code base so engineers and agents can ship faster and with more"
  },
  {
    "startMs": 669990,
    "text": "can ship faster and with more"
  },
  {
    "startMs": 670000,
    "text": "can ship faster and with more confidence. Here's a few more"
  },
  {
    "startMs": 672230,
    "text": "confidence. Here's a few more"
  },
  {
    "startMs": 672240,
    "text": "confidence. Here's a few more interesting stats. Since March, the"
  },
  {
    "startMs": 673750,
    "text": "interesting stats. Since March, the"
  },
  {
    "startMs": 673760,
    "text": "interesting stats. Since March, the pipeline has written 34% of all new test"
  },
  {
    "startMs": 675590,
    "text": "pipeline has written 34% of all new test"
  },
  {
    "startMs": 675600,
    "text": "pipeline has written 34% of all new test files in the repo, 44% of all new test"
  },
  {
    "startMs": 677750,
    "text": "files in the repo, 44% of all new test"
  },
  {
    "startMs": 677760,
    "text": "files in the repo, 44% of all new test cases, and 10% of every swift line added"
  },
  {
    "startMs": 679190,
    "text": "cases, and 10% of every swift line added"
  },
  {
    "startMs": 679200,
    "text": "cases, and 10% of every swift line added to the codebase. And interestingly, that"
  },
  {
    "startMs": 680949,
    "text": "to the codebase. And interestingly, that"
  },
  {
    "startMs": 680959,
    "text": "to the codebase. And interestingly, that share has actually dripped in the last"
  },
  {
    "startMs": 682150,
    "text": "share has actually dripped in the last"
  },
  {
    "startMs": 682160,
    "text": "share has actually dripped in the last few weeks because engineers are now"
  },
  {
    "startMs": 683269,
    "text": "few weeks because engineers are now"
  },
  {
    "startMs": 683279,
    "text": "few weeks because engineers are now writing a lot more tests themselves,"
  },
  {
    "startMs": 684550,
    "text": "writing a lot more tests themselves,"
  },
  {
    "startMs": 684560,
    "text": "writing a lot more tests themselves, which is honestly the end goal."
  },
  {
    "startMs": 687430,
    "text": "which is honestly the end goal."
  },
  {
    "startMs": 687440,
    "text": "which is honestly the end goal. Um, and here's the coverage curve since"
  },
  {
    "startMs": 689670,
    "text": "Um, and here's the coverage curve since"
  },
  {
    "startMs": 689680,
    "text": "Um, and here's the coverage curve since April when the backfield started running"
  },
  {
    "startMs": 690949,
    "text": "April when the backfield started running"
  },
  {
    "startMs": 690959,
    "text": "April when the backfield started running on a schedule. You can see it bent"
  },
  {
    "startMs": 692630,
    "text": "on a schedule. You can see it bent"
  },
  {
    "startMs": 692640,
    "text": "on a schedule. You can see it bent slightly upward around June and July,"
  },
  {
    "startMs": 693910,
    "text": "slightly upward around June and July,"
  },
  {
    "startMs": 693920,
    "text": "slightly upward around June and July, which is when we really ramped up the"
  },
  {
    "startMs": 695030,
    "text": "which is when we really ramped up the"
  },
  {
    "startMs": 695040,
    "text": "which is when we really ramped up the pipeline."
  },
  {
    "startMs": 696710,
    "text": "pipeline."
  },
  {
    "startMs": 696720,
    "text": "pipeline. So, what have we learned while doing"
  },
  {
    "startMs": 697910,
    "text": "So, what have we learned while doing"
  },
  {
    "startMs": 697920,
    "text": "So, what have we learned while doing this project? The first thing is that"
  },
  {
    "startMs": 699590,
    "text": "this project? The first thing is that"
  },
  {
    "startMs": 699600,
    "text": "this project? The first thing is that this pattern generalizes the loop we"
  },
  {
    "startMs": 701430,
    "text": "this pattern generalizes the loop we"
  },
  {
    "startMs": 701440,
    "text": "this pattern generalizes the loop we talked about identify, write, verify,"
  },
  {
    "startMs": 703110,
    "text": "talked about identify, write, verify,"
  },
  {
    "startMs": 703120,
    "text": "talked about identify, write, verify, resolve, iterate. It isn't specific to"
  },
  {
    "startMs": 704630,
    "text": "resolve, iterate. It isn't specific to"
  },
  {
    "startMs": 704640,
    "text": "resolve, iterate. It isn't specific to testing. Unit tests were just the lowest"
  },
  {
    "startMs": 706310,
    "text": "testing. Unit tests were just the lowest"
  },
  {
    "startMs": 706320,
    "text": "testing. Unit tests were just the lowest hanging fruit and any job that fits the"
  },
  {
    "startMs": 707990,
    "text": "hanging fruit and any job that fits the"
  },
  {
    "startMs": 708000,
    "text": "hanging fruit and any job that fits the shape can run on the same infrastructure"
  },
  {
    "startMs": 709590,
    "text": "shape can run on the same infrastructure"
  },
  {
    "startMs": 709600,
    "text": "shape can run on the same infrastructure and that's exactly what we're starting"
  },
  {
    "startMs": 710710,
    "text": "and that's exactly what we're starting"
  },
  {
    "startMs": 710720,
    "text": "and that's exactly what we're starting to do at the company. We're applying the"
  },
  {
    "startMs": 712310,
    "text": "to do at the company. We're applying the"
  },
  {
    "startMs": 712320,
    "text": "to do at the company. We're applying the same loop to modernization to"
  },
  {
    "startMs": 713910,
    "text": "same loop to modernization to"
  },
  {
    "startMs": 713920,
    "text": "same loop to modernization to modernizing screens to MVM and Swift UI,"
  },
  {
    "startMs": 715829,
    "text": "modernizing screens to MVM and Swift UI,"
  },
  {
    "startMs": 715839,
    "text": "modernizing screens to MVM and Swift UI, migrating code uh on Android to Corines"
  },
  {
    "startMs": 718150,
    "text": "migrating code uh on Android to Corines"
  },
  {
    "startMs": 718160,
    "text": "migrating code uh on Android to Corines and generating snapshot tests."
  },
  {
    "startMs": 720949,
    "text": "and generating snapshot tests."
  },
  {
    "startMs": 720959,
    "text": "and generating snapshot tests. So before I wrap up, I want to leave you"
  },
  {
    "startMs": 722710,
    "text": "So before I wrap up, I want to leave you"
  },
  {
    "startMs": 722720,
    "text": "So before I wrap up, I want to leave you with a thought experiment. There's a new"
  },
  {
    "startMs": 724310,
    "text": "with a thought experiment. There's a new"
  },
  {
    "startMs": 724320,
    "text": "with a thought experiment. There's a new class of fast general purpose"
  },
  {
    "startMs": 725430,
    "text": "class of fast general purpose"
  },
  {
    "startMs": 725440,
    "text": "class of fast general purpose classifiers like Jev. They taken some"
  },
  {
    "startMs": 727110,
    "text": "classifiers like Jev. They taken some"
  },
  {
    "startMs": 727120,
    "text": "classifiers like Jev. They taken some state and output an action in"
  },
  {
    "startMs": 728230,
    "text": "state and output an action in"
  },
  {
    "startMs": 728240,
    "text": "state and output an action in milliseconds. So what if we replace a"
  },
  {
    "startMs": 729990,
    "text": "milliseconds. So what if we replace a"
  },
  {
    "startMs": 730000,
    "text": "milliseconds. So what if we replace a handwritten decision tree in the PR life"
  },
  {
    "startMs": 731430,
    "text": "handwritten decision tree in the PR life"
  },
  {
    "startMs": 731440,
    "text": "handwritten decision tree in the PR life cycle workflow with one of these? PR"
  },
  {
    "startMs": 733430,
    "text": "cycle workflow with one of these? PR"
  },
  {
    "startMs": 733440,
    "text": "cycle workflow with one of these? PR state goes in, the classifier picks the"
  },
  {
    "startMs": 735190,
    "text": "state goes in, the classifier picks the"
  },
  {
    "startMs": 735200,
    "text": "state goes in, the classifier picks the next action and we execute it all in"
  },
  {
    "startMs": 736710,
    "text": "next action and we execute it all in"
  },
  {
    "startMs": 736720,
    "text": "next action and we execute it all in many seconds."
  },
  {
    "startMs": 739110,
    "text": "many seconds."
  },
  {
    "startMs": 739120,
    "text": "many seconds. I also want to thank a few folks who"
  },
  {
    "startMs": 740629,
    "text": "I also want to thank a few folks who"
  },
  {
    "startMs": 740639,
    "text": "I also want to thank a few folks who worked on this with me, Jesse, Gordalup,"
  },
  {
    "startMs": 742389,
    "text": "worked on this with me, Jesse, Gordalup,"
  },
  {
    "startMs": 742399,
    "text": "worked on this with me, Jesse, Gordalup, Hunter and Jiho. They built a lot of the"
  },
  {
    "startMs": 743750,
    "text": "Hunter and Jiho. They built a lot of the"
  },
  {
    "startMs": 743760,
    "text": "Hunter and Jiho. They built a lot of the platform pieces this project stands on"
  },
  {
    "startMs": 745030,
    "text": "platform pieces this project stands on"
  },
  {
    "startMs": 745040,
    "text": "platform pieces this project stands on and helped review a lot of this code."
  },
  {
    "startMs": 747350,
    "text": "and helped review a lot of this code."
  },
  {
    "startMs": 747360,
    "text": "and helped review a lot of this code. That's it for me. Thank you so much for"
  },
  {
    "startMs": 748710,
    "text": "That's it for me. Thank you so much for"
  },
  {
    "startMs": 748720,
    "text": "That's it for me. Thank you so much for making it this far. If you want to go"
  },
  {
    "startMs": 749910,
    "text": "making it this far. If you want to go"
  },
  {
    "startMs": 749920,
    "text": "making it this far. If you want to go deeper, we've written about this"
  },
  {
    "startMs": 750870,
    "text": "deeper, we've written about this"
  },
  {
    "startMs": 750880,
    "text": "deeper, we've written about this pipeline on the dualing engineering"
  },
  {
    "startMs": 751990,
    "text": "pipeline on the dualing engineering"
  },
  {
    "startMs": 752000,
    "text": "pipeline on the dualing engineering blog. And I'd also love to hear from you"
  },
  {
    "startMs": 753430,
    "text": "blog. And I'd also love to hear from you"
  },
  {
    "startMs": 753440,
    "text": "blog. And I'd also love to hear from you if you're building something similar."
  },
  {
    "startMs": 754389,
    "text": "if you're building something similar."
  },
  {
    "startMs": 754399,
    "text": "if you're building something similar. You can find me on LinkedIn or next"
  }
]