# Earning Autonomy: From Missions to Workstreams Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [03:42:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13365s) All set. [03:42:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13367s) >> Hi everyone. Uh I'm Luke. I lead our [03:42:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13370s) core agent team here at Factory. Uh [03:42:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13372s) today I want to talk about autonomy and [03:42:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13374s) why uh the teams that are getting the [03:42:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13376s) most out of coding agents are not [03:42:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13377s) actually the ones that are handing over [03:42:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13379s) like the most work on day one. It's [03:43:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13380s) instead the folks who are ramping up [03:43:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13382s) autonomy gradually, one checkpoint at a [03:43:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13385s) time until what they have starts to [03:43:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13386s) resemble a software factory. [03:43:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13389s) So for those who don't know us, a [03:43:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13391s) factory is a platform uh that builds [03:43:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13393s) droid and then a a whole like surface [03:43:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13395s) for software engineering and building [03:43:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13397s) software on top of that coding agent. [03:43:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13400s) Hundreds of thousands of developers use [03:43:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13401s) us around the world at companies like [03:43:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13403s) Nvidia, T-Mobile, uh and Adobe. What [03:43:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13406s) these companies engineers are building [03:43:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13408s) with us is is really like a software [03:43:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13410s) factory, a system that runs 24/7 and [03:43:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13413s) builds software under the company's [03:43:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13414s) control. And that's a loop that takes in [03:43:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13418s) signals from wherever information is [03:43:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13419s) already available, turns them into [03:43:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13421s) changes, validates them, uh, and then [03:43:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13423s) learns from what production shows. A [03:43:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13425s) signal is really like anything that [03:43:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13427s) tells you how your software should [03:43:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13428s) evolve. So that could be, you know, a [03:43:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13430s) sentry alert or someone's idea on Slack. [03:43:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13433s) Every signal goes through the same [03:43:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13434s) stages. So you standardize you [03:43:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13436s) standardize the the process not the [03:43:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13438s) output. [03:44:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13440s) So we built that platform on five [03:44:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13442s) foundations. It has to be transparent, [03:44:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13445s) modular, governable, efficient and [03:44:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13448s) frontier. [03:44:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13450s) Most people picture a software factory [03:44:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13452s) as this kind of like lights out building [03:44:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13454s) where uh there's no people inside of it. [03:44:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13456s) Requirements come in one end and merged [03:44:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13458s) PRs come out the other. And and really [03:44:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13461s) what we saw firsthand in the field is [03:44:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13463s) that that's not the case. The best ones [03:44:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13464s) aren't set up like that. They're built [03:44:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13466s) around people instead. Uh, humans decide [03:44:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13469s) what matters and how much to trust [03:44:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13471s) droids over time. And those droids earn [03:44:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13473s) their keep as they deliver more work [03:44:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13474s) that's higher quality. [03:44:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13476s) Every company's software factory looks [03:44:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13478s) very different to each other. So, what [03:44:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13480s) we what we sell is not really this [03:44:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13481s) factory, it's the tools and the building [03:44:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13483s) blocks to compose them. [03:44:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13486s) Now, it's it's worth kind of looking at [03:44:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13488s) how we got here in the first place. Back [03:44:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13490s) in April, one of our flagship products [03:44:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13491s) was Missions, which was one of like the [03:44:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13494s) first multi- aent uh products to really [03:44:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13496s) take like a bite out of the human [03:44:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13498s) attention bottleneck that existed in in [03:45:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13500s) software engineering. You'd describe a [03:45:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13502s) goal, you'd like argue with the [03:45:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13504s) orchestrator until you were aligned, [03:45:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13506s) you'd approve the plan, and then it [03:45:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13508s) would go handle implementation for like [03:45:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13510s) days or even weeks. When the goal was [03:45:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13512s) clear and verifiable, you really could [03:45:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13514s) come back to like completed work pretty [03:45:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13516s) consistently. [03:45:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13518s) But as more teams adopted it, we heard [03:45:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13520s) some consistent like pieces of feedback. [03:45:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13522s) The first was how missions asked you to [03:45:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13524s) work. The product assumed that you could [03:45:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13526s) plan everything up front and then it [03:45:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13527s) would just work. Uh and the UX sort of [03:45:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13529s) suggested that you shouldn't interrupt [03:45:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13531s) them. Missions didn't really like ask [03:45:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13532s) you any questions or provide like an [03:45:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13534s) interactive surface for that uh [03:45:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13536s) communication between you and the agent [03:45:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13538s) once it was starting implementation. So [03:45:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13540s) this suits like migrations and big uh [03:45:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13543s) big coding like feature projects but it [03:45:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13545s) didn't really fit fit the bill for like [03:45:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13547s) most kinds of other work. [03:45:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13549s) The second was cost and time and that [03:45:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13552s) came from honestly like the models [03:45:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13554s) getting really really good. I'm sure [03:45:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13556s) you've all seen now adversarial review [03:45:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13558s) is really really good at finding these [03:46:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13560s) issues in your code. And so, uh, at the [03:46:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13562s) time we were adding adversarial review [03:46:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13565s) into every milestone's boundary. That [03:46:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13567s) just kind of like grew the scope of that [03:46:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13569s) mission from something like 60 features [03:46:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13572s) to over 200. And so you ended up having [03:46:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13575s) a token bill and a timeline that matched [03:46:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13577s) that. And then the last one was that [03:46:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13580s) missions were keeping their context to [03:46:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13581s) one person. So that was fine for like a [03:46:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13584s) single project, but it meant that the [03:46:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13585s) rest of the team couldn't really build [03:46:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13587s) on what that mission had learned. [03:46:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13590s) So we went back to the drawing board and [03:46:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13592s) one mental model kept on coming up. When [03:46:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13594s) you have like an org that is running [03:46:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13596s) dozens or hundreds of agents at a given [03:46:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13599s) point in time, you actually have like a [03:46:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13601s) lot of the the problems that operating [03:46:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13604s) systems back in the day solved. How to [03:46:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13606s) schedule work, how to run what many [03:46:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13608s) things at once safely, and how to share [03:46:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13610s) state. [03:46:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13613s) So every operating system starts with [03:46:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13615s) the thing that actually does the work [03:46:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13617s) and we wanted to revisit the agent [03:46:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13618s) itself. Long horizon agents run for [03:47:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13621s) thousands of turns and every [03:47:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13623s) inefficiency across of those gets [03:47:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13625s) multiplied. So you could have, you know, [03:47:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13627s) an extra inference request that's [03:47:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13628s) unnecessary or a a tool result that's [03:47:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13631s) pretty bloated or a prompt cache miss. [03:47:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13633s) All those things compound over time and [03:47:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13636s) lead to waste. [03:47:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13638s) So we decided to make some pretty [03:47:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13639s) significant changes to the harness [03:47:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13640s) itself. The biggest one was programmatic [03:47:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13643s) tool calling, also known as code mode. [03:47:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13645s) Instead of calling one tool, waiting on [03:47:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13648s) the model, and then calling the next, we [03:47:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13649s) have the model write a small program [03:47:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13651s) that executes all those tools in one go, [03:47:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13653s) and then prints out only what's [03:47:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13655s) relevant. [03:47:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13656s) So there's this example here where you [03:47:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13658s) have 150 linear tickets that are getting [03:47:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13660s) moved from one project to another. done [03:47:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13663s) the normal way that's like 30 inference [03:47:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13664s) requests, a million tokens, but as one [03:47:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13667s) model generated program, you actually [03:47:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13669s) only have like 60k tokens. It runs much [03:47:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13671s) faster and the model only sees what is [03:47:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13673s) actually relevant to the task that the [03:47:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13675s) the fact whether it was like successful [03:47:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13677s) or not. [03:47:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13679s) So this layer of the OS is where being [03:48:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13680s) frontier and efficient matters the most. [03:48:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13683s) A new model lands every few weeks and no [03:48:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13685s) two fail fail the same way. So we tune [03:48:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13687s) our harness in such a way that evens out [03:48:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13689s) this jaggedness across all these [03:48:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13691s) different models. The result is that [03:48:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13693s) Droid is now, you know, higher quality [03:48:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13695s) at a lower cost than every other coding [03:48:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13697s) agent on the market. [03:48:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13700s) Now you have a faster agent, but that [03:48:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13701s) that doesn't really give you like a step [03:48:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13703s) function increase in in how much work [03:48:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13705s) gets done inside of your system. For [03:48:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13707s) that, you need many agents running at [03:48:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13709s) once, and they need somewhere to run. [03:48:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13711s) remote missions were kind of held back [03:48:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13713s) that by this like fundamental limitation [03:48:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13715s) in the system where there was one [03:48:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13717s) computer that handled all the work. But [03:48:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13719s) to run a software factory, you need uh [03:48:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13721s) compute that scales with the volume of [03:48:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13723s) work in the system. Otherwise, every [03:48:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13725s) agent is is fighting for the same files [03:48:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13727s) or system resources. And so that's why [03:48:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13730s) we build environments which are reusable [03:48:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13732s) definitions of the sandboxes that your [03:48:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13733s) droids need, complete with the repos, [03:48:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13736s) tools, and dependencies that they that [03:48:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13738s) they need to get started with. The [03:49:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13740s) system is governable. So there's one [03:49:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13742s) comp control plane that orchestrates all [03:49:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13744s) of these sandboxes together wherever you [03:49:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13746s) need them. They can run inside of your [03:49:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13748s) own VPC on on EC2, Kubernetes, whatever [03:49:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13752s) compute you already operate. So your [03:49:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13754s) code, your data and your network are in [03:49:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13756s) your control. Compute becomes pluggable [03:49:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13759s) just like we made the BY options and [03:49:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13761s) automodel uh options that we provide uh [03:49:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13764s) make that provide sort of like this like [03:49:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13765s) model pluggable layer. So now we had [03:49:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13768s) agents that were cheap enough to run all [03:49:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13770s) the time and the compute to run many at [03:49:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13773s) once. And so then the next question was [03:49:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13775s) how do we orchestrate all of this work? [03:49:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13778s) At first we tried making like this DAG [03:49:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13780s) very complex system of agents composed [03:49:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13783s) together by like these very like strong [03:49:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13785s) rules and it didn't really work. You end [03:49:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13787s) up having a lot of the same problems [03:49:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13789s) that plagued missions uh in the first [03:49:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13791s) place because it was too rigid and as a [03:49:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13793s) result it was too expensive to run and [03:49:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13795s) it didn't generalize well. [03:49:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13797s) So what we landed on instead is [03:49:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13799s) something that we call a workstream. [03:50:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13801s) It's a standing goal with a team of [03:50:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13803s) agents to support it. It has this goal. [03:50:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13805s) It has an identity, a an environment [03:50:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13807s) that it runs on and memory. [03:50:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13810s) A new workstream runs as you and it has [03:50:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13813s) your access. When it outgrows one [03:50:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13815s) person, it can be moved to a service [03:50:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13817s) account with its own credentials. So you [03:50:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13819s) stay on as a member, but the work keeps [03:50:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13821s) h happening under an identity that the [03:50:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13823s) the org actually controls. [03:50:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13826s) At the center of every workstream is an [03:50:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13828s) orchestrator and its job looks a lot [03:50:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13830s) like a scheduler in an OS. Unlike the [03:50:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13833s) graph that we started with, it assembles [03:50:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13835s) its agents in whatever way is most [03:50:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13837s) natural for the model. So the [03:50:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13838s) instructions are pretty short, right? [03:50:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13840s) It's like uh delegate first. Give each [03:50:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13843s) agent an area of ownership and then for [03:50:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13845s) tasks that can fall into an existing [03:50:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13847s) area of ownership, delegate to those. [03:50:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13850s) You start by talking to it. uh you just [03:50:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13852s) kind of hand things over as you think of [03:50:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13854s) them and then it fans out the work so [03:50:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13856s) that then uh you can get the the control [03:50:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13858s) back to you. Uh agents report back when [03:51:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13861s) they finish, when they get blocked, when [03:51:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13863s) there's something that changes the plan [03:51:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13864s) overall. [03:51:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13867s) But the work stream doesn't just stop [03:51:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13868s) there because it takes on more on its [03:51:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13871s) own and it learns from your habits. At [03:51:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13873s) some point the orchestrator may ask you [03:51:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13874s) how you decide what to work on and it [03:51:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13876s) learns that maybe bugs are coming from [03:51:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13878s) two Slack channels. [03:51:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13881s) So here it can set up an intake and a [03:51:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13883s) triage listener. Every 30 minutes it can [03:51:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13885s) go in and pull in those reports from [03:51:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13887s) those channels, ddup them against you [03:51:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13890s) know existing linear issues or uh sentry [03:51:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13893s) sentry alerts that already exist and [03:51:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13894s) then weigh each of those against the [03:51:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13896s) workstream's goal. So a Slack complaint [03:51:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13899s) and a customer error report become one [03:51:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13901s) change. The output of triage is a spec [03:51:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13905s) that you review and uh you're capable of [03:51:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13908s) like weighing in on before any code gets [03:51:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13910s) written. Once you approve a spec, the [03:51:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13913s) same worker opens a PR, stewards it [03:51:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13916s) until it's merge ready uh and then make [03:51:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13918s) sure that you know review uh comments [03:52:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13920s) and CI are passing. [03:52:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13923s) So each one of those stages actually [03:52:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13925s) maps to an autonomy level in this [03:52:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13927s) system. At L1, the workstream posits for [03:52:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13930s) human review at the spec creation, at [03:52:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13933s) the PR, and before hitting merge. But [03:52:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13937s) you know, as you level up that th those [03:52:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13939s) levels of autonomy, you can get to L4 [03:52:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13941s) where you're automerging everything by [03:52:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13943s) default. The owner gets to set the [03:52:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13945s) level, but the confidence to raise it [03:52:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13947s) comes from watching the work. So when [03:52:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13950s) proposed changes are accurate or PRs [03:52:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13953s) consistently merge at the quality that [03:52:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13954s) you'd expect, that's actually your cue [03:52:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13956s) to move to the next checkpoint. And so [03:52:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13958s) that becomes the dial for you. You don't [03:52:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13960s) need to decide on day one whether you [03:52:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13962s) trust a coding agent to own, you know, [03:52:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13963s) your like CI health workstream off the [03:52:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13966s) bat. You just need to trust it to to [03:52:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13968s) like unlock the next checkpoint. [03:52:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13972s) So in a factory, an agent that starts [03:52:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13974s) from scratch repeats the same mistakes [03:52:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13976s) the last one already did. And so we [03:52:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13978s) built workstreams to have a shared file [03:53:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13981s) system. Agents are are encouraged to [03:53:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13984s) keep memory and version skills and so [03:53:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13987s) their PRs get better over time. They're [03:53:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13989s) also proactive and they go looking for [03:53:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13991s) context that they're missing. So for [03:53:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13993s) example, say that your work notices that [03:53:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13995s) other people are shipping related [03:53:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13997s) changes. It sets up an automation that [03:53:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=13999s) pulls in their PRs, [03:53:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14002s) analyzes what they've been working on [03:53:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14003s) and what comments they've been receiving [03:53:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14005s) in those PRs, and then incorporates them [03:53:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14007s) into their memory and skills. So, what [03:53:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14010s) one agent learns, others can use, and [03:53:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14012s) nothing starts cold. [03:53:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14016s) That folder is also how agents work [03:53:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14018s) together on a project. When the [03:53:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14020s) orchestrator spawns an agent, it's [03:53:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14022s) capable of telling, you know, each agent [03:53:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14024s) where does it where where does its [03:53:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14026s) result sort of belong inside of this [03:53:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14028s) shared folder. And so every agent can [03:53:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14031s) read and write to the same place. If you [03:53:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14034s) ask it a workstream why checkout got [03:53:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14036s) slower, for example, it can send dozens [03:53:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14039s) of agents out to track down all those [03:54:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14041s) different leads that you have. And then [03:54:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14042s) ultimately each one can write its [03:54:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14045s) findings to the folder and the [03:54:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14046s) orchestrator can merge them. [03:54:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14049s) Now you still need a barrier between [03:54:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14050s) implementation and merging. And for us [03:54:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14053s) that barrier is platform level [03:54:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14054s) validation. Every PR gets code review, [03:54:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14057s) security review, and dedicated QA [03:54:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14060s) against the live preview. And so once [03:54:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14062s) every review thread is resolved, a risk [03:54:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14064s) judge then goes in and decides whether [03:54:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14066s) it's safe to approve or not. The judge [03:54:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14068s) gets the exact if and a check out of the [03:54:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14070s) branch so it can like build and test it. [03:54:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14072s) Uh but ultimately it reads its policy [03:54:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14074s) from our mainline branch so that the [03:54:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14076s) change can't rewrite the rules that it's [03:54:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14078s) being judged by. [03:54:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14080s) Every week a droid minds the review [03:54:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14083s) comments and goes through figures out [03:54:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14085s) which one authors have actually acted on [03:54:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14087s) and then pulls them pulls those into [03:54:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14089s) like review guidelines so that the the [03:54:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14091s) loop sort of improves over time. We also [03:54:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14094s) have droids that every day run and [03:54:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14096s) analyze reverts, hot fixes, new sentry [03:54:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14099s) errors, for example, and traces them [03:55:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14101s) back to the PRs that that caused them. [03:55:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14103s) And so that also is a signal into this [03:55:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14105s) same loop. [03:55:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14107s) What we're really hell climbing on here [03:55:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14109s) is the regretfree merge rate, the share [03:55:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14112s) of changes that merge and never need a [03:55:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14114s) revert or a follow-up fix. We measure it [03:55:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14117s) for droid ridden and humanridden PRs [03:55:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14119s) side by side. And it's the same way that [03:55:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14121s) we let like self-driving cars onto the [03:55:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14122s) road. We didn't just give them the green [03:55:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14124s) light. We only uh did so once they [03:55:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14127s) crashed less often than human-driven [03:55:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14129s) ones did. And so once AI written changes [03:55:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14132s) break things less often than human ones [03:55:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14134s) and you can actually measure that then [03:55:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14136s) you can let them merge and that's the [03:55:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14138s) bar for turning a workstream to L4. [03:55:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14141s) Workstream is now like the platform that [03:55:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14143s) we build new features on top of. So [03:55:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14145s) migrate is the first one. It sort of [03:55:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14147s) fulfills a niche in our product um like [03:55:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14150s) framework for for what missions used to [03:55:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14153s) be this like dedicated oneshot migration [03:55:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14155s) experience and continuous remediation is [03:55:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14157s) the second where it follows that [03:55:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14159s) circular shape that workstreams are best [03:56:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14161s) suited for because there the work never [03:56:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14162s) ends. [03:56:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14166s) We run factory uh on top of workstreams [03:56:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14168s) ourselves and so this is what it [03:56:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14170s) actually looks like in production. In a [03:56:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14172s) weekend this past October, we shipped [03:56:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14174s) about 800 poll requests in our dev [03:56:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14176s) branch and Droid wrote 99% of them and [03:56:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14179s) reviewed all of them. Now, work landed [03:56:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14181s) in one of two ways. It was either a [03:56:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14183s) human picking up a signal manually uh or [03:56:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14186s) increasingly nowadays, it's a workstream [03:56:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14188s) picking it up uh automatically triaging [03:56:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14191s) it against the goal of the workstream [03:56:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14193s) and then ultimately writing a spec and [03:56:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14195s) merging it. [03:56:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14197s) 52% of PRs that we ship right now are [03:56:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14200s) coming from workshops themselves and [03:56:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14201s) some of these are operating on full [03:56:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14203s) autonomy. So for example, the one that [03:56:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14206s) uh fixes bug reports from a dedicated [03:56:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14208s) Slack channel merges 53 PRs per week. [03:56:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14210s) Another one is our flaky test work [03:56:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14212s) stream which merges about 40. [03:56:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14216s) Fully autonomous PRs aren't really the [03:56:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14219s) only thing worth tracking though. It [03:57:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14220s) what matters a lot more is the [03:57:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14222s) progression. how much of your work flows [03:57:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14224s) through work streams and how many [03:57:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14225s) checkpoints does each one still need. As [03:57:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14228s) more work flows unrestricted, your [03:57:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14230s) engineers become the regulators of this [03:57:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14232s) system. They decide, you know, what [03:57:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14234s) signals to watch, uh what are the goals [03:57:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14236s) and the owners of those, what are the [03:57:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14238s) skills and review guidelines that you [03:57:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14239s) need in order to keep this code high [03:57:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14241s) quality and how much autonomy each [03:57:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14242s) workstream gets. [03:57:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14244s) And so there's just one question that [03:57:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14246s) tells you, you know, whether whether you [03:57:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14248s) have a software factory or faster [03:57:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14250s) engineers, and that is, can you name and [03:57:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14252s) track your signals? Every team already [03:57:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14254s) has them. You have bug reports in Slack, [03:57:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14257s) you know, failing CI jobs, error spikes, [03:57:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14260s) tickets that nobody ever picks up, but [03:57:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14262s) most of them wait on a person to notice. [03:57:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14264s) And so my ask to all of you is to write [03:57:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14267s) them down and then pick the one that you [03:57:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14268s) waste the most time on. give it to [03:57:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14271s) workstream on L1 and then set up our [03:57:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14273s) validation experience and then you're [03:57:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14274s) well on your way to building software [03:57:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14276s) that improves itself with all the [03:57:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14278s) signals that you choose to track. Thanks [03:58:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14281s) everyone and I'll be around to answer [03:58:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=14282s) any questions today. [applause]