1 01:01:22,840 --> 01:01:28,780 All right. Hi, everyone. Excited to be here. And today I'm going to chat about the evolution 2 01:01:28,780 --> 01:01:35,140 of software engineering and how it's becoming factory engineering. So excited to do this. 3 01:01:35,360 --> 01:01:40,660 First, a little bit about me. I've been an engineer for a long time. I was a principal 4 01:01:40,660 --> 01:01:46,420 engineer at Google. I used to lead engineering on the Google Docs suite, helped scale that 5 01:01:46,420 --> 01:01:47,980 product to hundreds of millions of users. 6 01:01:48,440 --> 01:01:51,540 I'm currently the founder and CEO of Warp. 7 01:01:51,760 --> 01:01:56,160 Warp builds a platform for companies to build their own 8 01:01:56,160 --> 01:01:57,560 internal software factories. 9 01:01:57,820 --> 01:01:59,500 I'm going to show a little bit of that today. 10 01:02:00,700 --> 01:02:04,940 Warp is also well known for our open source agentic terminal. 11 01:02:05,160 --> 01:02:07,060 It has about 65,000 GitHub stars. 12 01:02:07,960 --> 01:02:10,280 It's used by almost a million developers. 13 01:02:10,840 --> 01:02:13,540 Our mission has always been helping developers ship better 14 01:02:13,540 --> 01:02:18,660 software more quickly and it's cool to evolve what we're doing as the world changes around 15 01:02:18,660 --> 01:02:19,000 us. 16 01:02:20,780 --> 01:02:23,640 is where I think we're still mostly at. 17 01:02:24,280 --> 01:02:31,740 Most developers that I talk to, most platform teams, they're working with individuals who 18 01:02:31,740 --> 01:02:34,520 are running their own coding agents locally. 19 01:02:34,720 --> 01:02:37,820 Maybe they're running multiple of them at a time. 20 01:02:38,160 --> 01:02:39,560 It could be through their terminal. 21 01:02:39,560 --> 01:02:42,240 It could be through an agentic development environment. 22 01:02:43,540 --> 01:02:45,100 Just a show of hands quickly, actually. 23 01:02:45,200 --> 01:02:48,540 How many people in here are mostly working in this fashion? 24 01:02:48,740 --> 01:02:50,120 Local, multi-threading. 25 01:02:51,700 --> 01:02:54,580 Okay, so still most people, but actually not everyone. 26 01:02:56,020 --> 01:02:57,920 I think that this is going to change, 27 01:02:57,980 --> 01:02:59,580 and it's starting to change already. 28 01:03:01,360 --> 01:03:03,580 The reasons are, there's a few of them. 29 01:03:03,720 --> 01:03:07,040 So one is, like, if you are doing local interactive prompting 30 01:03:07,040 --> 01:03:09,920 is your main way of working with coding agents. 31 01:03:10,880 --> 01:03:14,340 You're kind of limited by human throughput, really. 32 01:03:14,500 --> 01:03:18,340 You cannot automate if every agent needs 33 01:03:18,340 --> 01:03:19,440 to be interactively prompted. 34 01:03:19,880 --> 01:03:22,300 There's a second reason, too, that I hear a lot 35 01:03:22,300 --> 01:03:26,400 from platform teams and from engineering leaders. 36 01:03:26,700 --> 01:03:29,900 This mechanism of everyone choosing their own local setup, 37 01:03:30,440 --> 01:03:32,440 their own models, their own skills, 38 01:03:33,200 --> 01:03:35,860 makes it very hard to actually make sure 39 01:03:35,860 --> 01:03:40,360 that you're using these agents efficiently from a cost perspective and a productivity 40 01:03:40,360 --> 01:03:46,560 perspective. And then finally, there's a big security problem and governance problem when 41 01:03:46,560 --> 01:03:51,800 everyone is setting up and running their own local agents. And so the thing that is coming, 42 01:03:52,640 --> 01:03:57,000 I loosely call a software factory. That means different things to different people. 43 01:03:59,420 --> 01:04:05,400 I'm curious for people in this room, how many folks are currently trying to build or building 44 01:04:05,400 --> 01:04:07,620 or using their own internal software factories? 45 01:04:07,860 --> 01:04:08,420 Show of hands. 46 01:04:09,680 --> 01:04:12,380 Okay, so I would say that's almost more 47 01:04:12,380 --> 01:04:13,760 than the interactive agents at this point. 48 01:04:13,840 --> 01:04:14,460 That's super cool. 49 01:04:16,480 --> 01:04:20,220 So I'm gonna define what I think of as a software factory, 50 01:04:20,360 --> 01:04:21,740 but I'll start by saying what I think 51 01:04:21,740 --> 01:04:23,780 a software factory is not. 52 01:04:24,940 --> 01:04:28,260 I think it's not just a loose set of automations. 53 01:04:29,100 --> 01:04:30,980 I don't think of a software factory 54 01:04:30,980 --> 01:04:33,260 as a collection of AI teammates. 55 01:04:34,100 --> 01:04:38,860 I don't think it's a shared set of skills and MCPs, and I also don't think software 56 01:04:38,860 --> 01:04:40,780 factories are personal agent swarms. 57 01:04:40,960 --> 01:04:47,140 I think these are all useful primitives, but they're not the end state that we should be 58 01:04:47,140 --> 01:04:47,680 shooting for. 59 01:04:48,540 --> 01:04:55,600 And so to me, the way that I define software factory is very specific, and I'll unpack 60 01:04:55,600 --> 01:05:02,420 this, but I define it as a code-defined closed-loop system for continuously measuring and improving 61 01:05:02,420 --> 01:05:03,900 cloud coding agents. 62 01:05:04,100 --> 01:05:06,020 So that's a mouthful, but let me unpack it. 63 01:05:06,940 --> 01:05:12,500 So if you want to build a software factory and really scale it, I believe the first thing 64 01:05:12,500 --> 01:05:17,100 you need to do is fully encode it in a declarative language. 65 01:05:17,120 --> 01:05:22,740 So you need to say, okay, these are all of the coding agents that I want to run. 66 01:05:22,940 --> 01:05:24,140 These are how they're triggered. 67 01:05:24,620 --> 01:05:30,820 When work comes into this factory, this is the workflow from triage to spec, whatever 68 01:05:30,820 --> 01:05:31,580 your workflow is. 69 01:05:31,580 --> 01:05:34,560 you need to specify what MCPs are available, 70 01:05:34,880 --> 01:05:36,080 where your agents run, 71 01:05:36,340 --> 01:05:38,320 really as much as you can 72 01:05:38,320 --> 01:05:42,420 in a Terraform-style declarative, you know, 73 01:05:42,740 --> 01:05:44,700 set of code files that define your factory. 74 01:05:44,840 --> 01:05:46,440 And I'll talk about why this matters in a bit. 75 01:05:47,520 --> 01:05:48,920 The second part of it 76 01:05:48,920 --> 01:05:51,300 is a continuous closed loop of improvement. 77 01:05:51,880 --> 01:05:54,820 And that closed loop can look something like this. 78 01:05:54,820 --> 01:05:55,960 This is the typical one. 79 01:05:56,060 --> 01:05:57,940 This is just really the SDLC, 80 01:05:58,900 --> 01:06:01,280 where you have ideas that come in from people 81 01:06:02,460 --> 01:06:04,600 They could come in from automated signals 82 01:06:04,600 --> 01:06:07,080 And then your agents do some amount of work 83 01:06:08,040 --> 01:06:11,060 Taking that issue through various stages 84 01:06:11,060 --> 01:06:15,220 From triage, implementation, verification, monitoring, and so on 85 01:06:15,220 --> 01:06:18,680 The important piece here is really this closed loop concept 86 01:06:18,680 --> 01:06:21,700 Where the more that your agents run 87 01:06:21,700 --> 01:06:24,660 The more efficient you can make this loop 88 01:06:24,660 --> 01:06:26,600 And I'll talk about how you do that in a second 89 01:06:27,480 --> 01:06:29,620 The third really key aspect 90 01:06:29,620 --> 01:06:31,480 if you're building a software factory 91 01:06:31,480 --> 01:06:33,380 is that everything needs to be measurable. 92 01:06:34,540 --> 01:06:36,780 If you're just going on vibes 93 01:06:36,780 --> 01:06:38,820 in terms of how you deploy coding agents, 94 01:06:39,100 --> 01:06:41,520 you can't be confident that you're actually 95 01:06:41,520 --> 01:06:45,120 getting the best, highest quality outputs 96 01:06:45,120 --> 01:06:46,520 at the best cost. 97 01:06:46,700 --> 01:06:48,580 You can't be confident you're maximizing throughput. 98 01:06:48,800 --> 01:06:50,660 So you really need a measurement mindset 99 01:06:50,660 --> 01:06:52,000 and measurement capabilities. 100 01:06:53,820 --> 01:06:55,280 so if there's one thing 101 01:06:55,280 --> 01:06:59,000 that I'm hoping people walk away from this talk with, 102 01:06:59,000 --> 01:07:03,200 it's the idea that the way to work with coding agents 103 01:07:03,200 --> 01:07:04,580 and the way to think about them 104 01:07:04,580 --> 01:07:09,480 is that there's a really cool fundamental new engineering problem, 105 01:07:09,880 --> 01:07:13,860 which is how do you take this magic new primitive 106 01:07:13,860 --> 01:07:15,920 that we all have, that we've got in the last year, 107 01:07:16,180 --> 01:07:17,920 that you have these intelligent agents, 108 01:07:18,100 --> 01:07:21,580 and how do you engineer a system around them 109 01:07:21,580 --> 01:07:23,600 to use them most effectively? 110 01:07:23,960 --> 01:07:26,920 To me, it's not about growing a garden 111 01:07:26,920 --> 01:07:28,220 or following a recipe. 112 01:07:29,200 --> 01:07:30,580 we're all engineers. 113 01:07:30,900 --> 01:07:32,340 We've spent our careers 114 01:07:32,340 --> 01:07:34,600 learning how to solve problems from an engineering mindset. 115 01:07:35,040 --> 01:07:36,940 And when you think of coding agents 116 01:07:36,940 --> 01:07:38,980 as a new engineering primitive, 117 01:07:39,760 --> 01:07:41,840 then you naturally want to, 118 01:07:41,980 --> 01:07:43,580 you'll naturally start to build 119 01:07:43,580 --> 01:07:44,880 a certain way of using them. 120 01:07:45,040 --> 01:07:45,880 That's the mindset. 121 01:07:47,400 --> 01:07:49,280 So yeah, just to unpack that a bit, 122 01:07:49,400 --> 01:07:51,640 so there's continual measurement 123 01:07:51,640 --> 01:07:52,780 and continual learning, 124 01:07:53,460 --> 01:07:55,760 but there's also a new set of tools. 125 01:07:56,600 --> 01:08:02,940 And so when I was learning how to become a better engineer at Google, you know, I really 126 01:08:02,940 --> 01:08:07,720 had to get good at using things like debuggers and profilers and Chrome DevTools. 127 01:08:07,980 --> 01:08:12,400 And basically, there were all of these tools of the trade for building software by hand 128 01:08:12,400 --> 01:08:19,500 that maybe are not as important now, but they really separated the best engineers from the 129 01:08:19,500 --> 01:08:20,140 okay engineers. 130 01:08:20,140 --> 01:08:26,340 And in the new world of factory engineering, there's a new set of tools that everyone should 131 01:08:26,340 --> 01:08:27,680 be trying to master. 132 01:08:27,820 --> 01:08:30,260 And so those are things like evals. 133 01:08:30,880 --> 01:08:34,520 It's can you set up self-improvement loops that work? 134 01:08:34,860 --> 01:08:36,080 Can you write benchmarks? 135 01:08:36,740 --> 01:08:38,400 Can you A-B test your factory? 136 01:08:39,740 --> 01:08:41,940 And that's a really important skill set. 137 01:08:42,120 --> 01:08:47,820 And mastering that skill set is something that I think of as factory engineering. 138 01:08:48,760 --> 01:08:52,720 Now, that doesn't mean that that's the only kind of engineering you're going to be doing. 139 01:08:53,040 --> 01:08:59,040 You're still going to be doing product engineering, meaning you as an engineer should still feel 140 01:08:59,040 --> 01:09:05,460 responsible for, you know, does the product you ship actually solve a real user or customer 141 01:09:05,460 --> 01:09:05,940 problem? 142 01:09:06,600 --> 01:09:11,700 But the best engineers, and I would say like every engineer really, is going to need to 143 01:09:11,700 --> 01:09:15,920 learn this new tool set of how do you improve the system, not just build the product. 144 01:09:17,480 --> 01:09:19,500 And so what does that look like? 145 01:09:20,280 --> 01:09:25,840 If you think from first principles about how do you engineer an agentic system, I think 146 01:09:25,840 --> 01:09:27,380 you come up with something like this. 147 01:09:27,460 --> 01:09:32,800 It doesn't have to be exactly this, but it's kind of an infrastructure stack that you're 148 01:09:32,800 --> 01:09:34,220 going to want to use to engineer your agents. 149 01:09:34,520 --> 01:09:38,520 And so at the bottom of this is what I call factories is code. 150 01:09:38,720 --> 01:09:39,560 I talked about this earlier. 151 01:09:40,260 --> 01:09:45,300 It's defining all of your coding agent workflows in a code format. 152 01:09:45,300 --> 01:09:50,920 And the value of that is that you get the ability to branch off it and test different factory configurations 153 01:09:51,520 --> 01:09:53,520 you can roll back and 154 01:09:54,060 --> 01:09:57,940 then crucially it opens up the way for self-improvement 155 01:09:57,940 --> 01:09:59,940 Which is the ability for coding 156 01:10:00,000 --> 01:10:00,780 agents themselves 157 01:10:00,780 --> 01:10:06,100 to suggest changes to how your factory works to get that continual loop going. 158 01:10:07,000 --> 01:10:13,020 Above that is a layer of data and context. This is also crucial. So if you are running 159 01:10:13,020 --> 01:10:20,120 coding agents at scale in an organization, you really must be recording everything that 160 01:10:20,120 --> 01:10:24,680 those agents are producing. So all of their conversations, all of their artifacts, the 161 01:10:24,680 --> 01:10:29,520 videos that they make, the screenshots. And the reason for doing that is that's the raw 162 01:10:29,520 --> 01:10:36,360 material that you will use to improve the way your factory works over time. On top of 163 01:10:36,360 --> 01:10:42,280 that is a compute layer. So you need to have sandboxing. Sandboxing increases security. 164 01:10:42,340 --> 01:10:46,760 It makes your factory more repeatable. Your sandboxing should be probably multi-platform. 165 01:10:47,680 --> 01:10:52,560 On top of that is the inference layer. The way that I would think about inference and 166 01:10:52,560 --> 01:10:57,360 models in a factory is they are a degree of freedom to test. 167 01:10:57,720 --> 01:11:02,700 And so your factory, however you build it, should support multiple models. 168 01:11:03,860 --> 01:11:08,900 It should also support multiple coding agents because, you know, Cloud Code might be better 169 01:11:08,900 --> 01:11:09,460 for some things. 170 01:11:09,540 --> 01:11:13,620 Codex, Pi, Warp, these harnesses all have different properties. 171 01:11:13,820 --> 01:11:17,860 You might want to write your own coding agent harness that has some of your organizational 172 01:11:17,860 --> 01:11:18,860 context baked in. 173 01:11:19,000 --> 01:11:20,740 So your platform should support that. 174 01:11:21,200 --> 01:11:23,540 On top of that, you need an orchestration layer. 175 01:11:23,680 --> 01:11:26,320 So this is what actually moves work through the factory. 176 01:11:26,640 --> 01:11:29,200 In some ways, the least interesting part, but you need that. 177 01:11:29,500 --> 01:11:34,140 And then finally, your factory should be all API and MCP first. 178 01:11:34,260 --> 01:11:39,740 So that it can be used not just by, you know, people, but it can be used by other coding 179 01:11:39,740 --> 01:11:40,180 agents. 180 01:11:40,860 --> 01:11:43,200 And it can go to where the work is being done. 181 01:11:43,380 --> 01:11:49,360 So your factory should integrate into things like Slack, Linear, Jira, GitHub, and so on. 182 01:11:50,660 --> 01:11:55,180 I'm going to give a quick demo of what some of these tools look like in the context of 183 01:11:55,180 --> 01:12:01,340 warp factories, but this stuff I'm going to show I think applies also if you're building 184 01:12:01,340 --> 01:12:01,780 your own. 185 01:12:04,320 --> 01:12:05,840 So let me start here. 186 01:12:06,040 --> 01:12:11,920 So at the foundation I mentioned earlier is factories in code. 187 01:12:12,180 --> 01:12:14,240 And so I'm going to show some code. 188 01:12:14,360 --> 01:12:18,240 It's not going to be that interesting, but if you're familiar with something like Terraform, 189 01:12:18,240 --> 01:12:19,660 you will get what's going on here. 190 01:12:20,320 --> 01:12:23,960 You basically want a declarative way of saying, 191 01:12:24,120 --> 01:12:25,380 this is my factory. 192 01:12:25,840 --> 01:12:28,580 My factory has certain policies. 193 01:12:28,580 --> 01:12:30,240 It accesses certain repos. 194 01:12:30,640 --> 01:12:32,280 It has certain triggers. 195 01:12:32,700 --> 01:12:35,860 It has a sort of blueprint that defines the workflow 196 01:12:35,860 --> 01:12:37,160 of work going through it. 197 01:12:37,360 --> 01:12:39,080 And then if you zoom out, 198 01:12:39,440 --> 01:12:41,860 all of these other things that are part of the factory 199 01:12:41,860 --> 01:12:43,920 also have code definitions. 200 01:12:44,800 --> 01:12:47,040 The second thing, moving up a layer, 201 01:12:47,040 --> 01:12:56,640 is the idea that everything that you do in the factory 202 01:12:58,080 --> 01:13:03,220 should be tracked across your entire team. 203 01:13:03,420 --> 01:13:05,900 And so think of what I'm showing here. 204 01:13:06,120 --> 01:13:09,120 This is a sort of Google Docs style live view 205 01:13:09,120 --> 01:13:11,620 of what every coding agent on Warp's team 206 01:13:11,620 --> 01:13:15,100 across all of our engineers are doing at all times. 207 01:13:15,100 --> 01:13:16,860 And so I could click it and I could see, 208 01:13:17,040 --> 01:13:19,620 okay, I don't know what Maggie is working on, 209 01:13:19,700 --> 01:13:21,140 but I know I could look at it. 210 01:13:21,220 --> 01:13:23,300 She's implementing Opus 5.5 on the high plan. 211 01:13:23,680 --> 01:13:24,860 She's doing it from Slack. 212 01:13:25,280 --> 01:13:26,360 It has this artifact. 213 01:13:26,700 --> 01:13:29,220 I could look in and see the actual conversation here. 214 01:13:29,280 --> 01:13:31,460 I could see what it's generated. 215 01:13:31,700 --> 01:13:34,000 And so I could see this in real time, which is cool. 216 01:13:34,300 --> 01:13:35,720 But the really important thing 217 01:13:35,720 --> 01:13:37,380 is that you gather all this information 218 01:13:37,380 --> 01:13:41,840 so that it can be used for improving the factory. 219 01:13:42,120 --> 01:13:45,260 And so that brings me to the third thing, 220 01:13:45,360 --> 01:13:46,440 moving up the stack here. 221 01:13:46,440 --> 01:13:48,920 is once you have all of this information 222 01:13:48,920 --> 01:13:50,600 that's being stored centrally, 223 01:13:51,100 --> 01:13:53,240 you get the ability very easily, 224 01:13:53,480 --> 01:13:56,020 without doing any, like, install stuff 225 01:13:56,020 --> 01:13:58,200 on everybody's laptop, anything like that, 226 01:13:58,440 --> 01:14:02,280 to understand how these coding agents are working. 227 01:14:02,640 --> 01:14:04,340 And you can think of that in terms of, like, 228 01:14:04,440 --> 01:14:07,800 traditional output metrics, like your DORA metrics, 229 01:14:07,960 --> 01:14:09,280 like, okay, cycle time. 230 01:14:09,500 --> 01:14:11,140 But you can also think of that in terms of, like, 231 01:14:11,180 --> 01:14:14,560 how automated am I, like, how many human interactions per PR, 232 01:14:15,060 --> 01:14:16,040 what's your throughput. 233 01:14:16,440 --> 01:14:20,900 you should be able to see data very clearly here 234 01:14:20,900 --> 01:14:22,560 on, like, what's your cost. 235 01:14:22,880 --> 01:14:26,020 Again, surfacing this, if you don't have this data, 236 01:14:26,960 --> 01:14:29,040 you're kind of flying, uh-oh. 237 01:14:30,440 --> 01:14:31,940 You're kind of flying blind. 238 01:14:32,340 --> 01:14:33,680 So this is really important. 239 01:14:33,800 --> 01:14:37,180 And then there's this next layer of data, 240 01:14:37,220 --> 01:14:39,260 which is also really important that I mentioned earlier, 241 01:14:39,420 --> 01:14:42,260 which is the ability to set up online evals 242 01:14:42,260 --> 01:14:44,640 that run across all your coding agent sessions. 243 01:14:44,640 --> 01:14:48,600 And so what I mean by this, an eval is just like, 244 01:14:48,760 --> 01:14:50,960 as these coding agent sessions are running 245 01:14:50,960 --> 01:14:52,120 and they're recording their output, 246 01:14:52,380 --> 01:14:56,320 you want the ability to go back and have another agent 247 01:14:56,320 --> 01:15:00,260 look at them and score them on some dimension 248 01:15:00,260 --> 01:15:02,340 that might indicate how well they're doing. 249 01:15:02,640 --> 01:15:04,260 And to make that more concrete, 250 01:15:04,540 --> 01:15:08,160 so it's like we have an eval, a score here that we use 251 01:15:08,160 --> 01:15:10,380 that is looking for redundant tests. 252 01:15:10,480 --> 01:15:11,140 This is a good example. 253 01:15:11,320 --> 01:15:14,340 And so one common failure mode of coding agents 254 01:15:14,340 --> 01:15:17,240 is that sometimes they will write tests that are unnecessary. 255 01:15:17,860 --> 01:15:20,700 We want to catch that, and that's what this eval does. 256 01:15:21,020 --> 01:15:23,000 We want the ability then to see, 257 01:15:23,140 --> 01:15:26,600 okay, when it's writing too many tests, 258 01:15:26,620 --> 01:15:28,480 why is it doing that? 259 01:15:28,580 --> 01:15:31,680 And is there a mechanism that we could change 260 01:15:31,680 --> 01:15:33,420 in the factory for improving that? 261 01:15:33,660 --> 01:15:36,420 And so, again, just to kind of recap here, 262 01:15:36,960 --> 01:15:39,840 you want a way of recording all the runs. 263 01:15:40,720 --> 01:15:42,780 When these runs run, I didn't show this, 264 01:15:42,780 --> 01:15:47,840 but all of the runs are, you know, you want a way of specifying 265 01:15:49,500 --> 01:15:52,180 what is the sort of, like, actual loop that they're running through. 266 01:15:52,460 --> 01:15:54,640 You want to be able to see where they are in that loop. 267 01:15:55,340 --> 01:15:59,160 As the runs work, you want to be able to gather their data. 268 01:15:59,480 --> 01:16:03,080 Once you gather their data, you want to be able to set up these online evals. 269 01:16:03,500 --> 01:16:05,800 And then these online evals give you a sense of, like, 270 01:16:05,860 --> 01:16:07,780 okay, where are there opportunities for improvement? 271 01:16:08,520 --> 01:16:10,040 Once you have that in place, 272 01:16:10,040 --> 01:16:12,920 You can do something that we call self-improvement 273 01:16:12,920 --> 01:16:15,160 Which is rather than having humans 274 01:16:15,160 --> 01:16:16,820 Look at the outputs of these scorers 275 01:16:16,820 --> 01:16:18,360 You have agents look at the output 276 01:16:18,360 --> 01:16:21,800 And so the way to think about self-improvement 277 01:16:21,800 --> 01:16:24,220 Is like it looks at the score output 278 01:16:24,220 --> 01:16:28,400 It says okay we saw some number of failed runs 279 01:16:30,340 --> 01:16:32,260 Let's try to look for patterns 280 01:16:32,260 --> 01:16:33,900 In those failures 281 01:16:33,900 --> 01:16:37,040 And suggest a possible solution 282 01:16:37,040 --> 01:16:46,140 And then the form that that solution takes is as a PR against your factory definition. 283 01:16:46,440 --> 01:16:51,840 And so this is like the awesome thing about having your factory defined as code and being 284 01:16:51,840 --> 01:16:56,600 able to use coding agents is that you can then have the coding agents diagnose problems 285 01:16:57,160 --> 01:17:00,760 with how they're working internally and suggest fixes to them. 286 01:17:01,880 --> 01:17:06,380 The final tool that I'm going to point out that I mentioned earlier is the ability to 287 01:17:06,380 --> 01:17:13,040 create and run benchmarks. This is also essential. So if you're following this course online, 288 01:17:13,120 --> 01:17:16,740 you'll often see things like, okay, this is the best model, this is the new best skill, 289 01:17:16,880 --> 01:17:22,220 this is the best harness. The only way to answer those questions for real is to test 290 01:17:22,220 --> 01:17:27,120 on your own data. And so you should have in place some mechanism where you can basically 291 01:17:27,120 --> 01:17:30,440 create a benchmark set out of your real data 292 01:17:31,520 --> 01:17:35,500 and then replay those tasks. 293 01:17:35,800 --> 01:17:37,420 Oops, I'm on the wrong one here. 294 01:17:37,660 --> 01:17:41,120 And replay those tasks against different configurations 295 01:17:41,120 --> 01:17:44,500 to see how they would have done 296 01:17:44,500 --> 01:17:47,740 if you had varied some model or varied some harness 297 01:17:47,740 --> 01:17:49,600 or varied your factory configuration. 298 01:17:49,900 --> 01:17:51,640 So just to make this clear, 299 01:17:54,820 --> 01:18:02,520 this is something that looks like what you might get when a new model drops like in sweet bench or 300 01:18:02,520 --> 01:18:08,200 terminal bench it's showing the trade-offs of different models here in terms of cost and 301 01:18:08,200 --> 01:18:14,000 quality you want to be in this upper left-hand quadrant but it's on your own actual data and you 302 01:18:14,000 --> 01:18:18,780 can you can test not just models but also you want to be able to test hey if i got rid of these skills 303 01:18:18,780 --> 01:18:20,540 is does my factory perform better or worse? 304 01:18:20,860 --> 01:18:22,860 So these are some of the tools. 305 01:18:24,440 --> 01:18:27,140 And just to return to this big concept, 306 01:18:27,320 --> 01:18:28,820 like this has worked well for us. 307 01:18:28,980 --> 01:18:31,900 We have managed to use these tools 308 01:18:31,900 --> 01:18:33,540 to drastically improve our throughput 309 01:18:33,540 --> 01:18:35,280 and lower our cost. 310 01:18:35,940 --> 01:18:38,420 We are just constantly doing this 311 01:18:38,420 --> 01:18:40,100 because this cost really matters to us, 312 01:18:40,280 --> 01:18:42,140 where we're getting a performance baseline, 313 01:18:42,460 --> 01:18:45,040 we're scoring, we're benchmarking, we're testing, 314 01:18:45,340 --> 01:18:47,120 and we're doing factory engineering. 315 01:18:47,580 --> 01:18:50,660 And so that brings me back to this last point 316 01:18:50,660 --> 01:18:52,000 that I just want to emphasize again. 317 01:18:53,180 --> 01:18:54,360 Engineer your agents. 318 01:18:54,960 --> 01:18:56,460 It's a new engineering primitive. 319 01:18:56,660 --> 01:18:57,260 We're all engineers. 320 01:18:57,840 --> 01:18:59,700 Think about how you can get the most out of them 321 01:18:59,700 --> 01:19:01,020 and build the system for that. 322 01:19:01,960 --> 01:19:03,320 I think this is very exciting. 323 01:19:04,300 --> 01:19:07,040 I know there's a lot of, like, angst 324 01:19:07,040 --> 01:19:08,940 around what it means to be an engineer right now. 325 01:19:09,420 --> 01:19:11,200 I know we're not writing code, 326 01:19:11,200 --> 01:19:13,520 but I'm still using all the engineering skills 327 01:19:13,520 --> 01:19:15,200 that I spent my whole career learning, 328 01:19:15,200 --> 01:19:17,400 And so to me it's still a really exciting time 329 01:19:18,500 --> 01:19:19,800 If you're interested in more factories 330 01:19:19,800 --> 01:19:21,220 You can check it out here 331 01:19:21,660 --> 01:19:23,040 Thank you all for coming to the talk