# Software Engineering Is Becoming Factory Engineering

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[01:01:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3682s) All right. Hi, everyone. Excited to be here. And today I'm going to chat about the evolution

[01:01:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3688s) of software engineering and how it's becoming factory engineering. So excited to do this.

[01:01:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3695s) First, a little bit about me. I've been an engineer for a long time. I was a principal

[01:01:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3700s) engineer at Google. I used to lead engineering on the Google Docs suite, helped scale that

[01:01:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3706s) product to hundreds of millions of users.

[01:01:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3708s) I'm currently the founder and CEO of Warp.

[01:01:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3711s) Warp builds a platform for companies to build their own

[01:01:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3716s) internal software factories.

[01:01:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3717s) I'm going to show a little bit of that today.

[01:02:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3720s) Warp is also well known for our open source agentic terminal.

[01:02:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3725s) It has about 65,000 GitHub stars.

[01:02:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3727s) It's used by almost a million developers.

[01:02:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3730s) Our mission has always been helping developers ship better

[01:02:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3733s) software more quickly and it's cool to evolve what we're doing as the world changes around

[01:02:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3738s) us.

[01:02:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3740s) is where I think we're still mostly at.

[01:02:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3744s) Most developers that I talk to, most platform teams, they're working with individuals who

[01:02:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3751s) are running their own coding agents locally.

[01:02:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3754s) Maybe they're running multiple of them at a time.

[01:02:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3758s) It could be through their terminal.

[01:02:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3759s) It could be through an agentic development environment.

[01:02:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3763s) Just a show of hands quickly, actually.

[01:02:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3765s) How many people in here are mostly working in this fashion?

[01:02:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3768s) Local, multi-threading.

[01:02:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3771s) Okay, so still most people, but actually not everyone.

[01:02:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3776s) I think that this is going to change,

[01:02:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3777s) and it's starting to change already.

[01:03:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3781s) The reasons are, there's a few of them.

[01:03:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3783s) So one is, like, if you are doing local interactive prompting

[01:03:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3787s) is your main way of working with coding agents.

[01:03:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3790s) You're kind of limited by human throughput, really.

[01:03:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3794s) You cannot automate if every agent needs

[01:03:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3798s) to be interactively prompted.

[01:03:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3799s) There's a second reason, too, that I hear a lot

[01:03:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3802s) from platform teams and from engineering leaders.

[01:03:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3806s) This mechanism of everyone choosing their own local setup,

[01:03:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3810s) their own models, their own skills,

[01:03:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3813s) makes it very hard to actually make sure

[01:03:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3815s) that you're using these agents efficiently from a cost perspective and a productivity

[01:03:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3820s) perspective. And then finally, there's a big security problem and governance problem when

[01:03:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3826s) everyone is setting up and running their own local agents. And so the thing that is coming,

[01:03:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3832s) I loosely call a software factory. That means different things to different people.

[01:03:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3839s) I'm curious for people in this room, how many folks are currently trying to build or building

[01:04:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3845s) or using their own internal software factories?

[01:04:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3847s) Show of hands.

[01:04:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3849s) Okay, so I would say that's almost more

[01:04:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3852s) than the interactive agents at this point.

[01:04:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3853s) That's super cool.

[01:04:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3856s) So I'm gonna define what I think of as a software factory,

[01:04:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3860s) but I'll start by saying what I think

[01:04:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3861s) a software factory is not.

[01:04:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3864s) I think it's not just a loose set of automations.

[01:04:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3869s) I don't think of a software factory

[01:04:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3870s) as a collection of AI teammates.

[01:04:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3874s) I don't think it's a shared set of skills and MCPs, and I also don't think software

[01:04:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3878s) factories are personal agent swarms.

[01:04:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3880s) I think these are all useful primitives, but they're not the end state that we should be

[01:04:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3887s) shooting for.

[01:04:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3888s) And so to me, the way that I define software factory is very specific, and I'll unpack

[01:04:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3895s) this, but I define it as a code-defined closed-loop system for continuously measuring and improving

[01:05:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3902s) cloud coding agents.

[01:05:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3904s) So that's a mouthful, but let me unpack it.

[01:05:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3906s) So if you want to build a software factory and really scale it, I believe the first thing

[01:05:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3912s) you need to do is fully encode it in a declarative language.

[01:05:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3917s) So you need to say, okay, these are all of the coding agents that I want to run.

[01:05:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3922s) These are how they're triggered.

[01:05:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3924s) When work comes into this factory, this is the workflow from triage to spec, whatever

[01:05:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3930s) your workflow is.

[01:05:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3931s) you need to specify what MCPs are available,

[01:05:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3934s) where your agents run,

[01:05:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3936s) really as much as you can

[01:05:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3938s) in a Terraform-style declarative, you know,

[01:05:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3942s) set of code files that define your factory.

[01:05:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3944s) And I'll talk about why this matters in a bit.

[01:05:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3947s) The second part of it

[01:05:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3948s) is a continuous closed loop of improvement.

[01:05:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3951s) And that closed loop can look something like this.

[01:05:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3954s) This is the typical one.

[01:05:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3956s) This is just really the SDLC,

[01:05:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3958s) where you have ideas that come in from people

[01:06:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3962s) They could come in from automated signals

[01:06:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3964s) And then your agents do some amount of work

[01:06:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3968s) Taking that issue through various stages

[01:06:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3971s) From triage, implementation, verification, monitoring, and so on

[01:06:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3975s) The important piece here is really this closed loop concept

[01:06:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3978s) Where the more that your agents run

[01:06:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3981s) The more efficient you can make this loop

[01:06:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3984s) And I'll talk about how you do that in a second

[01:06:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3987s) The third really key aspect

[01:06:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3989s) if you're building a software factory

[01:06:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3991s) is that everything needs to be measurable.

[01:06:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3994s) If you're just going on vibes

[01:06:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3996s) in terms of how you deploy coding agents,

[01:06:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=3999s) you can't be confident that you're actually

[01:06:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4001s) getting the best, highest quality outputs

[01:06:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4005s) at the best cost.

[01:06:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4006s) You can't be confident you're maximizing throughput.

[01:06:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4008s) So you really need a measurement mindset

[01:06:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4010s) and measurement capabilities.

[01:06:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4013s) so if there's one thing

[01:06:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4015s) that I'm hoping people walk away from this talk with,

[01:06:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4019s) it's the idea that the way to work with coding agents

[01:07:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4023s) and the way to think about them

[01:07:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4024s) is that there's a really cool fundamental new engineering problem,

[01:07:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4029s) which is how do you take this magic new primitive

[01:07:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4033s) that we all have, that we've got in the last year,

[01:07:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4036s) that you have these intelligent agents,

[01:07:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4038s) and how do you engineer a system around them

[01:07:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4041s) to use them most effectively?

[01:07:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4043s) To me, it's not about growing a garden

[01:07:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4046s) or following a recipe.

[01:07:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4049s) we're all engineers.

[01:07:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4050s) We've spent our careers

[01:07:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4052s) learning how to solve problems from an engineering mindset.

[01:07:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4055s) And when you think of coding agents

[01:07:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4056s) as a new engineering primitive,

[01:07:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4059s) then you naturally want to,

[01:07:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4061s) you'll naturally start to build

[01:07:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4063s) a certain way of using them.

[01:07:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4065s) That's the mindset.

[01:07:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4067s) So yeah, just to unpack that a bit,

[01:07:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4069s) so there's continual measurement

[01:07:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4071s) and continual learning,

[01:07:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4073s) but there's also a new set of tools.

[01:07:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4076s) And so when I was learning how to become a better engineer at Google, you know, I really

[01:08:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4082s) had to get good at using things like debuggers and profilers and Chrome DevTools.

[01:08:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4087s) And basically, there were all of these tools of the trade for building software by hand

[01:08:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4092s) that maybe are not as important now, but they really separated the best engineers from the

[01:08:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4099s) okay engineers.

[01:08:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4100s) And in the new world of factory engineering, there's a new set of tools that everyone should

[01:08:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4106s) be trying to master.

[01:08:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4107s) And so those are things like evals.

[01:08:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4110s) It's can you set up self-improvement loops that work?

[01:08:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4114s) Can you write benchmarks?

[01:08:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4116s) Can you A-B test your factory?

[01:08:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4119s) And that's a really important skill set.

[01:08:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4122s) And mastering that skill set is something that I think of as factory engineering.

[01:08:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4128s) Now, that doesn't mean that that's the only kind of engineering you're going to be doing.

[01:08:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4133s) You're still going to be doing product engineering, meaning you as an engineer should still feel

[01:08:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4139s) responsible for, you know, does the product you ship actually solve a real user or customer

[01:09:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4145s) problem?

[01:09:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4146s) But the best engineers, and I would say like every engineer really, is going to need to

[01:09:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4151s) learn this new tool set of how do you improve the system, not just build the product.

[01:09:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4157s) And so what does that look like?

[01:09:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4160s) If you think from first principles about how do you engineer an agentic system, I think

[01:09:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4165s) you come up with something like this.

[01:09:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4167s) It doesn't have to be exactly this, but it's kind of an infrastructure stack that you're

[01:09:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4172s) going to want to use to engineer your agents.

[01:09:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4174s) And so at the bottom of this is what I call factories is code.

[01:09:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4178s) I talked about this earlier.

[01:09:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4180s) It's defining all of your coding agent workflows in a code format.

[01:09:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4185s) And the value of that is that you get the ability to branch off it and test different factory configurations

[01:09:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4191s) you can roll back and

[01:09:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4194s) then crucially it opens up the way for self-improvement

[01:09:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4197s) Which is the ability for coding

[01:10:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4200s) agents themselves

[01:10:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4200s) to suggest changes to how your factory works to get that continual loop going.

[01:10:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4207s) Above that is a layer of data and context. This is also crucial. So if you are running

[01:10:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4213s) coding agents at scale in an organization, you really must be recording everything that

[01:10:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4220s) those agents are producing. So all of their conversations, all of their artifacts, the

[01:10:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4224s) videos that they make, the screenshots. And the reason for doing that is that's the raw

[01:10:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4229s) material that you will use to improve the way your factory works over time. On top of

[01:10:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4236s) that is a compute layer. So you need to have sandboxing. Sandboxing increases security.

[01:10:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4242s) It makes your factory more repeatable. Your sandboxing should be probably multi-platform.

[01:10:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4247s) On top of that is the inference layer. The way that I would think about inference and

[01:10:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4252s) models in a factory is they are a degree of freedom to test.

[01:10:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4257s) And so your factory, however you build it, should support multiple models.

[01:11:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4263s) It should also support multiple coding agents because, you know, Cloud Code might be better

[01:11:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4268s) for some things.

[01:11:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4269s) Codex, Pi, Warp, these harnesses all have different properties.

[01:11:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4273s) You might want to write your own coding agent harness that has some of your organizational

[01:11:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4277s) context baked in.

[01:11:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4279s) So your platform should support that.

[01:11:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4281s) On top of that, you need an orchestration layer.

[01:11:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4283s) So this is what actually moves work through the factory.

[01:11:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4286s) In some ways, the least interesting part, but you need that.

[01:11:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4289s) And then finally, your factory should be all API and MCP first.

[01:11:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4294s) So that it can be used not just by, you know, people, but it can be used by other coding

[01:11:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4299s) agents.

[01:11:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4300s) And it can go to where the work is being done.

[01:11:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4303s) So your factory should integrate into things like Slack, Linear, Jira, GitHub, and so on.

[01:11:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4310s) I'm going to give a quick demo of what some of these tools look like in the context of

[01:11:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4315s) warp factories, but this stuff I'm going to show I think applies also if you're building

[01:12:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4321s) your own.

[01:12:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4324s) So let me start here.

[01:12:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4326s) So at the foundation I mentioned earlier is factories in code.

[01:12:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4332s) And so I'm going to show some code.

[01:12:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4334s) It's not going to be that interesting, but if you're familiar with something like Terraform,

[01:12:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4338s) you will get what's going on here.

[01:12:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4340s) You basically want a declarative way of saying,

[01:12:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4344s) this is my factory.

[01:12:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4345s) My factory has certain policies.

[01:12:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4348s) It accesses certain repos.

[01:12:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4350s) It has certain triggers.

[01:12:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4352s) It has a sort of blueprint that defines the workflow

[01:12:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4355s) of work going through it.

[01:12:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4357s) And then if you zoom out,

[01:12:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4359s) all of these other things that are part of the factory

[01:12:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4361s) also have code definitions.

[01:12:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4364s) The second thing, moving up a layer,

[01:12:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4367s) is the idea that everything that you do in the factory

[01:12:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4378s) should be tracked across your entire team.

[01:13:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4383s) And so think of what I'm showing here.

[01:13:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4386s) This is a sort of Google Docs style live view

[01:13:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4389s) of what every coding agent on Warp's team

[01:13:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4391s) across all of our engineers are doing at all times.

[01:13:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4395s) And so I could click it and I could see,

[01:13:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4397s) okay, I don't know what Maggie is working on,

[01:13:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4399s) but I know I could look at it.

[01:13:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4401s) She's implementing Opus 5.5 on the high plan.

[01:13:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4403s) She's doing it from Slack.

[01:13:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4405s) It has this artifact.

[01:13:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4406s) I could look in and see the actual conversation here.

[01:13:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4409s) I could see what it's generated.

[01:13:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4411s) And so I could see this in real time, which is cool.

[01:13:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4414s) But the really important thing

[01:13:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4415s) is that you gather all this information

[01:13:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4417s) so that it can be used for improving the factory.

[01:13:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4422s) And so that brings me to the third thing,

[01:13:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4425s) moving up the stack here.

[01:13:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4426s) is once you have all of this information

[01:13:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4428s) that's being stored centrally,

[01:13:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4431s) you get the ability very easily,

[01:13:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4433s) without doing any, like, install stuff

[01:13:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4436s) on everybody's laptop, anything like that,

[01:13:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4438s) to understand how these coding agents are working.

[01:14:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4442s) And you can think of that in terms of, like,

[01:14:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4444s) traditional output metrics, like your DORA metrics,

[01:14:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4447s) like, okay, cycle time.

[01:14:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4449s) But you can also think of that in terms of, like,

[01:14:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4451s) how automated am I, like, how many human interactions per PR,

[01:14:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4455s) what's your throughput.

[01:14:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4456s) you should be able to see data very clearly here

[01:14:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4460s) on, like, what's your cost.

[01:14:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4462s) Again, surfacing this, if you don't have this data,

[01:14:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4466s) you're kind of flying, uh-oh.

[01:14:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4470s) You're kind of flying blind.

[01:14:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4472s) So this is really important.

[01:14:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4473s) And then there's this next layer of data,

[01:14:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4477s) which is also really important that I mentioned earlier,

[01:14:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4479s) which is the ability to set up online evals

[01:14:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4482s) that run across all your coding agent sessions.

[01:14:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4484s) And so what I mean by this, an eval is just like,

[01:14:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4488s) as these coding agent sessions are running

[01:14:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4490s) and they're recording their output,

[01:14:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4492s) you want the ability to go back and have another agent

[01:14:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4496s) look at them and score them on some dimension

[01:15:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4500s) that might indicate how well they're doing.

[01:15:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4502s) And to make that more concrete,

[01:15:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4504s) so it's like we have an eval, a score here that we use

[01:15:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4508s) that is looking for redundant tests.

[01:15:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4510s) This is a good example.

[01:15:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4511s) And so one common failure mode of coding agents

[01:15:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4514s) is that sometimes they will write tests that are unnecessary.

[01:15:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4517s) We want to catch that, and that's what this eval does.

[01:15:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4521s) We want the ability then to see,

[01:15:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4523s) okay, when it's writing too many tests,

[01:15:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4526s) why is it doing that?

[01:15:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4528s) And is there a mechanism that we could change

[01:15:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4531s) in the factory for improving that?

[01:15:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4533s) And so, again, just to kind of recap here,

[01:15:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4536s) you want a way of recording all the runs.

[01:15:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4540s) When these runs run, I didn't show this,

[01:15:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4542s) but all of the runs are, you know, you want a way of specifying

[01:15:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4549s) what is the sort of, like, actual loop that they're running through.

[01:15:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4552s) You want to be able to see where they are in that loop.

[01:15:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4555s) As the runs work, you want to be able to gather their data.

[01:15:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4559s) Once you gather their data, you want to be able to set up these online evals.

[01:16:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4563s) And then these online evals give you a sense of, like,

[01:16:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4565s) okay, where are there opportunities for improvement?

[01:16:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4568s) Once you have that in place,

[01:16:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4570s) You can do something that we call self-improvement

[01:16:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4572s) Which is rather than having humans

[01:16:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4575s) Look at the outputs of these scorers

[01:16:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4576s) You have agents look at the output

[01:16:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4578s) And so the way to think about self-improvement

[01:16:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4581s) Is like it looks at the score output

[01:16:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4584s) It says okay we saw some number of failed runs

[01:16:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4590s) Let's try to look for patterns

[01:16:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4592s) In those failures

[01:16:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4593s) And suggest a possible solution

[01:16:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4597s) And then the form that that solution takes is as a PR against your factory definition.

[01:16:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4606s) And so this is like the awesome thing about having your factory defined as code and being

[01:16:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4611s) able to use coding agents is that you can then have the coding agents diagnose problems

[01:16:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4617s) with how they're working internally and suggest fixes to them.

[01:17:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4621s) The final tool that I'm going to point out that I mentioned earlier is the ability to

[01:17:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4626s) create and run benchmarks. This is also essential. So if you're following this course online,

[01:17:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4633s) you'll often see things like, okay, this is the best model, this is the new best skill,

[01:17:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4636s) this is the best harness. The only way to answer those questions for real is to test

[01:17:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4642s) on your own data. And so you should have in place some mechanism where you can basically

[01:17:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4647s) create a benchmark set out of your real data

[01:17:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4651s) and then replay those tasks.

[01:17:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4655s) Oops, I'm on the wrong one here.

[01:17:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4657s) And replay those tasks against different configurations

[01:17:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4661s) to see how they would have done

[01:17:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4664s) if you had varied some model or varied some harness

[01:17:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4667s) or varied your factory configuration.

[01:17:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4669s) So just to make this clear,

[01:17:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4674s) this is something that looks like what you might get when a new model drops like in sweet bench or

[01:18:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4682s) terminal bench it's showing the trade-offs of different models here in terms of cost and

[01:18:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4688s) quality you want to be in this upper left-hand quadrant but it's on your own actual data and you

[01:18:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4694s) can you can test not just models but also you want to be able to test hey if i got rid of these skills

[01:18:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4698s) is does my factory perform better or worse?

[01:18:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4700s) So these are some of the tools.

[01:18:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4704s) And just to return to this big concept,

[01:18:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4707s) like this has worked well for us.

[01:18:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4708s) We have managed to use these tools

[01:18:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4711s) to drastically improve our throughput

[01:18:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4713s) and lower our cost.

[01:18:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4715s) We are just constantly doing this

[01:18:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4718s) because this cost really matters to us,

[01:18:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4720s) where we're getting a performance baseline,

[01:18:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4722s) we're scoring, we're benchmarking, we're testing,

[01:18:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4725s) and we're doing factory engineering.

[01:18:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4727s) And so that brings me back to this last point

[01:18:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4730s) that I just want to emphasize again.

[01:18:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4733s) Engineer your agents.

[01:18:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4734s) It's a new engineering primitive.

[01:18:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4736s) We're all engineers.

[01:18:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4737s) Think about how you can get the most out of them

[01:18:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4739s) and build the system for that.

[01:19:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4741s) I think this is very exciting.

[01:19:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4744s) I know there's a lot of, like, angst

[01:19:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4747s) around what it means to be an engineer right now.

[01:19:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4749s) I know we're not writing code,

[01:19:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4751s) but I'm still using all the engineering skills

[01:19:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4753s) that I spent my whole career learning,

[01:19:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4755s) And so to me it's still a really exciting time

[01:19:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4758s) If you're interested in more factories

[01:19:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4759s) You can check it out here

[01:19:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=4761s) Thank you all for coming to the talk