How to Build a Software Factory That Improves Itself - Zach Lloyd, Warp

An engineer opens several coding agents, assigns each one a task and switches between them when they ask for input. It can be a productive afternoon. At the end of it, a team lead may still struggle to answer three questions: Which changes reached users? What did they cost? And what did one agent learn that the next engineer can reuse?

Zach Lloyd, Warp’s founder and a former engineering lead on Google Docs, describes that arrangement as interactive prompting. Each laptop has its own tools, permissions and configuration. Someone has to start each job and keep it moving. A local agent may also inherit access its particular assignment never required. Issuing more subscriptions gives the company more of these sessions, with little insight into how the work adds up.

In the alternative Lloyd calls a software factory, a person submits a feature or reports a bug. The work can pass through triage, a specification, implementation, review and verification. Some steps need a person. Others suit an agent running in an isolated cloud environment. A reviewer can see what went into a step and what came out after the person who started it has gone home.

Lloyd wants the engineering team to own the definition of this process. Its repository records which agent handles a step, which tools it may use, the instructions it gets, the model it calls and where it runs. The same repository can hold benchmarks and the events that start a job. Someone reviewing a change can see precisely which part of tomorrow’s workflow will differ from today’s.

There is a business reason to spell these things out. A team might want to keep its data on its own infrastructure, pay for inference through an existing account or replace a model when another proves better for its work. Those decisions should be visible to the people responsible for the result. A stable record also gives an engineer two versions of a workflow they can test against each other.

Warp gives its product engineers familiar entry points. A task can arrive through Slack, a Linear ticket or a GitHub review comment. In the demo, an orchestrator selects the next step. A dashboard brings the runs and their artifacts together. Lloyd distinguishes the engineer shipping a customer feature from the engineer who improves the prompts, skills and routing that many such features will use. On a small team, those can be two jobs for the same person.

The dashboard starts with ordinary operating questions: cycle time, spend, pull requests and the number of times a person had to intervene. The interesting part lies behind the charts. A team can inspect what an agent received, the conversation as it worked and the change it left behind. That record gives a reviewer much more to go on than a green check beside a pull request.

Warp runs model-based evaluators on these traces. One looks for redundant tests. A task can produce passing code and an unnecessary test, so a passing test suite alone would miss the behavior that reviewer cares about. Storing the evaluator’s judgment with the run lets an engineer inspect the cases where it claims to have found the problem.

In a separate example, the improvement system found 37 instances of agents beginning implementation without reading the relevant skills already in the repository. Its suggested fix was a pull request against the factory definition. Lloyd opened the proposal onstage. He showed how a recurring mistake can lead to a specific edit someone can review. Whether that edit reduces the failure rate is a question for the runs that follow.

How would the team know whether to buy a more expensive model for front-end work? A public leaderboard says little about its own repository or conventions. Lloyd wants to take representative work the team has already completed, preserve the starting conditions and replay it against competing configurations.

The live demo supplied a mundane but important detail. Lloyd tried to add a past run to a benchmark and could not do it on the example he had selected. He switched to an existing suite to show the result. In his spoken comparison, one configuration scored about 93 percent for roughly two dollars. Another reached about 95 percent and cost around three times as much. A team can decide what those last points are worth once it knows which tasks the cheaper choice misses.

A question from the audience established what “correct” meant on that chart. The scores came from an LLM judge. Human grading was something Lloyd recommended, and Warp had yet to add it to that workflow. His advice for building the first suite was equally practical: pick a few easy, medium and difficult tasks by hand. With a baseline in place, use the archive of previous sessions to look for cases it leaves out.

More agent throughput can leave humans staring at a longer queue of pull requests. Asked about verification, Lloyd suggests writing a specification the result can be checked against and asking an agent to produce evidence from the user’s point of view. For an interface change, a reviewer who can see the changed interaction has a more concrete starting point than a summary saying the implementation is complete. The talk describes that practice; the demonstration focused on scoring and benchmarking.

Verification costs money too. Lloyd’s suggested unit is the cost per pull request, with scoring and benchmark expenditure broken out. That lets a team estimate a project from work it recognizes and decide where another evaluation is worth paying for. It also reveals when a seemingly cheap model needs expensive supervision.

Warp’s terminal is open source. Lloyd clarified in the Q&A that the factory product he demonstrated is proprietary and designed to accommodate a team’s own tools. An individual with one interactive agent may have little need for it. The problem becomes pressing when one engineer’s change to a shared workflow can improve the next hundred tasks, and the team needs evidence to decide whether to keep it.