How Software Factories Improve Themselves — Suraj Gupta, Warp

A coding agent can finish an assignment while leaving the next run no better prepared. That becomes a maintenance problem when agents repeatedly handle work from issue triage through production. Suraj Gupta, who leads harness development at Warp, frames the next engineering responsibility as building and maintaining these software factories—and improving how the factory works over time. 0:12

Warp began with a modern terminal and an agentic development environment; the work described here extends into a cloud agent platform for teams running factories. Three mechanisms organize the talk: skills retain procedures, persistent memory retains facts from previous work, and model routing changes which model handles a task. Each gives the factory something it can carry forward into later runs.

A skill gives an agent procedural memory: instructions for a task it will perform repeatedly. A triage skill might explain how to reproduce an issue. But the procedure can become dated as agents discover useful approaches and humans correct their decisions. Unless that experience changes the skill, later runs keep starting from the old instructions. 2:06

Warp separates execution from improvement. The inner-loop agent applies the skill and does the triage. An outer-loop agent examines those runs, looks for mistakes, and collects human feedback. Its output is a proposed change to the procedure that future triage runs will use.

The demonstration uses the triage workflow in Warp’s open-source client repository. A newly submitted GitHub issue triggers the agent. It identifies missing information that would help decide whether the request should become new work, be deduplicated against an existing issue, or receive another disposition. The outer loop then examines those decisions alongside feedback from the issue thread.

• Reactions: Thumbs-up and thumbs-down provide feedback signals on the issue.
• User comments: Follow-up discussion supplies additional feedback about the triage.
• Employee comments: Warp employees can directly correct or guide the agent’s work.

How does feedback become a change without immediately rewriting the running agent’s behavior? The loop below puts a pull request between the outer-loop proposal and the skill used in subsequent runs. Git records how the procedure changes, and a human reviews the update before accepting it. That review matters because an improvement agent can also make a mistake and produce a worse triage procedure.

Procedures do not capture everything worth remembering. Consider a Sentry agent that investigates an issue, gathers context, finds the root cause, and fixes it. When a similar issue appears later, a fresh run may fail to find the same cause. Even if it succeeds, it spends tokens gathering context the factory already gathered once. 5:39

Persistent memory carries that investigation forward. Gupta describes it as a fact store scoped to an agent. An outer-loop agent extracts facts, learnings, and outcomes from the inner-loop run, making them available to future runs. The retained material answers what previous work discovered; the skill still describes how to do the work.

In Oz, Warp’s cloud agent platform, the Sentry agent has accumulated memories from earlier root-cause analyses. Reuse also creates a curation problem: an agent can retain an unimportant observation or a conclusion that worked locally but should not guide future work. Oz exposes versioning and human controls to create, edit, or delete memories, along with links to their source runs. Opening a source run fails during this demonstration; Gupta suspects that changing IP addresses and staging IP restrictions explain the failure. That provenance navigation is described rather than completed on screen. Its practical purpose is clear: a questionable memory can be examined in the context that produced it and removed if it should not persist.

The Sentry example then develops into a visible record of reuse. A later run performs another investigation and reports which memories it used—five in the demonstrated store. Gupta attributes faster work to that reuse. The concrete change is that earlier investigation results now participate in a later run; the demonstration does not quantify the speed or token savings.

The memory layer is presented as working across harnesses running in Oz. Automatic memory creation therefore belongs to the platform’s surrounding workflow, while humans retain the ability to review, version, delete, and edit what it remembers. Changing the harness need not mean giving up this shared mechanism for retaining previous work.

A factory multiplies the cost of a model choice across recurring work. Using Opus for every triage task or simple CI fix can become prohibitively expensive. Model routing makes that choice depend on the task, so relatively simple work does not automatically inherit the cost of the model selected for harder work. 8:46

Warp offers two ways to make that decision:

• Auto models: Warp maintains routers and evaluates new models for Pareto efficiency—the tradeoff between useful performance and cost, rather than assuming one model should handle everything. Gupta recommends this as a starting point over pinning every task to Opus or Haiku.
• Custom routing rules: A configuration assigns classes of tasks to models. The demonstration includes database migrations assigned to GLM, plus a separate assignment for runbooks and API documentation. These rules express the operator’s judgment about which model suits each kind of work.

Custom rules alone leave the choice “more of an art than a science.” The next step described in the talk is customer-facing evaluation: define what matters in your workflow, vary routing choices, and compare models on a specific class of issues. A generic benchmark cannot settle whether Haiku, GLM, or Opus handles your particular work best. At the time of the recording, these customer-facing evaluation tools are planned work.

Warp already uses an internal evaluation sidecar alongside its routing rules. A prompt is run through multiple agents in Oz using different models, and the comparisons help determine which models work better for which tasks. This supplies the missing improvement mechanism. Routing rules begin as choices; repeated comparisons give the team evidence for changing those choices. 11:42

What relationship does this evaluation make visible? The diagram separates the rules used to choose a model from the comparisons used to inform those rules. The useful output is knowledge about a task class, which can guide later routing rather than remaining an isolated result from one run.

One internal finding changes a concrete routing decision: Gupta reports that GLM handles UI tasks well enough that Warp can use it more efficiently than Opus for that work. The talk supplies no quantitative scores, scoring procedure, or task distribution, so this supports Warp’s reported routing choice rather than a general ranking of the models. The closing product direction is to bring these evaluations into customers’ own workflows, allowing each factory to improve its routing with evidence from the work it actually performs.