# Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp Ben Holmes, Warp [Talk page](https://ai.engineer/talks/pz74r-4dGfQ-your-ai-agent-is-hiding-its-mistakes) · [Video](https://www.youtube.com/watch?v=pz74r-4dGfQ) ## Your AI Agent Is Hiding Its Mistakes From You Ben Holmes shows how Warp turns failed tool calls and repeated human corrections into pull requests for better agent skills, then uses later sessions to check whether the fixes helped. ### Who engineers the software factory? [0:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=12s) Ben Holmes, a DevX engineering lead at Warp, works on Warp Factories, infrastructure for teams building their own software factories. The familiar part of a factory is the path from a task to a reviewed change. Work might arrive through a task list, a board, or a Slack message. An agent researches or triages it. Another might work with a human to write a specification before an agent implements the change. Computer use gives the implementation agent a way to include screenshots or recordings with its pull request. Agents can run rounds of adversarial code review before the change reaches a human reviewer, and monitor the system afterward. ![Eight boxes form a clockwise loop inside a dashed blue circle. Work moves from assignment through implementation, verification, review and shipping. Monitoring usage leads back to a new assignment.](https://ai.engineer/api/talk-essay-media/b2e50efc7e273e796d2d6b48ac42c8ceb2c4ba26952f6cebc11dd8cf3abec5df) Inner loop - improve the product Each handoff gives an engineer something to inspect. A reviewer can read the spec, try the change, and decide whether the PR is ready to merge. A different engineering question concerns the process that produced it. If every implementation agent has to rediscover how to start the test environment, the team pays for that discovery on every task. The finished PR gives the reviewer little reason to suspect it happened. Holmes calls work on the system that drives the agents “factory engineering.” It can extend to changing the architecture of the codebase or using static analysis to steer agents toward the right parts of it. He concentrates here on a narrower surface that a team can edit and review directly, its agent skills and configuration. The loop starts with completed conversations. Scoring agents judge how the work went, a scheduled agent examines recurring failures and proposes changes, and subsequent conversations supply evidence about the effects of those changes. ### Collect sessions where the work runs [3:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=192s) For an engineer improving a personal setup, the collection step is small. Local agents such as Claude Code, Codex, and Pi keep session files on the machine where they run. Claude and Codex each have their own directory. A reviewer that can read the relevant local histories can look across tasks, including sessions the engineer has long since stopped thinking about. The team version needs a place to read work from more than one engineer. Holmes has spoken with teams that automate zipping up everyone's local Claude directories and putting the archives in a shared S3 bucket. That gives the reviewing agent a common collection even when the engineers continue to run their coding agents on separate laptops. A shared runner offers another collection point. The team sends tasks to a common system, which might run on a long-lived compute instance or spin up an isolated sandbox for each run. Warp uses isolated Docker sandboxes for its agents. The individual agents can have separate execution environments while their traces pass through one system. That distinction matters when looking for a pattern. What looks like an awkward test setup on one engineer's machine might turn up across many tasks that used the same skill. ### Keep the request, the tool results, and the output together [5:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=311s) Holmes wants three kinds of evidence available when a scoring agent opens a session. First comes the transcript, including what the human sent, how the agent responded, and what the human said next. That last part can matter as much as the initial request. If the human repeatedly asks for a changelog entry to be rewritten, the correction is evidence about the work required to get the desired result. Second come the tool calls and their results. Include filesystem operations, calls to MCP tools, and the other interactions that connect the agent to its environment. Their sequence lets the judge distinguish a test command that worked the first time from one that worked after several failed attempts. Both runs might end with the same accurate statement that the tests passed. Third comes the output the judge will evaluate for quality. A code diff stored with the conversation will do. If the output lives in a pull request, pin the commit and inspect its diff so the judge sees the version associated with the run. Each kind of evidence answers a different question about the same piece of work. The transcript establishes what the agent was asked to do, the calls show how it proceeded, and the diff lets a judge inspect the tests or comments it actually wrote. ![The left column lists Claude, Codex and Warp conversation storage. The right lists material available for review. Three boxes below name the scoring categories.](https://ai.engineer/api/talk-essay-media/5ab92619fa11dffe376e9a6d78859d7f3e39c2dea2e72182a8972818dcc3c97f) Local agent conversation sources and scoring categories ### Define redundant tests and verbosity in terms a reviewer can judge [6:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=371s) Holmes asks for a show of hands from people who feel their agents write too many tests. He estimates about 40 percent of the room raised a hand. On his team the response was closer to 100 percent. Those are informal measures of how many people recognize the problem. To score a particular PR, the judge still needs to know what the team considers appropriate coverage. Holmes proposes a scorer called Redundant Test. The decision depends in part on the team's expectations for unit, integration, and end-to-end coverage. An agent can produce a substantial suite of unit tests for a change the team primarily wants exercised end to end. Counting the new tests tells the reviewer how much was added. Reading them against the team's guidance lets the judge explain why that particular investment in coverage was excessive. Holmes notes that both model selection and instructions can affect what a team sees. Verbosity concerns two things a human reviewer has to read. One is the agent's explanation of its work, including the summary that helps someone understand a PR. The other is the commentary it leaves in the code itself. Holmes asks the room about excessive code comments because they impose a cost beyond visual clutter. When review is the bottleneck, a reviewer spends time working through those comments to understand the change. A verbosity scorer can inspect both the conversation and the diff and judge whether the agent communicated the change clearly. ### A passing test can conceal a broken testing skill [7:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=461s) Efficiency is where the tool trace earns its keep. An engineer sitting through a short interactive session might notice a string of failed calls. Send the agent on a longer run in a cloud sandbox and the first thing the engineer reads may be its final report. Coding agents are persistent enough to get around quite a few obstacles. Once a workaround succeeds, the agent can report the outcome without mentioning the failed route it took to get there. Warp encountered this in its own test harness. The testing skills contained out-of-date instructions, so the agent had to work out what environment it was in. It tried a succession of port numbers until it found a way to proceed, then got on with the code. From outside the run, the symptom was that the agent seemed to take roughly two minutes longer than expected. That was Holmes's estimate of the initial clue, rather than a timed comparison of two skill versions. A judge looking at efficiency can follow the attempts in order and ask which ones moved the task forward. The successful call explains how the agent recovered. The unsuccessful calls explain why recovery was necessary and where better instructions could have saved work. As long as the stale skill was the starting point for the next run, a successful workaround helped only the agent that found it. ### Write a Markdown rubric with an investigation threshold [8:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=522s) Warp expresses a scorer in Markdown, much like a specialized skill definition. For efficiency, the question is whether the agent worked toward the result or spent effort on redundant steps and unnecessary back-and-forth. The scorer gives an LLM judge labels describing different levels of performance. Holmes likens them to a teacher's essay rubric. A grader has a much more specific job when it knows which features of the work distinguish one mark from another. The efficiency definition calls out three places to look. Cost to the human includes the extra exchanges required to get the work done. The information-gathering phase covers what the agent had to explore before it could proceed. Routine step overhead covers calls consumed by ordinary operations, such as getting a test environment running. These distinctions allow the judge to recognize wasted effort even in a run that needed no human intervention, or repeated human effort in a run with few tool errors. The team also chooses where a grade merits investigation. In Holmes's example, 0.4 and 0.2 fall below the line. Scores above it pass and can usually be left out of the failure review. The threshold turns the rubric into a practical filter once there are many sessions. Holmes recommends starting with a couple of bullets and a rubric, trying them against conversations, and building up the definition as needed. The team has to decide what counts as wasted effort in its own environment. ![Warp displays four efficiency labels in a rendered Markdown file. Their scores range from 1 for a direct path with batched steps to 0.2 for repeated rework, user correction or looping.](https://ai.engineer/api/talk-essay-media/ebaadcca405a298051fd5e70fd0b2aeb69f345a1b885aa23ce345ecf04d0c1ed) efficiency.md scoring labels ### Grade one merged authentication PR three ways [10:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=641s) Holmes walks through a hypothetical full-stack authentication service. The agent implemented it, the conversation is available, and the PR has merged. A merge is Warp's usual trigger for grading. The unit of evaluation is one judge run per selected conversation, with that judge assessing the conversation against each criterion. A team can grade every conversation or select a random sample to control token use. Start with the tests. The implementation agent wrote a large number of unit tests for the service, while the team's instructions say to prefer end-to-end tests for most of this coverage. The judge can fail Redundant Test and identify the mismatch. Move to verbosity and the same PR passes. In this example the team has selected a model that leaves few code comments, and the PR is easy to understand. The testing issue supplies no reason to reduce the communication grade. Finally, follow the calls the agent made while figuring out how to test the service. It spent considerable effort exploring the environment to discover something the team could have explained up front. That earns an efficiency failure under the example rubric. The distinction between the two failures is consequential for the person editing the skills. The agent overlooked existing guidance on which tests to write. It needed additional guidance on how to run them. ![The example session fails the redundant-tests and efficiency checks and passes verbosity. Each grade appears beneath its rubric filename. A sessions list and a pass/fail legend appear below.](https://ai.engineer/api/talk-essay-media/a456adc7ed261e3d704c69c54db03774279b59cb3121fdeabe56997f586bc467) Scoring update-auth-service.jsonl with three rubrics The judge records a reason alongside each pass or failure. Over many sessions the team can calculate pass and failure rates for each criterion, but a rate alone gives the next agent little to act on. The explanations preserve details such as which testing preference the agent missed or which part of setup consumed the extra calls. ### Give a daily agent the failed reports and the skills repository [12:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=761s) Grading happens close to the individual run. Finding a shared cause happens on a different schedule. Holmes describes a cron job that wakes every 24 hours and looks over a sample from the preceding day. His illustrative numbers are 200 conversations and 50 failures. The scheduled agent receives the failing reports with the judges' notes about what failed and why. Its assignment is to find commonalities across them. Give that agent access to the skills repository so it can propose an edit where the relevant instruction lives. Holmes also describes giving it access to the codebase if architectural changes are in scope. For example, if redundant tests recur across PRs, it might propose adding a coverage pyramid that makes the team's expectations for different kinds of tests clearer. The recurring reports give a reviewer a concrete reason to consider that edit. A single disappointing score on an authentication PR would say much less about what belongs in a shared skill. ![Four steps circle the core factory loop. The scoring example marks efficiency and code quality with checks and test coverage with a cross. A proposed edit to the test-writing skill adds a coverage pyramid.](https://ai.engineer/api/talk-essay-media/b61147995a967e0616436bf251442d8872e2626675f094df1c177f5c8f899d7c) Every 24 hr: review agent conversations and propose improvements ### Skill Doctor finds two different fixes in local Codex history [13:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=821s) Skill Doctor gives an individual engineer a one-time version of the loop. The skill is available at [warp.dev/skilldoctor](https://warp.dev/skilldoctor) and can review local Claude Code, Codex, or Pi conversations. In Holmes's example it reads Codex sessions from work on the project he is showing and grades them in batches to use tokens more efficiently. The grading stage produces score reports. The self-improvement agent then reads the scores and notes and produces a report card with suggested changes. Holmes identifies efficiency and code quality as criteria in the test environment and says verbosity looked fine. One finding concerns the end-to-end testing skill. Across sessions, the agent was struggling to get a port on the machine where it ran tests. The proposed edit changes the order of operations in the instructions. Find an available port first, then use it to start the service. That gives the next agent a specific action at the point where its predecessors began guessing. Holmes says this kind of problem is easy to miss when the run happens in the cloud or its activity is tucked behind a working indicator in the agent interface. The changelog finding comes from a different part of the record. Holmes kept asking for changes to the language of the changelog entries agents produced for PRs. Skill Doctor picked up that repetition and rewrote the changelog skill to prescribe the wording more closely. In the port case, the agent had been doing the extra work. In the changelog case, Holmes had been doing it by steering the agent back to his preferred language. A one-time pass over existing history surfaced both. ![A highlighted three-line addition tells the agent to name a feature once. It pairs a short keyboard-shortcut release note with the list of options a user previously had to trim.](https://ai.engineer/api/talk-essay-media/7fb5a365b6c2787bb2181285a1a12c0bc0d4e5d2a8f65ec85d057fafbb6b77d0) Proposed changelog skill edit with a keyboard-shortcut release note ### Check later scores against the date a skill changed [16:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=971s) Holmes loses access to his staging view as he reaches the score graphs, so he describes how to read them. He compares the setup with an agent harness that reflects on conversations overnight and stores memories. Both approaches let past sessions influence future work. With a memory system, an engineer can have difficulty seeing what the agent concluded. Explicit grades give the engineer a record of how the sessions went and a quantity to follow after changing a skill. His proposed graph plots average efficiency over time as new conversations are graded. He uses 50 percent and 60 percent as example averages. Mark when a proposed skill was merged and look for an inflection in the subsequent scores. For the port change, the underlying traces also give the reviewer a specific behavior to check in later test runs, whether the agent finds a port before trying to start the service. The percentages illustrate how to read the graph. Holmes gives no measured before-and-after gain for that fix. ### A merged triage fix changes the factory configuration [17:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1061s) Warp Factories runs the scoring continuously across the agents supporting Holmes's team. On a schedule, another agent reviews those scores and suggests PRs. The team's inbox is the human review point. Holmes shows a proposal the team had already merged, involving a triage skill that told an agent to spawn sub-agents. The triage agent repeatedly spent tool calls trying to work out how to spawn them. The missing piece was a factory configuration setting that enabled sub-agents. The proposed PR patched that configuration, bringing the capabilities available to the agent into line with the skill it was already following. This is the value of retaining the attempted calls alongside the instruction. They can lead a reviewer all the way from a costly sequence in a completed session to the particular setting the next session needs. ![In a side-by-side GitHub diff, three green lines add a spawnableBy list to the TRIAGE agent definition. The entries are foreman and triage.](https://ai.engineer/api/talk-essay-media/72d05865f752628d73fc878c6faee51547156d8e04b12410277f2362a6a7f0a8) GitHub diff adds foreman and triage to spawnableBy ## Key takeaways - Collect the human-agent transcript, tool calls and results, and a commit-specific output together. A judge needs different evidence to assess communication, wasted effort, and code quality. - Score each criterion separately and keep the judge's reason. Two failures in one PR can call for different repairs, such as clarifying a testing preference and supplying a missing setup step. - Group failed reports across sessions before proposing shared skill or configuration changes. After a merge, use later scores and the relevant tool calls to check whether the targeted behavior improved. ## Slides ### Inner loop - improve the product [0:56](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=56s) [Eight boxes form a clockwise loop inside a dashed blue circle. Work moves from assignment through implementation, verification, review and shipping. Monitoring usage leads back to a new assignment.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2e50efc7e273e796d2d6b48ac42c8ceb2c4ba26952f6cebc11dd8cf3abec5df.webp) Inner loop - improve the product New work assigned Triage Spec writing Implementation Verification (computer use) Code review Ship Monitor usage ### Every 24 hr: review agent conversations and propose improvements [3:09](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=189s) [Four steps circle the core factory loop. The scoring example marks efficiency and code quality with checks and test coverage with a cross. A proposed edit to the test-writing skill adds a coverage pyramid.](https://assets.ai.engineer/sites/aiecode2025/sha256/b61147995a967e0616436bf251442d8872e2626675f094df1c177f5c8f899d7c.webp) Core factory loop Every 24 hr Review agent conversations Score quality Isolate failures Propose improvements Scoring rubrics ✅ efficiency ✅ code quality ❌ test coverage test-writing/SKILL.md +57 -20 ... +Coverage pyramid When to add unit vs. integration tests... ### Local agent conversation sources and scoring categories [7:50](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=470s) [The left column lists Claude, Codex and Warp conversation storage. The right lists material available for review. Three boxes below name the scoring categories.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ab92619fa11dffe376e9a6d78859d7f3e39c2dea2e72182a8972818dcc3c97f.webp) Local agents ~/.claude/**.jsonl ~/.codex/*.jsonl Warp conversation storage Transcript Tool calls / MCP calls Outputs (code) Redundant tests Verbosity Efficiency ### efficiency.md scoring labels [9:55](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=595s) [Warp displays four efficiency labels in a rendered Markdown file. Their scores range from 1 for a direct path with batched steps to 0.2 for repeated rework, user correction or looping.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebaadcca405a298051fd5e70fd0b2aeb69f345a1b885aa23ce345ecf04d0c1ed.webp) ~/Projects/common-skills/.agents/skills/skill-doctor/scorers efficiency.md rework, or unnecessary back-and-forth." labels: • value: "highly_efficient" description: "The agent took a direct path: nothing re-read or re-run, independent steps batched, no work redone." score: 1 • value: "mostly_efficient" description: "The agent slipped once or twice: a duplicated read, an early retry, or a small correction — with no knock-on cost." score: 0.8 • value: "mostly_inefficient" description: "The agent wasted effort repeatedly, or caused a round of rework an earlier check would have prevented." score: 0.4 • value: "highly_inefficient" description: "The agent's waste dominated the run: the same defect reworked across cycles, repeated user correction, or extended flailing / looping." score: 0.2 ### efficiency.md criteria for assessing wasted effort [10:25](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=625s) [Warp displays the assessment section of the efficiency rubric. Five bullets describe avoidable rework, repeated user correction, redundant searches, overhead in routine steps and missed opportunities to batch work.](https://assets.ai.engineer/sites/aiecode2025/sha256/27eb3ed54ce54e12c2cb0dfd0862ff1402921d7398e97c78d859b77ab6cb7081.webp) ~/Projects/common-skills/.agents/skills/skill-doctor/scorers efficiency.md result: the steps the agent took, rework it caused, and human attention it consumed. Score against what a competent engineer with the same tools would have needed, not against what was achievable with only what the agent happened to have. A mistake that looks unavoidable in context still counts if better tooling, a skill, or a check would have prevented it — name that cause in the reason. The bullets below are common sources of waste, not an exhaustive checklist — judge any other way the run cost more than it should have. Assess: • Rework from mistakes. Work redone because the agent got it wrong the first time: a test or build failure a local check would have caught, edits to the wrong file, a misread requirement later reverted. • Cost to the human. Repeated correction or steering from the user is the most expensive waste. A question asked up front is cheap; the same question asked after building the wrong thing is not. • Information gathering. Re-reading, re-running, or re-searching for something already found; reading a large file end to end when a targeted search would answer it. • Routine-step overhead. A roundabout way of doing something that's a standard, repeated part of this agent's job — more steps, more calls, or a broader operation than the step needs — when a more direct path was available. Weight this beyond its one-run cost: the same avoidable overhead recurs on every future conversation until a skill or rule fixes the pattern. • Batching. Independent reads, searches, or workstreams run serially across turns instead of together. ### Scoring update-auth-service.jsonl with three rubrics [12:04](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=724s) [The example session fails the redundant-tests and efficiency checks and passes verbosity. Each grade appears beneath its rubric filename. A sessions list and a pass/fail legend appear below.](https://assets.ai.engineer/sites/aiecode2025/sha256/a456adc7ed261e3d704c69c54db03774279b59cb3121fdeabe56997f586bc467.webp) update-auth-service.jsonl Redundant tests redundant-test.md FAIL Verbosity verbosity.md PASS Efficiency efficiency.md FAIL Sessions add-chat-mvp.jsonl update-auth-service.j Grades PASS FAIL ### Reviewing agent conversations and scoring failures [12:50](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=770s) [A four-step flowchart runs from reviewing conversations to proposing improvements. Beside it, three example sessions each have a red FAIL, a green PASS and a red FAIL.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8c6859c4d392853f360636a2a5997e216fdffc785e981acd60227661ff55aa1.webp) Review agent conversations ↓ Score quality ↓ Isolate failures ↓ Propose improvements Local agents ~/.claude/**.jsonl ~/.codex/*.jsonl Warp conversation storage Transcript Tool calls / MCP calls Outputs (code) Redundant tests Verbosity Efficiency add-chat-mvp.jsonl FAIL PASS FAIL update-auth-service.jsonl FAIL PASS FAIL add-sentry-webhook.jsonl FAIL PASS FAIL Redundant tests redundant-test.md FAIL Sessions add-chat-mvp.jsonl /testing when to use e2e tests ### Scorer-b reviews a wrong-row UI fix and gaps in the transcripts [14:19](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=859s) [In Warp, the selected scorer-b tab reports that the wrong row remained highlighted after the agent claimed to verify a fix. The scorer also identifies two limits of the condensed transcripts.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb600ec9f9081fc9fa351f0053dff7bfddd0c9f1a78d8ddebe2f8167a964b6be.webp) Orchestrator scorer-b scorer-c scorer-a node_modules, without asking the user. That destructive action is why procedure is noncompliant. It never changed any source code, so code quality couldn't be judged. • 019fe354: The user had to correct it twice after it said it was done. The second time, it had said it verified the fix in the running app, but the wrong row was still highlighted. The taste-review step failed because the claude CLI wasn't logged in, and the agent carried on without it. The condensed transcripts leave out many entries, so two things couldn't be fully checked: • In 019fe354 I couldn't see whether pnpm check:react-compiler was run after the React changes, so I didn't penalize it. • Three sessions used a request-pr-review skill that is no longer in the repo's skills folder. I judged them against the version quoted in the transcripts. No other files were changed. ### Agent Skill Report grades 12 conversations and proposes a push-to-pr skill [15:09](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=909s) [The report shows a B with an overall score of 83, four labeled score bars and counts of conversations and skills. Findings about PR branches, missed skill triggers and UI defects precede a proposed skill.](https://assets.ai.engineer/sites/aiecode2025/sha256/6595df093e29cf7384b407a0f4709edf9a88b0f47d2ffc9d67fc5073fd076343.webp) B OVERALL 83 code quality 75 procedure compliance 80 verbosity 100 skill coverage 100 12 conversations scored 19 skills installed 16 skills used Findings • Agents push to the wrong PR branch. In 3 of 12 sessions, the agent pushed to origin instead of the contributor fork, made a new upstream branch, or force-pushed a contributor's branch without asking. • Skills that exist do not trigger. writing-tests did not load in 3 sessions that added tests, and changelog did not load in 2 wrap-ups. Users had to remove a useless test and rewrite changelog entries. • The user finds UI defects after the agent says it is done. In 4 sessions, the user reported a visual or focus bug after the agent said it verified the change. Codex also wastes calls on 1-second polling and on wait_agent timeouts below 10000 ms. Suggested skill changes 1. push-to-pr (new) — Add a skill that resolves the PR head repo and ref before pushing, checks for `[new branch]`, and forbids amending or force-pushing someone else's branch without asking. Evidence: Failed sessions: 019fd202 (procedure, efficiency): commits went to origin, not to the zcuric fork behind the PR. 019fe357 (efficiency): `git push -u origin fix/header-alignment` created a new branch on upstream, and the agent reported it as done. 01a00d64 (efficiency): the agent amended and ran `--force-with-lease` on the contributor's fork without asking. No skill or AGENTS.md rule covers this. proposed/push-to-pr/SKILL.md +14 Automatically improve your skills with Warp Factories continuous scoring · continuous skill tuning Request access ### Proposed changelog skill edit with a keyboard-shortcut release note [15:50](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=950s) [A highlighted three-line addition tells the agent to name a feature once. It pairs a short keyboard-shortcut release note with the list of options a user previously had to trim.](https://assets.ai.engineer/sites/aiecode2025/sha256/7fb5a365b6c2787bb2181285a1a12c0bc0d4e5d2a8f65ec85d057fafbb6b77d0.webp) 3. changelog — Make the skill trigger at merge-ready and before a PR push, and add a rule to name the feature without listing its sub-features. Evidence: Failed sessions: 01a00d64 (efficiency): the user had to trim an entry that listed 'search, conflict warnings, disabling, and resetting bindings'. 019fbfcf (efficiency): the agent called the branch merge-ready with no changelog entry. 019fbe70 (procedure): the entry was rewritten 3 times after user corrections. …enjamin/Projects/hubble.md/.agents/skills/changelog/SKILL.md → …osed/changelog/SKILL.md −1 +4 1 --- 2 name: changelog 3 description: Add a user-facing entry to CHANGELOG.md for work that just landed. Use when wrapping 3 description: Add a user-facing entry to CHANGELOG.md for work that just landed. Use when wrapping 4 --- 6 # Changelog 11 unmodified lines 18 3. Write it for a user reading release notes, not as a commit message: 19 - One line, present tense. 20 - Describe the effect, not the implementation or file paths. 21 - Name the feature once; do not list its sub-features, options, or edge 22 cases ("Customize keyboard shortcuts in Settings.", not "...including 23 search, conflict warnings, disabling, and resetting bindings"). 24 - Skip internal-only churn (refactors, deps, CI, tests) unless it changes 25 what the user experiences. 26 4. Link the PR for every entry (not the issue). SHOW LESS ### GitHub diff adds foreman and triage to spawnableBy [18:33](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1113s) [In a side-by-side GitHub diff, three green lines add a spawnableBy list to the TRIAGE agent definition. The entries are foreman and triage.](https://assets.ai.engineer/sites/aiecode2025/sha256/72d05865f752628d73fc878c6faee51547156d8e04b12410277f2362a6a7f0a8.webp) Let triage start research subagents by warp-agent-staging[bot] · Pull Request #1059 · warpdotdev/factory-dev Files changed 1 0 / 1 viewed Submit review +3 1 --- 2 description: Triages requests and establishes task state. 3 agentType: TRIAGE 4 + spawnableBy: 5 + - foreman 6 + - triage 7 model: grok-4-6-medium 8 idleTimeoutMinutes: 10 9 --- ## Full transcript [0:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=12s) I'm Ben Holmes. I'm a DevX engineering lead at Warp, and I'm gonna be talking about building self-improving agents, agents that look through their past work and figure out how to improve their own setup. Uh, so for a bit of context, I work on Warp Factories, which is open, flexible infrastructure to help you all build software factories of your own and crucially, ones that improve themselves as part of the stack. I'm not gonna be talking too much about software factories as a whole today, but I'm gonna be talking about [0:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=42s) one crucial part of it, which is having agents introspect all of the work that they do inside of one of those systems and suggest improvements so that the agent performs better and better over time. Uh, so I'm gonna start by just presenting what you might think of when you hear the word software factory for a bit of definition setting. It's the idea of automating the software development life cycle. You may have seen this as an assembly line of different agents doing different things and handing off. You may have also seen it as a wheel, as [1:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=72s) you see in this case, where you start from a task list, a board, Slack messages, wherever you're tracking work, and you have an assembly line of agents go through the process of researching it or triaging, maybe working with a human to write a spec, implementing that change and bringing computer use into the picture to take screenshots and recordings so that it's actually a good pull request that you get on the other side. And also maybe doing rounds of adversarial code review in the mix, uh, to make sure that it's something that is ready for [1:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=102s) human eyes. Um, and on the other end, you have agents monitor the system and around and around we go. Uh, but the more interesting part, I would say, is all of the sort of meta engineering that goes around making that wheel run as efficiently as it can. Um, so a few terms have been maybe floating around right now, like meta engineering, uh, gardening, or something that we've been using, which is factory engineering. It's the idea of actually putting in engineering [2:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=132s) work to improve the system that is driving the agents themselves. And there are a lot of things that you might wanna optimize inside of that system. It may be the skills that are driving the agents. It may be improving the code architecture and doing static analysis to make sure the agents are working on the right parts of the code base. I've been hearing a lot about, like, SonarQube and other sponsors at this event that are doing things in this vein. Uh, the improvement loop that we're gonna be talking about is specifically around agent skills and configuration. Uh, and the loop is gonna look something like this. [2:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=162s) It's the idea that you can have agents review their past conversations. You can set up some scoring agents to score the quality of how those conversations went. So think of this like an LLM as a judge reviewing against rubrics that you define across metrics like efficiency or code quality or test coverage. And then once you've graded a whole lot of conversations across all the work your team is doing, you could find all the failures, figure out reasons that those failures are happening, and then maybe an agent wakes up on a cron to suggest improvements. And then you [3:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=192s) can continue evaluating, seeing how the scores perform, and so on. Um, so I'm gonna break down that wheel into the individual segments that we can talk about each of them individually. Um, so the first part of that loop that I showed you is getting all of your agent conversations in one place so that the agent can actually reason about all the work that your team is doing. Uh, if you're working with, you know, local agents like Claude Code or Codex or Pi or any of those, those are [3:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=222s) going to be files on your system. So Claude, you're gonna have a Claude folder where all that lives. Codex, you're gonna have a Codex folder. And that works well enough if you want to sorta optimize your single player setup. I've actually talked to teams that set up an S3 bucket and an automation to, like, take everyone's doc Claude folders and zip them up and put them in a shared S3 instance just so we have everyone's conversations in one place. It, it's a way of doing things. Um, but there are other ways that you can [4:12](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=252s) sort of centralize this information. Uh, if you're using a software factory system, it kinda comes for free. This is the idea that agents are running inside of maybe a shared computer. So not on everyone's individual laptops, but, like, a shared computer that everyone's going to send tasks into. That might be, like, a long-running compute instance on OpenCloude, or it might be, like, isolated sandboxes that spin up for every run that you do. Uh, if you're using Warp's factory setup, it would be isolated docker [4:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=281s) sandboxes for each agent working on tasks. Using other cloud runners, you'll probably end up in the same sorta system. But it's all the same idea. We wanna find a way to get everyone's team-- your whole team's conversations in one place so that they're easy to introspect and figure out how to improve your setup. And if you're doing it on one cloud runner where all the tasks flow through, you kinda get that for free because all the agent traces belong to a single instance. Um, so that's something that's baked in. And inside of that, you're gonna wanna make sure you have all the [5:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=311s) information up front. So not just the, like, transcript talking to the agent, what did it send back, what did the human send back. You also want all of the metadata around it, so that means the tool calls, the file system calls, the MCPs that it's connecting to. Everything that connects the agent to the outside world should also be inside of that agent trace when you start doing this analysis. And you also want the outputs to be in there. That can just be the code diffs that live inside of all of your conversations. [5:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=341s) It can also be the actual pull requests on the other side if you, like, pin the commit and look at the diff. In any case, you just wanna make sure that you have three things. You got the transcript of what was said, you've got the tool calls that show what the agent did, and then you have the output, which is something that you may want to grade for quality purposes. Um- Uh, and once you have all of that, you can start actually scoring the quality. Or rather, you can have an agent start scoring the quality. Um, so I like doing a, a show of hands for this little part. Um, how many [6:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=371s) people in here feel like their agents write too many tests? Too many tests. Okay. I expected more, but for those at home, it's about like 40% of people, which I think is a pretty big number. On our team, it was about 100% of hands. Uh, it can depend on your model and it can depend on your instructions. But let's say that your team is noticing your agents are writing a lot of tests, maybe extraneous ones, maybe you have your own opinions about, do you need end-to-end test coverage versus integration coverage [6:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=401s) versus unit test coverage? People are experimenting with all sorts of things there, so you might want to set up a scorer for that called Redundant Test that evaluates how often is the agent writing too many tests across each conversation that it has. Uh, another thing that you may want to score is, um, the way the agent communicates with you. So maybe another show of hands. How many people feel like their agents write too many comments? Yeah. Curious about this one. Yeah, some of them are bad. Yeah. I'm not gonna call out the model name, but I, I [7:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=431s) have an idea. It, it could depend. Uh, but this is also a common issue with agents, and it, it's not just for noise or for visuals, because we all know that code review is kind of the biggest bottleneck right now. It does matter how well the agent communicates changes, both summarizing them and also in the comments that it leaves within the code itself, because they can be barriers to actually get a code change reviewed quickly. Um, so you may have another metric like verbosity, which is what we call it, which is, is the agent [7:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=461s) overly verbose when it's talking to you or when it's writing code? Both of those are examples of kind of the same thing, which is a handle on language. Um, so you could set up a score that sort of defines that. And a third one I like to bring up is just efficiency of the agent, and this speaks to the need for tool calls and MCP calls in your traces. Because when you're running things a little bit, uh, without a human in the loop, so you're g- sending them off on longer and longer runs, maybe they're running in a cloud sandbox, so you're not looking [8:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=491s) at the conversation itself, it becomes easier and easier to miss all of the failed tool calls that an agent does. Like, an agent these days are very persistent and will work around whatever roadblock that they run into, and they won't tell you about it. They'll just tell you that it worked on the other side. And one common failure mode that we ran into was, like, in our test harness, there were a ton of bad instructions that were out of date in the skills, but the agent worked around them and never told us because it figured it out just by trying to ping a bunch of port numbers, figure out its environment, and then eventually start doing code. [8:42](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=522s) So the most that we knew is like, "Oh, the agent takes, like, two minutes longer than maybe we expect," but under the hood, there's actually a lot to be improved. So that's something you can evaluate as a scorer to just say, looking through the tool calls, is it efficiently getting to the solution or is it spinning its wheels a little bit? And I can show you what one of these scores looks like. Uh, it's just Markdown, as you would probably expect. Uh, in our setup here, it'll kind of be like a specialized skill definition where you can define what thing you're trying to score with an [9:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=551s) LLM-as-a-judge approach. Um, so here we're evaluating efficiency in this scorer, saying whether the agent worked towards the result or wasted effort on redundant steps and unnecessary back and forth. And then we set some labels that define what success and failure look like on that criteria. So if you remember back in high school writing five-paragraph essays and the teacher gives you a rubric of, like, you wrote a good essay if you have exactly five paragraphs and you only use active voice instead of passive, that sort of thing, it is still kind of [9:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=581s) helpful to put an agent through those paces for things that can be pretty clearly defined, efficiency being one of them. Um, so we set up one of these rubrics to have that teacher agent go through and decide, do you get good marks on efficiency, or is it kind of below the bar? And you can decide what pass and fail look like. In our case, maybe the bound is halfway through this list, so if it's a .4 or .2, that's kind of a failing grade. Something went wrong in that conversation. And above it, it is a passing grade and probably doesn't need [10:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=611s) to be evaluated. Things went mostly okay. And then down here is the actual definition of what that rubric means, and it's gonna restate some stuff that you would expect. So cost to human, the information gathering phase, routine step overhead. And you should keep this as succinct as possible and just build it up as you go. Um, this is a template that we like to give to people, but you can set it up with literally just, like, a couple bullet points and just your rubric and test it out yourself. It's more of an art than a science. Um, but [10:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=641s) once you have all these scores set up, then you can start having agents actually grade the conversations. So over here, let's say we have, uh, some sample sessions that we want agents to look through, uh, starting with, like, an auth service that the agent implemented in a full stack context. Let's say that we had a full agent conversation about building a full stack implementation of this, and the PR was merged, and now it's time for an agent to go grade how that conversation went. Well, the agent will spin up. In our case, [11:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=671s) we're saying one agent per conversation or one agent run per conversation run. You can set up a random sampling for, you know, just the sake of tokens, or you can run it across every conversation. Really depends. Um, but let's say in a random sampling, it pulled out this one and then told an agent to grade across each of those criteria. So walking down the line, it might say, looking at the test coverage, it wrote a ton of unit tests for the auth service when our guidelines say that you should mostly prefer end-to-end test. And so maybe [11:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=701s) we give it a fail because it didn't really follow the instructions that we expected. And then maybe for verbosity, it was actually pretty clear. It was using a new model selection that doesn't write code comments. Uh, go GPT. Anyone, anyone in the room? I don't know. Uh, but let's say that it didn't really write too many comments. It was very easy to understand the PR, so it gets a pass. And then moving on to efficiency, let's say it was trying to do a lot of exploration in its test environment to figure out how to test this change in the first place when really it should have had [12:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=731s) some upfront instruction on that, and that would probably be a failing grade. On that score. And again, pass and fail is just did it fall below the line on the rubric that we set? Uh, and then you can just go through each of your agent conversations and continue scoring them. So automatically, usually the trigger being a PR merged, kick in an agent to review that conversation across your grading criteria. And then over time, you'll start aggregating a lot of data on how conversations go, and each of these will [12:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=761s) include like the pass and failure rates. So did it pass and fail across each definition? And then you also have an agent describe why one of those failures or passes happened, and then this becomes context that another agent can use to decide what improvements to make. So the way that we set it up, going back to this wheel, is the idea you grade conversations as soon as they happen, and then maybe on a cron job every twenty-four hours, go through a sampling of the past twenty-four hours of conversations. Maybe there's been like two hundred. [13:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=791s) Filter it down to the failures and then pass in all of those failing reports that describe what failed and why to a self-improvement agent and tell it, "These were the fifty conversations that failed. Here are the notes that were left on each conversation. Go ahead and find commonalities. Here's access to our skills repository and the code base if you wanna make architecture changes." Um, so in this little snippet I showed, maybe it adds a coverage pyramid because redundant tests are popping up across a lot of PRs, and I can show you a couple other examples, [13:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=821s) um, both in a skill that you can take home and also the automated system that we have inside of Warp Factories. So this is gonna be the easiest one that y'all get to try. Um, so it's called Skill Doctor. It's something you can find at, uh, warp.dev/skilldoctor. I'll show the link in a second. Um, but this is a skill that you can run on your single player setup, and it'll crawl through all of your agent conversations in Claude, Codex, Py, or whatever else, and just do a one-time grading. So if you've been wondering about all the hidden gems in your conversations because you haven't done [14:10](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=850s) this approach before, you can just run this across your setup and see what happens. So this was a sample I ran across some Codex sessions building this project here, and we can see on the side that it grades in batches for a little bit of token efficiency, and it came up with some score reports across all of the metrics that we care about, and this test environment has efficiency and code quality as the ones that it evaluates. Uh, and then on the other side, the self-improvement agent looks at all those scores and comes back with a report. Um, so let's go [14:40](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=880s) ahead and take a look at that. Uh, I like to show the animation. We love it. Uh, but this is like kind of your skill report card that shows how the agent performed across all of those rubrics that we defined. Again, this is one shot. It's probably something that you wanna do continuously on your setup rather than whenever someone remembers. But as a first go to just see what you can optimize right now, it's nice to get some low-hanging fruit. Um, so in this example, it graded across these metrics and noticed that these were a little bit lower than expected, but verbosity was fine. Um, [15:10](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=910s) so it went through all of the notes that were given by the scoring agents and then suggested some changes to the skill makeup, and this one is going to mirror what I was mentioning earlier, uh, where it noticed that the agent was usually trying to fight for port numbers on the machine that it was testing, and it didn't really understand how the environment was set up. So this proposed a change to the end-to-end testing skill to have the agent get better instructions on finding a port number first and then going for it instead of just banging against a wall until it got something running. Um, so that can save a little bit of efficiency, [15:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=941s) uh, in the tool calls. It also noticed that I constantly wanted to write a different change log report with the language that was specified, and so it rewrote my change log skill to have a more, uh, prescriptive language on how these things are written, so I don't have to do that manual back and forth every time an agent opens a PR. Um, and these were things that I could have gone and found myself, but especially this one on port numbers, since I'm not watching the traces super closely, it's usually just behind like a working toggle if you're using like an agent harness or it's behind a cloud. Uh, [16:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=971s) it's nice to pick up on things like this so you don't miss them. Um, so last thing that I'll leave you with is how you can run this continuously over time. Uh, oh, shoot, it was just there. No! I lost access to my staging. Uh, well, I can at least describe with words what that was. Uh, which is the set of sort of graphs on the scores that are written. Because one thing-- I was actually explaining this to someone who uses like a memory system in their harness. Uh, and memories are a really useful [16:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1001s) way of having agents like introspect. Maybe you've heard of like dreaming overnight and thinking about how conversations went. It's very similar to what we talked about here, but the one part that's kind of missing from that is it's very invisible what the agent actually decided or what takeaways that it got from all the conversations. It's useful when you're using scoring to have an actual dashboard that explains how all of those conversations went. So I'll kind of draw it out here, where if you have like the efficiency [17:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1031s) over time, it's pretty easy to start aggregating data. I have not practiced this live. Let's try it. Uh, where maybe you can actually track over time how the efficiency score has been fluctuating. So the agent's constantly scoring setups, and it's giving grades of like fifty percent, sixty percent on average, and you can attribute changes that you made in your agent setup to inflections in the score. So not only are you having the agent improve themselves, you also have some proof points of like, oh, it shipped this change and I gotten [17:41](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1061s) this result out of it. So I actually know that when I merge that skill that it suggested, something real happened on the other side. So you could say, "Oh, there was an inflection point here. This is when I merged the skill, and I can see that efficiency went up by a certain percentage." By having scores in the loop with self-improvement, you can get that sort of view into things. And this is the dashboard that we have for proposing those self-improvement PRs. So in Warp Factories, if you set up like your software factory with this, instead of running it one time, the [18:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1091s) agent is continuously scoring conversations all the time, and then on a cron, an agent can look through scores and suggest PRs to improve your setup. So this is the inbox of things that need our team's review right now. This is actually our agent setup. So this is across all of the different agents that back our team, and we can take a look at one of them that was merged where it noticed an issue in the, uh, triage skill where we were missing some config to let it spawn sub-agents. So it was continuously like banging its head trying to figure out, "How do I [18:40](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1120s) make sub-agents?" And it just wastes a bunch of tool calls. This found it and suggested you wanna patch that part of your factory config next time. That'll save you some tokens. Um, so this is one of many examples that we've merged in that just make our agents a little bit more efficient and higher quality. Um, so with that, uh, if you're interested in any of these ideas, you can check out Warp Factories. It's an easy Google search. Um, or you can search for Warp Skill Doctor to try out that skill I mentioned as just a way to try this out on your own setup and see if you get a little bit of benefit out of it. With that, uh, [19:11](https://www.youtube.com/watch?v=pz74r-4dGfQ&t=1151s) thank you so much, and enjoy the rest of the conference.