Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp

The first clue at Warp was that a coding agent seemed to take about two minutes longer than expected. The agent could still finish the task. Inside the run, though, it was trying port after port to figure out its test environment before it could get on with the code. Its testing skill contained stale instructions. Once the agent found a way around them, it reported success. The next agent inherited the same instructions.

Ben Holmes, a DevX engineering lead at Warp, uses that distinction to decide what deserves review. An engineer reviewing the pull request can judge the code the agent produced. To repair the testing skill, someone also has to see how the agent got there. Warp assigns that second review to another agent, which grades the completed session and explains its findings. Recurring findings then give the team a reason to change the setup that every subsequent run will use.

For an individual engineer, the local session histories from Claude Code or Codex are enough to start. Team-wide review requires a place to read sessions across engineers. Holmes has seen teams zip up local Claude histories and collect them in S3. A shared runner offers another collection point. Warp runs agents in isolated Docker sandboxes, but tasks flow through a common system where their traces can be collected.

Each record needs three kinds of evidence. The transcript establishes what the human asked for and what the agent said it did. Tool calls and their results show the steps it took, including filesystem operations and calls to MCP tools. The output gives the reviewer something to inspect, such as a code diff from the session or a pull request pinned to a particular commit. A reviewer investigating the port problem needs the sequence of calls. One judging whether an agent wrote redundant tests needs the diff and the testing instructions that applied to it.

Holmes starts with behaviors the team already spends time reviewing. For redundant tests, the question is whether the coverage fits the team’s expectations for unit, integration, and end-to-end testing. For verbosity, the judge reads both the summary and the comments in the code. The cost of an overly long explanation shows up when a reviewer has to work through it to understand the change.

Efficiency needs its own rubric because the same readable, well-tested PR could have taken a circuitous route. Warp’s Markdown rubric asks the judge to consider the cost to the human, the effort the agent spent gathering information, and the overhead of routine steps. It describes levels of performance and gives the judge a cutoff. In Holmes’s example, scores of 0.4 and 0.2 warrant investigation. He recommends starting with a couple of bullets and refining them against sessions. The cutoff expresses the team’s judgment about what merits a closer look.

Consider Holmes’s hypothetical merged PR for a full-stack authentication service. The agent wrote extensive unit tests despite guidance favoring end-to-end coverage, so it could fail the testing criterion. Its concise comments and clear PR could earn a verbosity pass. If it also spent several calls rediscovering how to run the tests, it could fail efficiency. Those separate grades preserve a useful distinction for whoever owns the skills. One failure concerns a preference the agent overlooked, while the other suggests an instruction it needed and never received.

Warp typically triggers the grading run when a PR merges. A judge reviews one conversation against the criteria and records a reason alongside each grade. Teams can score every session or take a random sample to manage token costs. A separate agent can then run every 24 hours, read the failing reports from that period, and look for a common cause. Access to the skills repository lets it propose an edit where the instruction lives.

Holmes demonstrated a one-time version of that process with Skill Doctor on his local Codex history. It graded sessions in batches. Repeated struggles with port selection led it to propose a change to the end-to-end testing skill so the agent would find an available port before trying to start the service. A different pattern showed up in the human messages. Holmes kept asking agents to revise the wording of changelog entries. Skill Doctor rewrote the changelog skill to specify the language he had repeatedly requested. The tool calls supplied the evidence for one edit and the human corrections supplied it for the other.

In Warp’s continuous setup, proposed changes arrive as pull requests for the team to review. Holmes showed one that the team had merged. Its triage skill called for spawning sub-agents, but the configuration lacked the setting that enabled them. The triage agent kept spending tool calls trying to discover how to carry out the instruction. The proposed PR patched the configuration, addressing a failure that clearer wording alone could never resolve.

Continuing to grade sessions after a merge makes the effect inspectable. Holmes described plotting efficiency scores over time and marking when a skill change landed, so the team could look for a shift in later runs. His percentages illustrated how to read such a graph. He gave no measured gain for the port or triage fixes. For the port change, the trace also offers a direct check on the score: in later sessions that use the testing skill, does the agent find an available port on the first attempt, or does it still have to probe?