Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic

Lance Martin built a queue of about 50 synthetic customer support tickets and simulated a migration from Opus 4.8 to Opus 5.5. He introduced familiar prompt-writing habits one at a time, including instructions to think step by step, double-check the answer and be maximally thorough. Averaged across those experiments, switching models improved the score a little and lowered cost. Auditing the instructions after the switch improved both again.

The old instructions had served a purpose. Engineers used scratchpads to give earlier models somewhere to reason and verification steps to get them to revisit an answer. Martin found that frontier Claude models already make many of those decisions themselves. A blanket demand for thoroughness can induce another round of verification whose tokens you pay for on every ticket. Contradictions cause a different failure. An older model might quietly ignore one of two conflicting rules. A model that follows instructions more closely can struggle to satisfy both.

Martin uses the prompt-audit skill in Claude Code to look through prompts, skills and tool instructions for these patterns. The comparison worth making is the one his experiment isolates: run the new model with the inherited prompt, then run it on the same cases after removing redundant reasoning instructions and resolving conflicts. He offered no precise model-generation cutoff for retiring these techniques. The audit belongs in the migration because the answer depends on the model that will actually receive the prompt.

A tool call might add only a little information to an agent session, yet the next request carries much of the same context as the previous one. Claude’s prompt cache can reuse that work at a lower cost and with less latency. The constraint is an exact prefix match. Change a byte early in the request and the cache can reuse only the portion before that change, even if everything following it is identical.

Martin orders context by how widely it can be reused. Shared system instructions and tool definitions come first. Project-specific material such as CLAUDE.md and project memory follows. Session state comes later, and messages accumulate at the end on each turn. With that order, a change of project can preserve the shared prefix, and a new message can preserve the project context. The Claude Console exposes the cache hit rate, so engineers can check what their request construction actually achieves.

Martin and colleagues on Anthropic’s applied AI team inspected the default harnesses for several internal benchmarks. Some made poor use of the cache. They found opportunities to reduce cost by up to 72% with no meaningful change in measured performance. That figure was the largest opportunity they found in those harnesses. The relevant measurement for another application is how much repeated input it currently sends past the point where its prefixes diverge.

An evaluation with an 80% score appears to leave 20 percentage points of room for improvement. Read the misses first. If the request is ambiguous or the grader rejects a valid answer, the apparent headroom can consume an entire optimization budget. Martin looks for four properties before turning an optimizer loose: stronger models generally score higher, additional thinking generally helps, the best model still has solvable failures, and repeated runs give similar scores.

Production traces help decide what belongs in the set. For a support application, a product owner can specify what matters and ask Claude to flag conversations where the application appears to fall short. Martin favors having the owner inspect the raw conversations before promoting examples. Selecting every case one version of Sonnet happens to miss can produce a suite tailored to that model’s particular weaknesses. A later model might solve the entire pocket of failures while leaving other important support problems untouched.

Use a deterministic grader when the task allows one. For open-ended work, Martin starts with distinctions a human can apply consistently. Several binary judgments give a grader clearer decisions than an unexplained score from one to ten. For generated interfaces, he has seen evaluators compare two candidate pages directly, a choice people can often make more reliably than assigning either page an absolute grade. Multiple grades can still work when a reviewer can explain the difference between adjacent levels. Run the grader repeatedly and across models to see whether those distinctions hold up.

Hill climbing here means running an evaluation, inspecting failures, changing something in the application and measuring again. Martin’s Claude API skill initially scored 66% on questions grounded in Anthropic’s documentation. He gave Claude Code the skill text and the documentation so it could trace wrong answers back to the guidance the skill supplied. Revising that text improved the score substantially. It was a tractable experiment because the evaluated model read the very file the optimizer could edit. Allowing broad changes to the agent harness would have made each round harder to interpret.

That same ease of editing makes prompts easy to overfit. An optimizer can pick up distinctive wording from benchmark questions. Given control over tools, it might even add OCR for one example when production users never submit that kind of task. Martin’s loop splits the evaluation into training and test cases. Claude sees only training failures when choosing its edits. The runner scores both splits each round. A rising training score alongside a flat test score gives the engineer an early reason to inspect the changes. Both splits still depend on the original sample representing production.

The loop can also stall. After several unsuccessful rounds, Martin’s hill climber groups the remaining failures to look for shared causes. In one run it uncovered ambiguous cases and rewrote them. That finding calls for a different interpretation from a successful skill edit: clarifying the question changes what the evaluation measures. Set a budget for the search and inspect those cases before spending more tokens trying to make the application pass them.

Even a saturated evaluation can answer a useful question: how cheaply can the application maintain its current score? On the support benchmark, Martin started with Opus 4.8 at high effort and gave the hill climber a cost-reduction objective. Moving to Opus 5 at low effort raised performance and cut cost slightly. Another low-effort configuration cut cost more sharply but gave back some of the quality gain. A subsequent prompt edit recovered performance. Martin reported a 90% cost reduction for the final configuration and a higher score than the starting point.

Effort guides how much reasoning Claude devotes to a problem. Token consumption still depends on the task. On a hard coding problem, for example, extra verification at high effort can use substantially more tokens for much the same outcome. A stronger model can therefore cost less per completed task despite charging more per token. Martin starts by trying capable models at low effort and increases effort when the evaluation warrants it. The quality check matters: in the support experiment, Haiku 4.5 at low effort produced a substantial regression.

A prompt edit is quick to make. If the task being evaluated takes hours, every edit can mean hours before the next score arrives. A sandbox failure midway through the night can also destroy the evidence needed to find out what happened. Claude Managed Agents addresses this by separating the harness that directs the work, the sandbox that executes actions and an independent, append-only session log. An orchestrator can replace a failed harness or sandbox, and the session history survives either failure.

Keeping the log independent also helps when a healthy agent compacts its context. At the time of summarization, the agent has to guess which details it will need later. With a queryable session log, it can retrieve an earlier detail that the summary omitted. Credentials have a separate boundary. Tools can use secrets held in a vault while Claude refers to them without seeing their values.

For engineers evaluating a multi-turn agent, that history supports two different questions. A grader can inspect the final outcome, or it can examine the sequence of actions that produced it. Martin described a continuing production loop in which an online evaluator flags poor traces, a person selects consequential cases for the offline suite, and resolved failures become regression tests. The choice that still needs an owner is what counts as a bad run in the first place.