Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah

Tanmay Sah starts with a coding agent working on a repository issue. It applies a patch, runs the tests and creates a release tag. Everything passes until a downstream integration fails. At that point the agent faces a different assignment. Can it return everything it touched to the state it was in before the patch?

Git can tell the agent what changed in the repository. The same piece of work may also have made an external API call, altered a runtime configuration or sent a notification. Each of those effects can outlive the code that triggered it. Running the task again could repeat an action that already succeeded, even if the integration still reports a failure.

Tanmay and Dolly Sah designed UndoBench to ask whether an agent that can complete a task can also recover when execution goes wrong. Across 2,880 paired trials, they compared execution under normal conditions with execution under an injected failure. The normal runs established which tasks the agent could do in the first place.

Under normal conditions, the success rate was 84%. The researchers then looked specifically at trials whose normal run had succeeded. When a failure was injected, only 47% of that group recovered successfully. The denominator matters. The second figure answers a question a practitioner might ask after seeing a successful demonstration: given that the agent knows how to do this work, how often can it recover from an interruption?

An agent sends a request to an external system to create one ticket. The server receives the request and creates ticket 781. On the return trip, the acknowledgment is lost. The server now holds the ticket, while the agent has only a timeout. From the agent’s side, the missing response leaves the outcome of the write unresolved.

The agent sends the create request again without checking whether the first write committed. The server creates a second ticket. Both requests did what the agent asked, yet the result violates its original intent to create exactly one. In UndoBench, naïve retries produced duplicate effects in 54% of trials of this kind.

Tanmay distinguishes three points at which failure can occur. Before the mutation, retrying is generally safe because the state is still untouched. During the mutation, some changes may have taken effect, so the agent needs to inspect the current state. After the mutation, the full change may have committed even though its acknowledgment never arrived. Ticket 781 illustrates why a timeout alone gives the agent too little evidence to choose among those cases.

A repository change has a repository-level recovery mechanism. A deployment or database write has taken effect in a different system. Even within the running agent, a tool registry, a piece of middleware or a configuration may need its own restoration operation. To select that operation, the agent has to identify where the effect actually occurred.

This becomes especially important when an agent modifies its own harness, the environment that governs its execution. Tanmay includes prompts, tools, middleware, event listeners, resources and routing configurations in that environment. A modification can help the agent perform its next task while making the modification itself difficult to reverse. Testing the new capability exercises only one direction of the change.

Suppose an agent sets a timeout to 60. Knowing the new value tells it little about what to write during recovery. In Tanmay’s example the timeout used to be 30. EvoUndo captures that information before the mutation in a pre-state witness. The witness supplies both the value the recovery operation needs and a reference for checking the resulting state.

The researchers evaluated 600 self-evolution tasks. Of those, 478 produced changes that improved the agent’s capability. Recovery checks found defects in 197 of those improvements, or 41.2% of the capability-positive cases. Requiring both a capability gain and successful recovery left 281 admissible improvements with the original recovery mechanism.

Could the agent fix its undo plan if a verifier told it the plan had failed? The researchers tried four verifier-guided repair strategies. With the original recovery language, none recovered any of the 197 naturally occurring failures. A verifier could detect a bad result while still leaving the agent short of what it needed to produce a good one.

Tanmay separates two bottlenecks. The first, state grounding, concerns what changed and where. When the researchers supplied precise information about the change, successful recoveries in one experimental subgroup rose from zero to 38 out of 48. The second, recovery expressivity, concerns which operations the recovery language allows. Even with a known target, the agent needs an operation capable of restoring it. Introducing richer undo operations raised recovery in a different subgroup from zero to 142 out of 143.

The overall result uses the original set of 197 defective mutations. After extending the recovery language, EvoUndo recovered 180 of them (91.4%). Those 180 already represented capability improvements. Passing recovery brought the admissible total from 281 to 461 out of all 600 tasks, a gain of 30 percentage points. The remaining 17 defective mutations still fell short of admission.

Dolly takes over with the EvoUndo algorithm. The agent proposes a mutation against its current state, together with an effect contract and a recovery plan. Before applying the mutation, EvoUndo captures a witness of the original state. In a testing environment, the agent applies the proposed change and executes the recovery plan. An independent verifier then checks whether the original state has been restored.

Admission requires two results. The mutation must improve capability according to user-defined criteria, and it must pass the recovery check. If it falls short, the agent can attempt a repair or the verifier can reject the mutation. This puts the recovery decision at the point where the system is still considering the proposed change, before deployment.

Dolly traces the timeout example in execution order. EvoUndo first reads 30 from the current configuration and saves it as the witness. The agent then writes 60, the requested mutation. When recovery runs, it uses the saved value as its restoration target. An independent verifier reads the resulting configuration and compares it with the original 30.

Dolly says 60 once while describing the restore step. Her stated verification rule resolves the slip: a recovered value must match the original value captured before the change. A readback of 60 would show that the mutation was still in effect. Merely observing that the undo command ran would miss that failure.

Dolly distinguishes four responses. Retry when the system is known to be untouched and the desired action still needs to happen. Restore when the original state is available and the agent can put it back, as with the witnessed timeout. Compensate when an external action has already committed and a further action is needed to reverse its effect. Escalate when the agent cannot establish that recovery is safe or when the verifier rejects the recovery. In that last case, the agent stops.

The two research projects address different decisions in that process. UndoBench asks what happens to an otherwise successful execution when a failure interrupts it. EvoUndo moves the recovery test into evaluation of a proposed harness change. Before granting the agent an improvement, the system can require evidence that the improvement works and that the agent can recover from applying it.