# Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah

Tanmay Sah · Dolly Sah

[Talk page](https://ai.engineer/talks/BT1Mo-fD-g0-your-coding-agent-cant-undo-its-own) · [Video](https://www.youtube.com/watch?v=BT1Mo-fD-g0)

## What It Takes for a Coding Agent to Undo Its Work

Tanmay Sah and Dolly Sah use a lost ticket acknowledgment and a changed timeout to show how UndoBench measures recovery failures and how EvoUndo tests whether an agent can reverse a change to its own execution environment.

### The tests pass, the release is tagged, and an integration fails

[0:15](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=15s)

Tanmay Sah starts with a coding agent working on a repository issue. It applies a patch, runs the tests and creates a release tag. Everything passes until a downstream integration fails. At that point the agent faces a different assignment. Can it return everything it touched to the state it was in before the patch?

Git can tell the agent what changed in the repository. The same piece of work may also have made an external API call, altered a runtime configuration or sent a notification. Each of those effects can outlive the code that triggered it. Running the task again could repeat an action that already succeeded, even if the integration still reports a failure.

### UndoBench pairs normal execution with an injected failure

[1:21](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=81s)

Tanmay and Dolly Sah designed UndoBench to ask whether an agent that can complete a task can also recover when execution goes wrong. Across 2,880 paired trials, they compared execution under normal conditions with execution under an injected failure. The normal runs established which tasks the agent could do in the first place.

Under normal conditions, the success rate was 84%. The researchers then looked specifically at trials whose normal run had succeeded. When a failure was injected, only 47% of that group recovered successfully. The denominator matters. The second figure answers a question a practitioner might ask after seeing a successful demonstration: given that the agent knows how to do this work, how often can it recover from an interruption?

![Two panels show that 2,406 of 2,880 pairs passed the nominal control. Of those 2,406 pairs, 1,124 recovered when a fault was injected.](https://ai.engineer/api/talk-essay-media/1ce275d1fca8affee81f0ccce2b9daae64bda4c625f09f3dd0956a945ee15f5e)

Can a capable agent recover from failure?

### Ticket 781 exists before the agent sees a timeout

[2:22](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=142s)

An agent sends a request to an external system to create one ticket. The server receives the request and creates ticket 781. On the return trip, the acknowledgment is lost. The server now holds the ticket, while the agent has only a timeout. From the agent’s side, the missing response leaves the outcome of the write unresolved.

The agent sends the create request again without checking whether the first write committed. The server creates a second ticket. Both requests did what the agent asked, yet the result violates its original intent to create exactly one. In UndoBench, naïve retries produced duplicate effects in 54% of trials of this kind.

![An UndoBench trace connects a coding agent to a remote service. After a lost acknowledgment, the service shows two committed tickets, #781 and #782. Naïve retries produced duplicate effects in 512 of 960 trials.](https://ai.engineer/api/talk-essay-media/2772d334188927f4923f62f3c18daef111da51a4661ed72821a10132935e39dc)

Timeout ≠ rollback

Tanmay distinguishes three points at which failure can occur. Before the mutation, retrying is generally safe because the state is still untouched. During the mutation, some changes may have taken effect, so the agent needs to inspect the current state. After the mutation, the full change may have committed even though its acknowledgment never arrived. Ticket 781 illustrates why a timeout alone gives the agent too little evidence to choose among those cases.

### A Git revert reaches only part of the agent’s environment

[4:43](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=283s)

A repository change has a repository-level recovery mechanism. A deployment or database write has taken effect in a different system. Even within the running agent, a tool registry, a piece of middleware or a configuration may need its own restoration operation. To select that operation, the agent has to identify where the effect actually occurred.

![Three panels show the outcomes of a Git revert. The repository timeout returns to 30, the updated tool registry stays active, and a deployment persists.](https://ai.engineer/api/talk-essay-media/fdcb82aee8e5d8015bb6e6777380d011d6bd16c1b5cb9c83d514146baa37fb54)

Git revert is only one kind of undo

This becomes especially important when an agent modifies its own harness, the environment that governs its execution. Tanmay includes prompts, tools, middleware, event listeners, resources and routing configurations in that environment. A modification can help the agent perform its next task while making the modification itself difficult to reverse. Testing the new capability exercises only one direction of the change.

### EvoUndo records the value a reversal will need

[6:04](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=364s)

Suppose an agent sets a timeout to 60. Knowing the new value tells it little about what to write during recovery. In Tanmay’s example the timeout used to be 30. EvoUndo captures that information before the mutation in a pre-state witness. The witness supplies both the value the recovery operation needs and a reference for checking the resulting state.

The researchers evaluated 600 self-evolution tasks. Of those, 478 produced changes that improved the agent’s capability. Recovery checks found defects in 197 of those improvements, or 41.2% of the capability-positive cases. Requiring both a capability gain and successful recovery left 281 admissible improvements with the original recovery mechanism.

### Repair needs the right state information and an operation that can use it

[7:35](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=455s)

Could the agent fix its undo plan if a verifier told it the plan had failed? The researchers tried four verifier-guided repair strategies. With the original recovery language, none recovered any of the 197 naturally occurring failures. A verifier could detect a bad result while still leaving the agent short of what it needed to produce a good one.

Tanmay separates two bottlenecks. The first, state grounding, concerns what changed and where. When the researchers supplied precise information about the change, successful recoveries in one experimental subgroup rose from zero to 38 out of 48. The second, recovery expressivity, concerns which operations the recovery language allows. Even with a known target, the agent needs an operation capable of restoring it. Introducing richer undo operations raised recovery in a different subgroup from zero to 142 out of 143.

![Two panels compare state grounding, which raised recoveries from 0 to 38 of 48 in the base-language-solvable subgroup, and recovery expressivity, which raised recoveries from 0 to 142 of 143 in the rich-language-only subgroup.](https://ai.engineer/api/talk-essay-media/ba0ffa627388bfa3bfd84d4ab60f1e3f61fb01db38cf8ea90ed122b1698dcf13)

Two bottlenecks. Two engineering levers.

The overall result uses the original set of 197 defective mutations. After extending the recovery language, EvoUndo recovered 180 of them (91.4%). Those 180 already represented capability improvements. Passing recovery brought the admissible total from 281 to 461 out of all 600 tasks, a gain of 30 percentage points. The remaining 17 defective mutations still fell short of admission.

### Dolly Sah walks a proposed mutation through admission

[10:35](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=635s)

Dolly takes over with the EvoUndo algorithm. The agent proposes a mutation against its current state, together with an effect contract and a recovery plan. Before applying the mutation, EvoUndo captures a witness of the original state. In a testing environment, the agent applies the proposed change and executes the recovery plan. An independent verifier then checks whether the original state has been restored.

Admission requires two results. The mutation must improve capability according to user-defined criteria, and it must pass the recovery check. If it falls short, the agent can attempt a repair or the verifier can reject the mutation. This puts the recovery decision at the point where the system is still considering the proposed change, before deployment.

![A flowchart follows a proposed harness mutation through state capture, mutation and undo. Independent verification compares the recovered state with the original before the decision to admit, repair or reject the change.](https://ai.engineer/api/talk-essay-media/7c95fb7f2386d3457cd72c07ef50ac1aa5dc29fe142a5e22ca1b67d2adf52714)

EvoUndo: recoverability-constrained admission

### The timeout test starts and ends with a read

[12:00](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=720s)

Dolly traces the timeout example in execution order. EvoUndo first reads 30 from the current configuration and saves it as the witness. The agent then writes 60, the requested mutation. When recovery runs, it uses the saved value as its restoration target. An independent verifier reads the resulting configuration and compares it with the original 30.

Dolly says 60 once while describing the restore step. Her stated verification rule resolves the slip: a recovered value must match the original value captured before the change. A readback of 60 would show that the mutation was still in effect. Merely observing that the undo command ran would miss that failure.

![The configuration file shows a timeout of 30 beside a state witness recording the same value. Verify is highlighted as the fifth step, and the round-trip check reports a pass for the modeled state.](https://ai.engineer/api/talk-essay-media/d8aaafb6ebff682266aabcf1aa04fa460a4213228aa5dcb2b30be0a6e5645991)

EvoUndo in action: witnessed recovery

### Choose among retry, restore, compensate and escalate

[13:33](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=813s)

Dolly distinguishes four responses. Retry when the system is known to be untouched and the desired action still needs to happen. Restore when the original state is available and the agent can put it back, as with the witnessed timeout. Compensate when an external action has already committed and a further action is needed to reverse its effect. Escalate when the agent cannot establish that recovery is safe or when the verifier rejects the recovery. In that last case, the agent stops.

The two research projects address different decisions in that process. UndoBench asks what happens to an otherwise successful execution when a failure interrupts it. EvoUndo moves the recovery test into evaluation of a proposed harness change. Before granting the agent an improvement, the system can require evidence that the improvement works and that the agent can recover from applying it.

## Key takeaways

- After a timeout, check whether the first write committed before repeating an operation that creates an external record.
- Capture a pre-state witness before mutation and give the recovery plan an operation that can restore the witnessed state.
- Gate harness changes on both user-defined capability gains and an independent check of the recovered state.

## Slides

### The Missing Undo Button for Coding Agents

[0:09](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=9s) [A robot sits beside a log showing a timeout change from 30 to 600. Downstream checks fail, and the Undo button is disabled.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0e73ec25a7e71912c67ce77779a50cc5a1246ef8585610ea6aceefcfe959b0c.webp)

The Missing Undo Button for Coding Agents
Tanmay Sah, PhD · Dolly Sah
AI Engineer
Code Summit

agent-runtime / live-execution.log
● LIVE
$ agent.modify("service/config.py")
− timeout = 30
+ timeout = 600
✓ Patch applied. Changes committed.
! Downstream checks failed.
$ agent.undo()
× UNDO NOT AVAILABLE

UNDO
RECOVERY MISSING
Changes remain applied.

AGENT STATUS    RECOVERY NOT FOUND
1 / 17

### Everything passed. Can you get back?

[0:12](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=12s) [An execution log shows passing tests and a release tag before a downstream integration fails. Two panels describe effects outside Git and a timeout after a remote commit.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c3d25318cb3f002eb3f079047f447ccdd534ac950e0fcd6df31e2739662c406.webp)

Everything passed. Can you get back?

agent-runner · execution log
↻ REPLAY
$ agent fix --issue 214
✓ patch applied
✓ tests passing
✓ release tag created
× downstream integration failed
? what exactly changed outside Git?

Source control knows the diff
But not necessarily API effects, modified tool registries, runtime listeners or issued notifications.

A retry is not an undo
A timeout can arrive after a remote change has already committed.

2 / 17

### Can a capable agent recover from failure?

[1:19](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=79s) [Two panels show that 2,406 of 2,880 pairs passed the nominal control. Of those 2,406 pairs, 1,124 recovered when a fault was injected.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ce275d1fca8affee81f0ccce2b9daae64bda4c625f09f3dd0956a945ee15f5e.webp)

Can a capable agent recover from failure?

83.54%
Nominal control pass rate
2,406 / 2,880

46.72%
Recovery among nominally capable pairs
1,124 / 2,406

Same task intent. Fault injected into execution. A successful agent is not necessarily a recoverable agent.

3 / 17

### Timeout ≠ rollback

[3:19](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=199s) [An UndoBench trace connects a coding agent to a remote service. After a lost acknowledgment, the service shows two committed tickets, #781 and #782. Naïve retries produced duplicate effects in 512 of 960 trials.](https://assets.ai.engineer/sites/aiecode2025/sha256/2772d334188927f4923f62f3c18daef111da51a4661ed72821a10132935e39dc.webp)

Timeout ≠ rollback

UNDOBENCH / EXECUTION TRACE
FAULT: LOST_ACK · SIDE EFFECT: CREATE TICKET

CODING AGENT
DUPLICATE

TOOL CALL / NETWORK
TWO DURABLE EFFECTS

REMOTE SERVICE
2 COMMITS

✓ TICKET #781
COMMITTED
! TICKET #782
SECOND COMMIT
DUPLICATE EFFECT

53.33%
Duplicate effects from naïve retries
512 / 960 trials

CONTINUE →    NEXT SLIDE →    ↻ REPLAY    05 / 05

One request. Two tickets. A timeout doesn’t mean rollback.

4 / 17

### The failure phase changes the answer

[4:12](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=252s) [Three panels pair the timing of a failure with a response. Before mutation, redispatch is possible. During mutation, inspect and reconcile. After a commit with a missing acknowledgment, verify or deduplicate.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb162e3faae6abc8f4b736120ef66b160182b9399958458970ac22a3b657b644.webp)

The failure phase changes the answer

BEFORE mutation
No durable effect has occurred.
Safe redispatch is possible.

DURING mutation
External state may be partially updated.
Inspect and reconcile.

AFTER commit · no ACK
The operation may have completed, but the agent does not know.
Verify or deduplicate.

One generic retry policy cannot solve three different failure modes.

5 / 17

### Git revert is only one kind of undo

[4:38](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=278s) [Three panels show the outcomes of a Git revert. The repository timeout returns to 30, the updated tool registry stays active, and a deployment persists.](https://assets.ai.engineer/sites/aiecode2025/sha256/fdcb82aee8e5d8015bb6e6777380d011d6bd16c1b5cb9c83d514146baa37fb54.webp)

Git revert is only one kind of undo

WHAT DOES GIT REVERT ACTUALLY UNDO?

01 / REPOSITORY
Repository state
Source files · commits · branches
− timeout = 600
+ timeout = 30
✓ RESTORED BY GIT

02 / AGENT HARNESS
Runtime state
Tool registry · middleware · prompts · configuration
registry.update(tool_v2)
↳ still active
! REQUIRES STATE-AWARE UNDO

03 / OUTSIDE WORLD
External effects
Deployments · DB writes · messages · payments
POST /deploy → 200 OK
↳ effect persists
! REQUIRES EFFECT RECOVERY

Undo must match the effect. Reverting code does not reverse every action the agent took.

6 / 17

### Coding agents also change the harness

[5:21](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=321s) [Six cards identify parts of a coding agent’s harness. Prompts, tools, middleware and listeners are outlined in gold. A banner explains that a change can erase the information needed to reverse it.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a9bf989c255fa152d239ac44a3312ae98482f296144c4481b5d0e97a8d06e5c.webp)

Coding agents also change the harness

PROMPTS
templates and policies

TOOLS
registrations and schemas

MIDDLEWARE
ordered filters

LISTENERS
event bindings

RESOURCES
files and sockets

ROUTING
execution defaults

A forward-improving mutation can still destroy the information needed to reverse it.

7 / 17

### Undo needs more than an inverse command

[6:03](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=363s) [Four cards trace a timeout from 30 to 60 and back to 30. A witness records that the setting existed and held 30 before the change. A code strip shows the verified restoration.](https://assets.ai.engineer/sites/aiecode2025/sha256/88d8e375a198cdcc6f095269921d131354bb98754d16fb7569f71d1fd3b44d46.webp)

Undo needs more than an inverse command

Example: set timeout=60. What was there before?

PRE-STATE
timeout = 30

WITNESS
existed=true · old=30

MUTATION
timeout = 60

RECOVERY
restore timeout=30

config/runtime.ts                              RECOVERY VERIFIED
− timeout: 30  + timeout: 60  ✓ restored timeout: 30

Witness
Capture the previous value, existence and relevant structure
before changing it.

Receipt + typed undo
Use post-mutation allocation receipts when handles are created;
unwind dependent changes in reverse order.

8 / 17

### How Often Do Improved Harnesses Fail Recovery?

[6:57](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=417s) [Three cards show 600 unseen tasks, 478 capability-positive mutations and 197 of those mutations failing recovery verification. The banner gives the failure rate before repair as 41.2% of capability-positive mutations.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf2da314e59f0959bce758a5dd844f24f3fab47450ad495fd2ce09c8270396af.webp)

How Often Do Improved Harnesses Fail Recovery?

600
Unseen self-evolution tasks
6 families × 100

478
Capability-positive mutations

197
Capability-positive but failed recovery
verification

41.2% of capability-positive mutations were recovery-defective before repair.

9 / 17

### Why “please fix the undo” did not work

[7:34](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=454s) [The left card shows that each verifier-guided strategy rescued 0 of 197 failures under L0. The right card explains that the runtime’s recovery language limits which operations an LLM can produce.](https://assets.ai.engineer/sites/aiecode2025/sha256/52ea69ee16a7efb823eeb276d79b4f32bc2d4484e2b7bed55249c73a797e88ba.webp)

Why “please fix the undo” did not work

0 / 197
Natural failures rescued by each verifier-guided strategy under base
recovery language L0

A key distinction
An LLM cannot synthesize an operation that the runtime
recovery language does not allow.

More prompting does not create missing semantics.

The bottleneck may be missing state evidence—or missing expressive recovery operations.

10 / 17

### Two bottlenecks. Two engineering levers.

[8:32](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=512s) [Two panels compare state grounding, which raised recoveries from 0 to 38 of 48 in the base-language-solvable subgroup, and recovery expressivity, which raised recoveries from 0 to 142 of 143 in the rich-language-only subgroup.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba0ffa627388bfa3bfd84d4ab60f1e3f61fb01db38cf8ea90ed122b1698dcf13.webp)

Two bottlenecks. Two engineering levers.

1 · STATE GROUNDING
Point to exact state locations, residuals, and effect ordering.

0 → 38 / 48
in base-language-solvable subgroup

2 · RECOVERY EXPRESSIVITY
Make middleware, listeners, files, resources and ordered
reversal representable.

0 → 142 / 143
in rich-language-only subgroup

Important: adding more grounded feedback was not always better under the rich language.

11 / 17

### Can Better Undo Mechanisms Rescue Failed Mutations?

[9:29](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=569s) [With coarse diagnostics, the original language rescued 0 of 197 failed mutations and the extended language rescued 180 of 197 (91.4%). Three smaller cards show that admissible mutations rose from 281 to 461 out of 600, a gain of 30.0 percentage points.](https://assets.ai.engineer/sites/aiecode2025/sha256/6021c1d45e433599dfc36cac9f6b56c409c227e543cd81b173ff9fc7f9900bbf.webp)

Can Better Undo Mechanisms Rescue Failed Mutations?

0 / 197
Coarse diagnostics + original language

180 / 197
Coarse diagnostics + extended language
91.4% rescued

281 / 600
Initially admissible

461 / 600
Admissible after L1 repair

+30.0 pp
Absolute total yield gain

These are EvoUndo harness-mutation results—not UndoBench benchmark improvements.

12 / 17

### EvoUndo: recoverability-constrained admission

[11:55](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=715s) [A flowchart follows a proposed harness mutation through state capture, mutation and undo. Independent verification compares the recovered state with the original before the decision to admit, repair or reject the change.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c95fb7f2386d3457cd72c07ef50ac1aa5dc29fe142a5e22ca1b67d2adf52714.webp)

EvoUndo: recoverability-constrained admission
A forward improvement is admitted only when its recovery is verified.

01 / PROPOSE
Harness mutation
Forward candidate + declared effect contract
ξ = (m, w, u, Cₑ)

02 / RECORD
State witness
Capture pre-state evidence and receipts
w = witness(S₀)

03 / TEST
Mutate → Undo
Execute candidate and typed recovery
S₀ → S₁ → Ŝ₀

04 / INDEPENDENT VERIFICATION
Recovered state ≡ original state?
Effect coverage · IID and OOD recovery tests · Admission threshold

✓ ADMIT
Capability and recovery criteria pass
↻ REPAIR / REJECT
Improve the recovery specification—or do not admit the change

Simplified algorithm overview: empirical state-equivalence tests are scoped to the evaluated effects and sampled states; not a proof of universal reversibility.

EVOUNDO ALGORITHM
Admit when recovery criteria pass; otherwise repair or reject.
13 / 17

### EvoUndo in action: witnessed recovery

[12:48](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=768s) [The configuration file shows a timeout of 30 beside a state witness recording the same value. Verify is highlighted as the fifth step, and the round-trip check reports a pass for the modeled state.](https://assets.ai.engineer/sites/aiecode2025/sha256/d8aaafb6ebff682266aabcf1aa04fa460a4213228aa5dcb2b30be0a6e5645991.webp)

EvoUndo in action: witnessed recovery
WORKED EXAMPLE   One reversible harness mutation — not a generic inverse command

harness/config.py   VERIFIED
01  # Agent runtime configuration
02  config = {
03    "timeout": 30,
04    "retry": True
05  }

STATE WITNESS
previous.timeout = 30
Captured before execution. Retained for exact restoration.
WITNESS CAPTURED

$ verify(recovered == original)
Independent round-trip verification passes for the modeled state.

01 READ ORIGINAL
02 SAVE WITNESS
03 MUTATE
04 UNDO
05 VERIFY
✓ VERIFIED: recovered timeout = original

Illustrative example. EvoUndo's admission process also tests effect coverage and recoverability on held-out states.

EVOUNDO // LIVE EXECUTION
Independently verify restored timeout equals 30.
14 / 17

### Retry, restore, compensate—or stop

[13:32](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=812s) [Four panels list the conditions for retrying, restoring, compensating or escalating. A result below reports 300/300 for effect-scoped snapshots versus 243/300 for EvoUndo on different-surface changes in memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/c285e21f63adb620e8c7cf55a42c1188736f79d7e5193b0935af27dc42a00051.webp)

Retry, restore, compensate—or stop

RETRY
Only when the operation is known safe to repeat or protected by appropriate idempotency.

RESTORE
Use exact effect-scoped snapshots when state is serializable and directly restorable.

COMPENSATE
For external effects where exact restoration is impossible but a valid compensating action exists.

STOP / ESCALATE
When commit status is unknown and neither observability nor authorized recovery is sufficient.

Effect-scoped snapshots beat EvoUndo in its evaluated in-memory comparison: 300/300 vs 243/300 on different-surface changes.

15 / 17

### Three research takeaways

[14:56](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=896s) [Three numbered statements distinguish task competence from recovery, show EvoUndo’s four-step sequence and require recovery to pass before admitting a change. References for UndoBench and EvoUndo appear below.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ec7b15bfc4ff4536bc4e32e7e8221e3e213707e222fce9c01f9476aac64a760.webp)

Three research takeaways

01 UndoBench: task competence ≠ recovery.
02 EvoUndo algorithm: witness → mutate → recover → verify.
03 Admit evolution only when recovery passes.

UndoBench
arxiv.org/abs/2610.05622

EvoUndo
arxiv.org/abs/2608.28363

16 / 17

## Full transcript

[0:01](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=1s) Hi everyone, I am Tanmay. Today joining me is Dolly, and we will be presenting the missing undo button for coding agents.

[0:15](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=15s) So, imagine a coding agent working on a repository issue. It applies patch, run the tests, and everything passes. It even creates a release tag, but then a downstream integration fails. Now, here is the question. Can the agent actually get everything back to the state it was in before? So, Git can tell us what

[0:45](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=45s) changed in the repository, but what about changes outside Git, such as external API calls or runtime configurations or notifications, and simply retrying an operation does not necessarily undo what already happened. So we wanted to understand not just whether agents can complete tasks, but whether they can recover when something goes wrong.

[1:21](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=81s) This brings us to our first research work, UndoBench. We wanted to investigate whether an agent's ability to complete a task also translates into its ability to recover from failure. We evaluated agents across two thousand eight hundred eighty paired trials comparing normal execution with execution under an injected failure. So under normal

[1:50](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=110s) conditions, the agent achieved an eighty-four percent success rate, but when we introduced failures, only forty-seven percent of the normally successful trials recovered successfully. Notice that these two percentage though have different denominators. But yeah, so even when an agent, uh, can complete a task successfully, recovery remains a significant challenge.

[2:22](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=142s) Now, let's look at a specific failure scenario. Imagine a coding agent sending a request to an external system to create a ticket. The request has been sent, and the agent is waiting for a response.

[2:41](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=161s) Next, the server receives the request and successfully creates ticket number seven eighty-one, but the acknowledgment gets lost on its way to the agent. From the agent's perspective, the request has timed out, but the agent does not know that the ticket has already been created. A timeout tells us that the response didn't arrive, not that the res-- operation failed.

[3:12](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=192s) So the agent tries the request without checking whether the opera-- whether the first operation committed. And now we have two tickets, even though the agent intended to create only one. So in UndoBench, naive retries produced duplicate effects in fifty-four percent of these kind of trials. That's why we need to distinguish a failed response from a

[3:42](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=222s) failed operations.

[3:49](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=229s) Now, the failure phase also changes the answer. What do I mean is that recovery also depends on when the failure happens. Before the mutation, retrying is generally safe because nothing has changed. But during the mutation, the system has-- may be partially modified, so we need to inspect its state. And after the mutation, the

[4:18](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=258s) operation may have committed, even if the acknowledgment was lost. As we just saw, retrying could create duplicates. So we need different recovery strategies depending on the failure phase.

[4:41](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=281s) Now,

[4:43](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=283s) why can't we just use Git revert? So Git can restore repository changes, but coding agents also modify their runtime state, including tool registries, middleware, and configurations. They can also trigger external effects, such as deployment or database writes. So Git revert does not reverse all these changes. The recovery

[5:13](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=313s) mechanism has to match the type of effect the agent created.

[5:23](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=323s) And this becomes particularly important when coding agents start modifying their own execution environment or what we call the agent hardness. This includes prompts, tools, middleware, event listeners, resources, and routing configurations. Such changes may improve the agent's performance, but they can also make recovery more difficult. In other words, an agent

[5:53](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=353s) can become more capable while becoming less recoverable.

[6:04](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=364s) And that's why, and that's why undo needs more than an inverse command. Uh, now, like undoing a change is not always as simple as reversing a command. Suppose an agent changes a timeout from thirty to sixty to undo that change, it needs to know that the original value was thirty. That's why EVO Undo captures a pre-state witness

[6:34](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=394s) before applying the mutation, and that witness provides the information to restore the original state and later verify that, uh, verify that recovery actually worked. Now, question comes, how often, uh, do improved hardnesses fail? So in EVO Undo, we evaluated six hundred self-evolution tasks.

[7:04](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=424s) Out of those, four hundred and seventy-eight produced capability improvements, but one hundred and ninety-seven of those improvements had recovery defects. That is around forty-two percent of the capability positive cases. So here key takeaway is that a mutation can improve an agent's capability and still leave the system unable to recover reliably.

[7:35](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=455s) Now, now we can say that why please fix the undo did not work. So naturally, we wanted to use prompting and wanted to tell agent that, "Hey, could you simply repair all these recovery failures?" So we tested four verifier-guided rep-repair strategies, but under the original recovery language, none recovered the one hundred and ninety-seven naturally occurring failures. The

[8:05](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=485s) problem was not necessarily the agent's reasoning. In some cases, the agent lacked precise information about the original state. In, uh, in others, the recovery language could not, uh, express the operation needed to reverse the change. So we investigated these two limitations separately.

[8:34](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=514s) So two bottlenecks. First is state grounding, and another one is recovery expressivity. The-- When we provided like precise information about what had changed and where, then recovery improved from zero to thirty-eight out of forty-eight cases. And the second is recovery expressivity. So when we introduced richer undo operation, recovery improved significantly from

[9:04](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=544s) zero to one hundred and forty-two out of one hundred and forty-three cases. So these results come from tho-two different experimental subgroups, but together they show why recovery needs both accurate state information and ability to perform the required undo operation.

[9:30](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=570s) Now, question comes to our mind that can better undo mechanism rescue failed mutations. So what happens like when we give the agent better recovery capabilities? So with the original recovery language, none of the one hundred and ninety-seven recovery defective mutations could be restored. But after extending the recovery language, we successfully recovered one hundred and eighty out of one hundred and ninety s-seven cases. That is

[10:00](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=600s) around ninety-two percent. As a result, the number of admissible improvement increased from two hundred and eighty-one to four six- sixty-one out of six hundred tasks. That is a thirty percentage point improvement, and that brings us to the next question. How do we build these recovery checks into the agent's evolution process? I will now hand it over to Dolly to walk through

[10:30](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=630s) the EVO Undo algorithm.

[10:35](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=635s) All right. Thanks, Tanmay. Uh, so let's get into the working on how EVO Undo actually works. So first we will start with the agent pro... Uh, first, the agent will start. The agent pro-processes the mutation on which state it is currently in, along with the effect contract and the recovery plan. And before applying the mutation, we capture the state witness representing the original state.

[11:05](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=665s) And even before that, the-- then a testing in the testing environment, agent will apply the mutation and execute the recovery plan. Uh, an independent verifier will also verify the original s- uh, if, uh, verify to check if the original state has been restored or not. If the mutation improves, uh, the capability based on the user-defined criteria and passes the recovery, um, check, the agents admit the changes. Otherwise,

[11:35](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=695s) well, uh, the agent attempt to repair or the verifier itself rejects the mutation altogether. This way, the recoverability becomes a part of the admission decision rather than something that we strategize or even plan once we are, uh, deployed the, deployed the AI agent.

[11:55](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=715s) Okay, so with next...

[12:00](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=720s) Next, we will walk through as, as Tanmay have already explained in an-- g-given an example, let, let me walk through how the ex-- uh, how that example of a timeout changing from thirty to sixty would look like in action. So let's take Let's start with the original state, a simple example that, uh... simple example of, um, our agent having the time value of 30. Before changing anything, the EVO Undo would first read the original value, and then

[12:30](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=750s) it will save the-- it will capture the original value as a state witness. We know exact-- We-- So we know exactly what need to be restored when, when, and-- when if we have to restore it, we would know what, what need to be restored or which state would be the true state to return back to. Now the agent... So next come the mutation. Now the agent changes the timeout from thirty to 60, as we want it to do. And if recovery is required, uh, if

[13:00](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=780s) recovery is required, the undo uses the saved witness to restore the timeout to 60. Now, independently, the verifier-- finally, an independent verifier would also confirm that the, uh, recovered value matches the original value. So the recovery isn't considered successful simply because an undo command was run. But also the verifier would, uh, verify the resulting state to make sure that we are in fact doing the re-- uh, recovery, those in the state

[13:30](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=810s) we want it to be.

[13:33](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=813s) Well, now comes the bigger question. So, so we have this capability of recovery in our arsenal. Should we be using it for uniformly and applying the recovery mechanism to all the failures? Well, of course not.  Um, not every failures can be recovered the same way. If nothing has changed, then the, uh, then simply if, if nothing has changed in the, in the system, we would simply want it to, um, recover, uh, we, we would simply want it to safely

[14:03](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=843s) retry. We would want the agent to simply retry if, uh, and try to achieve the state that we want to be. However, if we have the original state, uh, once we have the original state, we can restore it. If the external action has already been committed and we need, uh... It's then when we would need a compensation mechanism, com-compensation action to comp-compensating action to reverse its effect. And if we cannot recover

[14:33](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=873s) the recovery, uh, and we cannot-- if, if and if we cannot, uh, cannot verify that recovery is safe or the verifier have rejected the recovery, then the agent should stop and escalate. The right response definitely depends on what actually happened and what we-- what can be safely reversed or not.

[14:57](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=897s) That brings me to the almost the last slide, which is about bringing everything together on our two paper, Undo Bench and EVO Undo. The-- So the key takeaways here are, first, the eng-- Undo Bench shows the successful task execution. Uh, the Undo Bench shows that a successful task execution doesn't guarantee-- doesn't always guarantee the successful recovery of it. Uh, secondly, the EVO Undo algorithm

[15:27](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=927s) provides a structured process, capture the original state, apply the mutation, recover, independently verify, and restoration. So the EVO Undo algorithm does work in the, in the fash-- in the way we intended it to. And third, we should admit self-evaluation changes only when the-- when both the capability and the recoverability requirements are met. So ultimately, as we give agent more autonomy, we should also give them more, uh, reliable ways to recover from their

[15:56](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=956s) actions.

[16:00](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=960s) All right.  So, and finally, I hope you remember the agent from the beginning. It was looking for an undo button. Well, now it has one. We have started building this capability through our research project, EVO Undo, and you can explore the work through the-- in, in the GitHub repository. You can scan the QR code on the screen to get to the page. Our goal is to make the recovery a fundamental capability of AI agents, so that we not only have, uh, the, the agent does not even

[16:30](https://www.youtube.com/watch?v=BT1Mo-fD-g0&t=990s) exec-- does not only execute the action, but also recover when something goes wrong as well. Thank you so much.
