[
  {
    "startMs": 1140,
    "text": "Hi everyone, I am Tanmay. Today joining me is Dolly, and we will be presenting the missing undo button for coding agents."
  },
  {
    "startMs": 15420,
    "text": "So, imagine a coding agent working on a repository issue. It applies patch, run the tests, and everything passes. It even creates a release tag, but then a downstream integration fails. Now, here is the question. Can the agent actually get everything back to the state it was in before? So, Git can tell us what"
  },
  {
    "startMs": 45260,
    "text": "changed in the repository, but what about changes outside Git, such as external API calls or runtime configurations or notifications, and simply retrying an operation does not necessarily undo what already happened. So we wanted to understand not just whether agents can complete tasks, but whether they can recover when something goes wrong."
  },
  {
    "startMs": 81020,
    "text": "This brings us to our first research work, UndoBench. We wanted to investigate whether an agent's ability to complete a task also translates into its ability to recover from failure. We evaluated agents across two thousand eight hundred eighty paired trials comparing normal execution with execution under an injected failure. So under normal"
  },
  {
    "startMs": 110740,
    "text": "conditions, the agent achieved an eighty-four percent success rate, but when we introduced failures, only forty-seven percent of the normally successful trials recovered successfully. Notice that these two percentage though have different denominators. But yeah, so even when an agent, uh, can complete a task successfully, recovery remains a significant challenge."
  },
  {
    "startMs": 142580,
    "text": "Now, let's look at a specific failure scenario. Imagine a coding agent sending a request to an external system to create a ticket. The request has been sent, and the agent is waiting for a response."
  },
  {
    "startMs": 161240,
    "text": "Next, the server receives the request and successfully creates ticket number seven eighty-one, but the acknowledgment gets lost on its way to the agent. From the agent's perspective, the request has timed out, but the agent does not know that the ticket has already been created. A timeout tells us that the response didn't arrive, not that the res-- operation failed."
  },
  {
    "startMs": 192380,
    "text": "So the agent tries the request without checking whether the opera-- whether the first operation committed. And now we have two tickets, even though the agent intended to create only one. So in UndoBench, naive retries produced duplicate effects in fifty-four percent of these kind of trials. That's why we need to distinguish a failed response from a"
  },
  {
    "startMs": 222500,
    "text": "failed operations."
  },
  {
    "startMs": 229280,
    "text": "Now, the failure phase also changes the answer. What do I mean is that recovery also depends on when the failure happens. Before the mutation, retrying is generally safe because nothing has changed. But during the mutation, the system has-- may be partially modified, so we need to inspect its state. And after the mutation, the"
  },
  {
    "startMs": 258800,
    "text": "operation may have committed, even if the acknowledgment was lost. As we just saw, retrying could create duplicates. So we need different recovery strategies depending on the failure phase."
  },
  {
    "startMs": 281340,
    "text": "Now,"
  },
  {
    "startMs": 283780,
    "text": "why can't we just use Git revert? So Git can restore repository changes, but coding agents also modify their runtime state, including tool registries, middleware, and configurations. They can also trigger external effects, such as deployment or database writes. So Git revert does not reverse all these changes. The recovery"
  },
  {
    "startMs": 313100,
    "text": "mechanism has to match the type of effect the agent created."
  },
  {
    "startMs": 323800,
    "text": "And this becomes particularly important when coding agents start modifying their own execution environment or what we call the agent hardness. This includes prompts, tools, middleware, event listeners, resources, and routing configurations. Such changes may improve the agent's performance, but they can also make recovery more difficult. In other words, an agent"
  },
  {
    "startMs": 353620,
    "text": "can become more capable while becoming less recoverable."
  },
  {
    "startMs": 364780,
    "text": "And that's why, and that's why undo needs more than an inverse command. Uh, now, like undoing a change is not always as simple as reversing a command. Suppose an agent changes a timeout from thirty to sixty to undo that change, it needs to know that the original value was thirty. That's why EVO Undo captures a pre-state witness"
  },
  {
    "startMs": 394480,
    "text": "before applying the mutation, and that witness provides the information to restore the original state and later verify that, uh, verify that recovery actually worked. Now, question comes, how often, uh, do improved hardnesses fail? So in EVO Undo, we evaluated six hundred self-evolution tasks."
  },
  {
    "startMs": 424620,
    "text": "Out of those, four hundred and seventy-eight produced capability improvements, but one hundred and ninety-seven of those improvements had recovery defects. That is around forty-two percent of the capability positive cases. So here key takeaway is that a mutation can improve an agent's capability and still leave the system unable to recover reliably."
  },
  {
    "startMs": 455620,
    "text": "Now, now we can say that why please fix the undo did not work. So naturally, we wanted to use prompting and wanted to tell agent that, \"Hey, could you simply repair all these recovery failures?\" So we tested four verifier-guided rep-repair strategies, but under the original recovery language, none recovered the one hundred and ninety-seven naturally occurring failures. The"
  },
  {
    "startMs": 485220,
    "text": "problem was not necessarily the agent's reasoning. In some cases, the agent lacked precise information about the original state. In, uh, in others, the recovery language could not, uh, express the operation needed to reverse the change. So we investigated these two limitations separately."
  },
  {
    "startMs": 514300,
    "text": "So two bottlenecks. First is state grounding, and another one is recovery expressivity. The-- When we provided like precise information about what had changed and where, then recovery improved from zero to thirty-eight out of forty-eight cases. And the second is recovery expressivity. So when we introduced richer undo operation, recovery improved significantly from"
  },
  {
    "startMs": 544340,
    "text": "zero to one hundred and forty-two out of one hundred and forty-three cases. So these results come from tho-two different experimental subgroups, but together they show why recovery needs both accurate state information and ability to perform the required undo operation."
  },
  {
    "startMs": 570480,
    "text": "Now, question comes to our mind that can better undo mechanism rescue failed mutations. So what happens like when we give the agent better recovery capabilities? So with the original recovery language, none of the one hundred and ninety-seven recovery defective mutations could be restored. But after extending the recovery language, we successfully recovered one hundred and eighty out of one hundred and ninety s-seven cases. That is"
  },
  {
    "startMs": 600480,
    "text": "around ninety-two percent. As a result, the number of admissible improvement increased from two hundred and eighty-one to four six- sixty-one out of six hundred tasks. That is a thirty percentage point improvement, and that brings us to the next question. How do we build these recovery checks into the agent's evolution process? I will now hand it over to Dolly to walk through"
  },
  {
    "startMs": 630340,
    "text": "the EVO Undo algorithm."
  },
  {
    "startMs": 635280,
    "text": "All right. Thanks, Tanmay. Uh, so let's get into the working on how EVO Undo actually works. So first we will start with the agent pro... Uh, first, the agent will start. The agent pro-processes the mutation on which state it is currently in, along with the effect contract and the recovery plan. And before applying the mutation, we capture the state witness representing the original state."
  },
  {
    "startMs": 665220,
    "text": "And even before that, the-- then a testing in the testing environment, agent will apply the mutation and execute the recovery plan. Uh, an independent verifier will also verify the original s- uh, if, uh, verify to check if the original state has been restored or not. If the mutation improves, uh, the capability based on the user-defined criteria and passes the recovery, um, check, the agents admit the changes. Otherwise,"
  },
  {
    "startMs": 695100,
    "text": "well, uh, the agent attempt to repair or the verifier itself rejects the mutation altogether. This way, the recoverability becomes a part of the admission decision rather than something that we strategize or even plan once we are, uh, deployed the, deployed the AI agent."
  },
  {
    "startMs": 715800,
    "text": "Okay, so with next..."
  },
  {
    "startMs": 720600,
    "text": "Next, we will walk through as, as Tanmay have already explained in an-- g-given an example, let, let me walk through how the ex-- uh, how that example of a timeout changing from thirty to sixty would look like in action. So let's take Let's start with the original state, a simple example that, uh... simple example of, um, our agent having the time value of 30. Before changing anything, the EVO Undo would first read the original value, and then"
  },
  {
    "startMs": 750760,
    "text": "it will save the-- it will capture the original value as a state witness. We know exact-- We-- So we know exactly what need to be restored when, when, and-- when if we have to restore it, we would know what, what need to be restored or which state would be the true state to return back to. Now the agent... So next come the mutation. Now the agent changes the timeout from thirty to 60, as we want it to do. And if recovery is required, uh, if"
  },
  {
    "startMs": 780420,
    "text": "recovery is required, the undo uses the saved witness to restore the timeout to 60. Now, independently, the verifier-- finally, an independent verifier would also confirm that the, uh, recovered value matches the original value. So the recovery isn't considered successful simply because an undo command was run. But also the verifier would, uh, verify the resulting state to make sure that we are in fact doing the re-- uh, recovery, those in the state"
  },
  {
    "startMs": 810440,
    "text": "we want it to be."
  },
  {
    "startMs": 813820,
    "text": "Well, now comes the bigger question. So, so we have this capability of recovery in our arsenal. Should we be using it for uniformly and applying the recovery mechanism to all the failures? Well, of course not.  Um, not every failures can be recovered the same way. If nothing has changed, then the, uh, then simply if, if nothing has changed in the, in the system, we would simply want it to, um, recover, uh, we, we would simply want it to safely"
  },
  {
    "startMs": 843680,
    "text": "retry. We would want the agent to simply retry if, uh, and try to achieve the state that we want to be. However, if we have the original state, uh, once we have the original state, we can restore it. If the external action has already been committed and we need, uh... It's then when we would need a compensation mechanism, com-compensation action to comp-compensating action to reverse its effect. And if we cannot recover"
  },
  {
    "startMs": 873540,
    "text": "the recovery, uh, and we cannot-- if, if and if we cannot, uh, cannot verify that recovery is safe or the verifier have rejected the recovery, then the agent should stop and escalate. The right response definitely depends on what actually happened and what we-- what can be safely reversed or not."
  },
  {
    "startMs": 897220,
    "text": "That brings me to the almost the last slide, which is about bringing everything together on our two paper, Undo Bench and EVO Undo. The-- So the key takeaways here are, first, the eng-- Undo Bench shows the successful task execution. Uh, the Undo Bench shows that a successful task execution doesn't guarantee-- doesn't always guarantee the successful recovery of it. Uh, secondly, the EVO Undo algorithm"
  },
  {
    "startMs": 927060,
    "text": "provides a structured process, capture the original state, apply the mutation, recover, independently verify, and restoration. So the EVO Undo algorithm does work in the, in the fash-- in the way we intended it to. And third, we should admit self-evaluation changes only when the-- when both the capability and the recoverability requirements are met. So ultimately, as we give agent more autonomy, we should also give them more, uh, reliable ways to recover from their"
  },
  {
    "startMs": 956780,
    "text": "actions."
  },
  {
    "startMs": 960160,
    "text": "All right.  So, and finally, I hope you remember the agent from the beginning. It was looking for an undo button. Well, now it has one. We have started building this capability through our research project, EVO Undo, and you can explore the work through the-- in, in the GitHub repository. You can scan the QR code on the screen to get to the page. Our goal is to make the recovery a fundamental capability of AI agents, so that we not only have, uh, the, the agent does not even"
  },
  {
    "startMs": 990200,
    "text": "exec-- does not only execute the action, but also recover when something goes wrong as well. Thank you so much."
  }
]