# An Agent's Change Is Not Safe Until Its Recovery Has Been Tested Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [04:26:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15980s) Today joining me is Dolly and we will be [04:26:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15983s) presenting the missing undo button for [04:26:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15986s) coding agents. [04:26:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15991s) So imagine a coding agent working on a [04:26:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15995s) repository issue. It applies patch run [04:26:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=15999s) the tests and everything passes. It even [04:26:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16003s) creates a release tag but then a [04:26:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16006s) downstream integration fails. Now here [04:26:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16009s) is the question. Can the agent actually [04:26:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16013s) get everything back to the state it was [04:26:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16016s) in before? [04:26:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16019s) So git can tell us what changed in the [04:27:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16022s) repository. But what about changes [04:27:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16025s) outside git such as external API calls [04:27:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16029s) or runtime configurations or [04:27:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16031s) notifications. [04:27:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16033s) And simply retrying an operation does [04:27:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16036s) not necessarily undo what already [04:27:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16039s) happened. So we wanted to understand not [04:27:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16042s) just whether agents can complete task [04:27:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16046s) but whether they can recover when [04:27:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16048s) something goes wrong. [04:27:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16057s) This brings us to our first research [04:27:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16060s) work undo bench. We wanted to [04:27:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16063s) investigate whether an agent's ability [04:27:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16066s) to complete a task also translates into [04:27:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16069s) its ability to recover from failure. We [04:27:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16073s) evaluated agents across 2,880 [04:27:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16077s) paired trials comparing normal execution [04:28:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16081s) with execution under an injected [04:28:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16084s) failure. So under normal conditions, the [04:28:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16087s) agent achieved an 84% success rate. But [04:28:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16091s) when we introduced failures only 47% [04:28:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16095s) of the normally successful trials [04:28:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16098s) recovered successfully. Notice that [04:28:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16101s) these two percentage though have [04:28:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16103s) different denominators. [04:28:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16105s) But yeah so even when an agent [04:28:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16109s) uh can complete a task successfully [04:28:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16112s) recovery remains a significant [04:28:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16114s) challenge. [04:28:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16118s) Now let's look at a specific failure [04:28:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16121s) scenario. Imagine a coding agent sending [04:28:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16125s) a request to an external system to [04:28:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16128s) create a ticket. The request has been [04:28:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16131s) sent and the agent is waiting for a [04:28:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16134s) response. [04:28:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16137s) Next, the server receives the request [04:28:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16139s) and successfully creates ticket number [04:29:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16142s) 781. [04:29:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16144s) But the acknowledgement gets lost on its [04:29:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16147s) way to the agent. [04:29:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16150s) From the agent's perspective, the [04:29:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16152s) request has timed out, but the agent [04:29:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16155s) does not know that the ticket has [04:29:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16157s) already been created. A timeout tells us [04:29:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16160s) that the response didn't arrive, not [04:29:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16163s) that the operation failed. [04:29:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16168s) So the agent tries the request without [04:29:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16171s) checking whether the oper whether the [04:29:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16173s) first operation committed [04:29:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16177s) and now we have two tickets even though [04:29:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16180s) the agent intended to create only one. [04:29:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16183s) So in undo bench knives [04:29:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16187s) produced duplicate effects in 54% of [04:29:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16191s) these kind of trials. That's why we need [04:29:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16194s) to distinguish a failed response from a [04:29:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16198s) failed operations. [04:30:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16205s) Now [04:30:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16207s) the failure phase also changes the [04:30:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16210s) answer. What do I mean is that recovery [04:30:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16214s) also depends on when the failure [04:30:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16217s) happens. Before the mutation retrying is [04:30:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16221s) generally safe because nothing has [04:30:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16223s) changed but during the mutation the [04:30:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16226s) system had may be partially modified. So [04:30:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16230s) we need to inspect its state and after [04:30:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16233s) the mutation the operation may have [04:30:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16236s) committed even if the acknowledgement [04:30:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16238s) was lost. As we just saw, retrying could [04:30:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16242s) create duplicates. So, we need different [04:30:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16246s) recovery strategies depending on the [04:30:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16249s) failure phase. [04:30:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16257s) Now, [04:31:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16260s) why can't we just use get reward? So, [04:31:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16264s) Git can restore repository changes, but [04:31:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16268s) coding agents also modify their runtime [04:31:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16272s) state including tool registries, [04:31:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16274s) middleware and configurations. They can [04:31:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16277s) also trigger external effects such as [04:31:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16280s) deployment or database rights. So, get [04:31:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16284s) rever does not reverse all these [04:31:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16287s) changes. The recovery mechanism has to [04:31:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16291s) match the type of effect the agent [04:31:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16293s) created. [04:31:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16299s) And this becomes particularly important [04:31:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16303s) when coding agents start modifying their [04:31:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16306s) own execution environment or what we [04:31:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16309s) call the agent hardness. This includes [04:31:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16313s) prompts, tools, middleware, event [04:31:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16316s) listeners, resources, and routing [04:31:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16318s) configurations. [04:32:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16320s) Such changes may improve the agent's [04:32:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16322s) performance, but they can also make [04:32:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16325s) recovery more difficult. In other words, [04:32:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16328s) an agent can become more capable while [04:32:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16332s) becoming less recoverable. [04:32:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16340s) Uh that's why uh that's why undo needs [04:32:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16345s) more than an inverse command. Uh now [04:32:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16348s) like undoing a change is not always as [04:32:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16351s) simple as reversing a command. Suppose [04:32:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16355s) an agent changes a timeout from 30 to [04:32:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16359s) 60. To undo that change, it needs to [04:32:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16362s) know that the original value was 30. [04:32:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16366s) That's why EVO undo captures a pre-state [04:32:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16369s) witness before applying the mutation and [04:32:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16373s) that witness provides the information to [04:32:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16376s) restore the original state and later [04:32:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16379s) verify that uh verify that recovery [04:33:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16384s) actually worked. [04:33:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16386s) Now [04:33:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16388s) question comes [04:33:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16390s) how often uh do improved hardnesses [04:33:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16393s) fail? So in Evo Undo, we evaluated 600 [04:33:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16398s) self- evvolution tasks. Out of those 478 [04:33:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16403s) produced capability improvements, but [04:33:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16406s) 197 of those improvements had recovery [04:33:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16410s) defects. That is around 42% of the [04:33:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16413s) capability positive cases. So here key [04:33:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16417s) takeaway is that a mutation can improve [04:33:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16420s) an agent's capability and still leave [04:33:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16424s) the system unable to recover reliably. [04:33:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16431s) Now now we can say that why please fix [04:33:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16435s) the undo did not work. So naturally we [04:33:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16439s) wanted to use prompting and wanted to [04:34:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16442s) tell agent that hey could you simply [04:34:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16444s) repair all these recovery failures. So [04:34:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16448s) we tested four verifier guided repair [04:34:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16452s) strategies but under the original [04:34:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16454s) recovery language none recovered the 197 [04:34:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16458s) naturally occurring failures. The [04:34:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16461s) problem was not necessarily [04:34:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16464s) the agent's reasoning. In some cases, [04:34:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16467s) the agent lacked precise information [04:34:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16470s) about the original state. In uh in [04:34:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16474s) others, the recovery language could not [04:34:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16478s) express the operation needed to reverse [04:34:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16481s) the change. So, we investigated these [04:34:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16484s) two limitations separately. [04:34:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16490s) So, two bottlenecks. [04:34:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16493s) First is state grounding and another one [04:34:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16495s) is recovery expressivity. [04:34:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16497s) The when we provided like precise [04:35:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16501s) information about what had changed and [04:35:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16504s) where then recovery improved from 0 to [04:35:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16507s) 38 out of 48 cases. And the second is [04:35:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16512s) recovery expressivity. So when we [04:35:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16514s) introduced richer undo operation [04:35:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16517s) recovery improved significantly from [04:35:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16520s) zero to 142 out of 143 cases. So these [04:35:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16524s) results come from the two different [04:35:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16527s) experimental tubros but together they [04:35:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16530s) show why recovery needs both accurate [04:35:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16534s) state information and ability to perform [04:35:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16537s) the required end to operation. [04:35:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16546s) Now question comes to our mind that can [04:35:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16549s) better undo mechanism rescue failed [04:35:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16552s) mutations. [04:35:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16553s) So what happens like when we give the [04:35:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16557s) agent better recovery capabilities. So [04:36:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16560s) with the original recovery language none [04:36:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16563s) of the 197 recovery defective mutations [04:36:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16566s) could be restored. But after extending [04:36:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16570s) the recovery language, we successfully [04:36:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16572s) recovered 180 out of 197 cases. That is [04:36:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16576s) around 92%. As a result, the number of [04:36:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16580s) admissible improvement increased from [04:36:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16583s) 281 to 461 [04:36:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16586s) out of 600 task. That is a 30 percentage [04:36:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16590s) point improvement. And [04:36:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16593s) that brings us to the next question. How [04:36:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16596s) do we build these recovery checks into [04:36:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16599s) the agent's evolution process? I will [04:36:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16603s) now hand it over to Dolly to walk [04:36:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16606s) through the EVO undo algorithm. [04:36:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16611s) All right, thanks San. Uh so let's get [04:36:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16614s) into the working on how Evo and actually [04:36:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16617s) works. So first we will start with the [04:37:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16620s) agent pro. Uh first [04:37:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16622s) the agent will start the agent pro [04:37:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16625s) processes the mutation on which state it [04:37:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16628s) is currently in along with the effect [04:37:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16629s) contract and the recovery plan. And [04:37:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16632s) before applying the mutation [04:37:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16635s) we captured the state witness [04:37:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16638s) representing the original state [04:37:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16641s) and even before that the then a testing [04:37:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16645s) in the testing environment agent will [04:37:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16646s) apply the mutation and execute the [04:37:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16648s) recovery plant uh an independent [04:37:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16651s) verifier will also verify the original [04:37:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16654s) if uh verify to check if the original [04:37:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16656s) state has been restored or not. If the [04:37:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16658s) mutation improves uh the capability [04:37:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16661s) based on the userdefined criteria and [04:37:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16664s) passes the recovery um check the agents [04:37:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16668s) admit the changes otherwise well [04:37:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16673s) uh the agent attempt to repair or the [04:37:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16675s) verifier itself rejects the mutation [04:37:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16677s) altogether. This way the recoveryability [04:38:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16680s) becomes a part of the admission decision [04:38:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16682s) rather than something that we strategize [04:38:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16684s) or even plan once we are uh deployed the [04:38:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16688s) deployed the AI agent. [04:38:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16692s) Okay. So with next [04:38:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16696s) we will walk through as as Tan have [04:38:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16699s) already explained in given an example [04:38:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16701s) let let me walk through how the ex uh [04:38:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16704s) how that example of a timer changing [04:38:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16706s) from 30 to 60 would look like in action. [04:38:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16709s) So let's take let's start with the [04:38:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16712s) original state the simple example that [04:38:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16714s) uh simple example of um our agent having [04:38:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16717s) the time value of 30 before changing [04:38:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16720s) anything the evoando would first read [04:38:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16723s) the original value [04:38:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16726s) and then it will save the it will [04:38:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16728s) capture the original value as a state [04:38:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16730s) witness. We know exact we so we know [04:38:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16733s) exactly what need to be restored when [04:38:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16736s) when and when if we have to restore it [04:38:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16738s) we would know what what need to be [04:39:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16740s) restored or which state would be the [04:39:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16742s) true state to return back to. Now the [04:39:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16746s) agent so next come the mutation. Now the [04:39:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16748s) agent changes the timeout from 30 to 60 [04:39:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16751s) as we want it to do. And if recovery is [04:39:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16754s) required uh if recovery is required the [04:39:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16758s) undo uses the saved witness to restore [04:39:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16760s) the timeout to 60. [04:39:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16762s) Now independently the verifier finally [04:39:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16765s) an independent verifier would also [04:39:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16766s) confirm that the uh recovered value [04:39:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16769s) matches the original value. So the [04:39:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16771s) recovery isn't considered successful [04:39:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16773s) simply because an undo command was run [04:39:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16776s) but also the verifier would verify the [04:39:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16779s) resulting state to make sure that we are [04:39:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16782s) in fact doing the uh recovery those in [04:39:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16786s) the state we want it to be. [04:39:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16790s) Well now comes the bigger question. So [04:39:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16793s) so we have this capability of recovery [04:39:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16795s) in our arsenal. Should we be using it [04:39:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16797s) for uniformly and applying the recovery [04:39:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16799s) mechanism to all the failures? Well, of [04:40:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16802s) course not. Um, not every failures can [04:40:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16805s) be recovered the same way. If nothing [04:40:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16807s) has changed, then the then simply if if [04:40:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16811s) nothing has changed in the in the [04:40:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16812s) system, we would simply want it to um [04:40:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16816s) recover uh we we would simply want it to [04:40:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16819s) safely retry. we would want the agent to [04:40:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16822s) simply retry if uh and try to achieve [04:40:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16825s) the state that we want to be. However, [04:40:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16827s) if we have the original state uh once we [04:40:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16830s) have the original state, we can restore [04:40:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16832s) it. If the external action has already [04:40:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16834s) been committed and we need uh [04:40:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16839s) it's then when we would need a [04:40:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16841s) compensation mechanism, compensation [04:40:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16843s) action to comp compensating action to [04:40:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16846s) reverse its effect. And if we cannot [04:40:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16848s) recover the recovery uh and we cannot if [04:40:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16852s) if and if we cannot the cannot verify [04:40:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16855s) that recovery is safe or the verifier [04:40:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16857s) have rejected the recovery then the [04:41:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16860s) agent should stop and escalate. The [04:41:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16863s) right response definitely depends on [04:41:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16864s) what actually happened and what we what [04:41:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16867s) can be safely reversed or not. [04:41:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16873s) That brings me [04:41:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16876s) to the almost the last slide which is [04:41:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16878s) about bringing everything together on [04:41:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16880s) our two paper undo bench and ebo undo. [04:41:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16882s) The so the key takeaways here are first [04:41:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16886s) the undo bench shows the successful task [04:41:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16889s) execution. [04:41:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16891s) Uh the undo bench shows that the [04:41:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16894s) successful task execution doesn't [04:41:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16896s) guarantee doesn't always guarantee the [04:41:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16898s) successful recovery of it. Uh secondly, [04:41:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16901s) the EVO undo algorithm provides a [04:41:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16903s) structured process. Capture the original [04:41:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16905s) state, apply the mutation, recover [04:41:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16907s) independently, verify and restoration. [04:41:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16910s) So the EVO undo algorithm does work in [04:41:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16913s) the in the fash in the way we intended [04:41:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16915s) it to. And third, we should admit [04:41:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16918s) self-evolution changes only when the [04:42:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16920s) when both the capability and the [04:42:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16922s) recoverability requirements are met. So [04:42:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16926s) ultimately as we give agent more [04:42:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16928s) autonomy, we should also give them more [04:42:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16931s) uh reliable ways to recover from their [04:42:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16932s) actions. [04:42:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16936s) All right. So and finally I hope you [04:42:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16939s) remember the agent from the beginning. [04:42:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16941s) It was looking for an undo button. Well, [04:42:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16944s) now it has one. We have started building [04:42:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16946s) this capability through our research [04:42:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16947s) project Evo undo. And you can explore [04:42:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16950s) the work through the in in the GitHub [04:42:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16952s) repository. can scan the QR code on the [04:42:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16956s) screen to get to the page. Our goal is [04:42:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16958s) to make the recovery a fundamental [04:42:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16960s) capability of AI agent so that we not [04:42:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16963s) only have the the agent does not even [04:42:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16966s) execute [04:42:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16968s) the action but also recover when [04:42:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16970s) something goes wrong as well. Thank you [04:42:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=16972s) so much.