1
04:26:20,399 --> 04:26:23,269
Today joining me is Dolly and we will be

2
04:26:23,279 --> 04:26:26,870
presenting the missing undo button for

3
04:26:26,880 --> 04:26:31,590
coding agents.

4
04:26:31,600 --> 04:26:35,510
So imagine a coding agent working on a

5
04:26:35,520 --> 04:26:39,910
repository issue. It applies patch run

6
04:26:39,920 --> 04:26:43,189
the tests and everything passes. It even

7
04:26:43,199 --> 04:26:46,389
creates a release tag but then a

8
04:26:46,399 --> 04:26:49,830
downstream integration fails. Now here

9
04:26:49,840 --> 04:26:53,590
is the question. Can the agent actually

10
04:26:53,600 --> 04:26:56,790
get everything back to the state it was

11
04:26:56,800 --> 04:26:59,349
in before?

12
04:26:59,359 --> 04:27:02,229
So git can tell us what changed in the

13
04:27:02,239 --> 04:27:05,269
repository. But what about changes

14
04:27:05,279 --> 04:27:09,269
outside git such as external API calls

15
04:27:09,279 --> 04:27:11,750
or runtime configurations or

16
04:27:11,760 --> 04:27:13,269
notifications.

17
04:27:13,279 --> 04:27:16,630
And simply retrying an operation does

18
04:27:16,640 --> 04:27:19,750
not necessarily undo what already

19
04:27:19,760 --> 04:27:22,790
happened. So we wanted to understand not

20
04:27:22,800 --> 04:27:26,309
just whether agents can complete task

21
04:27:26,319 --> 04:27:28,630
but whether they can recover when

22
04:27:28,640 --> 04:27:37,189
something goes wrong.

23
04:27:37,199 --> 04:27:39,990
This brings us to our first research

24
04:27:40,000 --> 04:27:43,349
work undo bench. We wanted to

25
04:27:43,359 --> 04:27:46,790
investigate whether an agent's ability

26
04:27:46,800 --> 04:27:49,910
to complete a task also translates into

27
04:27:49,920 --> 04:27:53,510
its ability to recover from failure. We

28
04:27:53,520 --> 04:27:57,670
evaluated agents across 2,880

29
04:27:57,680 --> 04:28:01,830
paired trials comparing normal execution

30
04:28:01,840 --> 04:28:04,550
with execution under an injected

31
04:28:04,560 --> 04:28:07,830
failure. So under normal conditions, the

32
04:28:07,840 --> 04:28:11,750
agent achieved an 84% success rate. But

33
04:28:11,760 --> 04:28:15,910
when we introduced failures only 47%

34
04:28:15,920 --> 04:28:18,710
of the normally successful trials

35
04:28:18,720 --> 04:28:21,670
recovered successfully. Notice that

36
04:28:21,680 --> 04:28:23,510
these two percentage though have

37
04:28:23,520 --> 04:28:25,670
different denominators.

38
04:28:25,680 --> 04:28:29,590
But yeah so even when an agent

39
04:28:29,600 --> 04:28:32,229
uh can complete a task successfully

40
04:28:32,239 --> 04:28:34,870
recovery remains a significant

41
04:28:34,880 --> 04:28:38,710
challenge.

42
04:28:38,720 --> 04:28:41,830
Now let's look at a specific failure

43
04:28:41,840 --> 04:28:45,269
scenario. Imagine a coding agent sending

44
04:28:45,279 --> 04:28:48,309
a request to an external system to

45
04:28:48,319 --> 04:28:51,349
create a ticket. The request has been

46
04:28:51,359 --> 04:28:54,070
sent and the agent is waiting for a

47
04:28:54,080 --> 04:28:57,349
response.

48
04:28:57,359 --> 04:28:59,910
Next, the server receives the request

49
04:28:59,920 --> 04:29:02,630
and successfully creates ticket number

50
04:29:02,640 --> 04:29:04,309
781.

51
04:29:04,319 --> 04:29:07,189
But the acknowledgement gets lost on its

52
04:29:07,199 --> 04:29:10,469
way to the agent.

53
04:29:10,479 --> 04:29:12,710
From the agent's perspective, the

54
04:29:12,720 --> 04:29:15,670
request has timed out, but the agent

55
04:29:15,680 --> 04:29:17,670
does not know that the ticket has

56
04:29:17,680 --> 04:29:20,950
already been created. A timeout tells us

57
04:29:20,960 --> 04:29:23,750
that the response didn't arrive, not

58
04:29:23,760 --> 04:29:28,630
that the operation failed.

59
04:29:28,640 --> 04:29:31,590
So the agent tries the request without

60
04:29:31,600 --> 04:29:33,910
checking whether the oper whether the

61
04:29:33,920 --> 04:29:37,269
first operation committed

62
04:29:37,279 --> 04:29:40,149
and now we have two tickets even though

63
04:29:40,159 --> 04:29:43,670
the agent intended to create only one.

64
04:29:43,680 --> 04:29:47,590
So in undo bench knives

65
04:29:47,600 --> 04:29:51,349
produced duplicate effects in 54% of

66
04:29:51,359 --> 04:29:54,790
these kind of trials. That's why we need

67
04:29:54,800 --> 04:29:58,630
to distinguish a failed response from a

68
04:29:58,640 --> 04:30:05,429
failed operations.

69
04:30:05,439 --> 04:30:07,349
Now

70
04:30:07,359 --> 04:30:10,469
the failure phase also changes the

71
04:30:10,479 --> 04:30:14,070
answer. What do I mean is that recovery

72
04:30:14,080 --> 04:30:17,189
also depends on when the failure

73
04:30:17,199 --> 04:30:21,349
happens. Before the mutation retrying is

74
04:30:21,359 --> 04:30:23,910
generally safe because nothing has

75
04:30:23,920 --> 04:30:26,630
changed but during the mutation the

76
04:30:26,640 --> 04:30:29,990
system had may be partially modified. So

77
04:30:30,000 --> 04:30:33,830
we need to inspect its state and after

78
04:30:33,840 --> 04:30:36,149
the mutation the operation may have

79
04:30:36,159 --> 04:30:38,710
committed even if the acknowledgement

80
04:30:38,720 --> 04:30:42,229
was lost. As we just saw, retrying could

81
04:30:42,239 --> 04:30:46,229
create duplicates. So, we need different

82
04:30:46,239 --> 04:30:49,429
recovery strategies depending on the

83
04:30:49,439 --> 04:30:57,510
failure phase.

84
04:30:57,520 --> 04:30:59,990
Now,

85
04:31:00,000 --> 04:31:04,149
why can't we just use get reward? So,

86
04:31:04,159 --> 04:31:08,229
Git can restore repository changes, but

87
04:31:08,239 --> 04:31:11,990
coding agents also modify their runtime

88
04:31:12,000 --> 04:31:14,790
state including tool registries,

89
04:31:14,800 --> 04:31:17,510
middleware and configurations. They can

90
04:31:17,520 --> 04:31:20,309
also trigger external effects such as

91
04:31:20,319 --> 04:31:24,870
deployment or database rights. So, get

92
04:31:24,880 --> 04:31:27,510
rever does not reverse all these

93
04:31:27,520 --> 04:31:31,110
changes. The recovery mechanism has to

94
04:31:31,120 --> 04:31:33,670
match the type of effect the agent

95
04:31:33,680 --> 04:31:39,910
created.

96
04:31:39,920 --> 04:31:43,590
And this becomes particularly important

97
04:31:43,600 --> 04:31:46,630
when coding agents start modifying their

98
04:31:46,640 --> 04:31:49,510
own execution environment or what we

99
04:31:49,520 --> 04:31:53,269
call the agent hardness. This includes

100
04:31:53,279 --> 04:31:56,149
prompts, tools, middleware, event

101
04:31:56,159 --> 04:31:58,389
listeners, resources, and routing

102
04:31:58,399 --> 04:31:59,990
configurations.

103
04:32:00,000 --> 04:32:02,950
Such changes may improve the agent's

104
04:32:02,960 --> 04:32:05,510
performance, but they can also make

105
04:32:05,520 --> 04:32:08,950
recovery more difficult. In other words,

106
04:32:08,960 --> 04:32:12,950
an agent can become more capable while

107
04:32:12,960 --> 04:32:20,950
becoming less recoverable.

108
04:32:20,960 --> 04:32:25,030
Uh that's why uh that's why undo needs

109
04:32:25,040 --> 04:32:28,469
more than an inverse command. Uh now

110
04:32:28,479 --> 04:32:31,910
like undoing a change is not always as

111
04:32:31,920 --> 04:32:35,030
simple as reversing a command. Suppose

112
04:32:35,040 --> 04:32:39,269
an agent changes a timeout from 30 to

113
04:32:39,279 --> 04:32:42,309
60. To undo that change, it needs to

114
04:32:42,319 --> 04:32:46,229
know that the original value was 30.

115
04:32:46,239 --> 04:32:49,750
That's why EVO undo captures a pre-state

116
04:32:49,760 --> 04:32:53,429
witness before applying the mutation and

117
04:32:53,439 --> 04:32:56,309
that witness provides the information to

118
04:32:56,319 --> 04:32:59,910
restore the original state and later

119
04:32:59,920 --> 04:33:04,150
verify that uh verify that recovery

120
04:33:04,160 --> 04:33:06,470
actually worked.

121
04:33:06,480 --> 04:33:08,230
Now

122
04:33:08,240 --> 04:33:10,230
question comes

123
04:33:10,240 --> 04:33:13,349
how often uh do improved hardnesses

124
04:33:13,359 --> 04:33:18,709
fail? So in Evo Undo, we evaluated 600

125
04:33:18,719 --> 04:33:23,109
self- evvolution tasks. Out of those 478

126
04:33:23,119 --> 04:33:26,390
produced capability improvements, but

127
04:33:26,400 --> 04:33:30,070
197 of those improvements had recovery

128
04:33:30,080 --> 04:33:33,510
defects. That is around 42% of the

129
04:33:33,520 --> 04:33:37,910
capability positive cases. So here key

130
04:33:37,920 --> 04:33:40,709
takeaway is that a mutation can improve

131
04:33:40,719 --> 04:33:44,150
an agent's capability and still leave

132
04:33:44,160 --> 04:33:51,750
the system unable to recover reliably.

133
04:33:51,760 --> 04:33:55,670
Now now we can say that why please fix

134
04:33:55,680 --> 04:33:59,429
the undo did not work. So naturally we

135
04:33:59,439 --> 04:34:02,390
wanted to use prompting and wanted to

136
04:34:02,400 --> 04:34:04,551
tell agent that hey could you simply

137
04:34:04,561 --> 04:34:07,990
repair all these recovery failures. So

138
04:34:08,000 --> 04:34:12,070
we tested four verifier guided repair

139
04:34:12,080 --> 04:34:14,390
strategies but under the original

140
04:34:14,400 --> 04:34:18,949
recovery language none recovered the 197

141
04:34:18,959 --> 04:34:21,510
naturally occurring failures. The

142
04:34:21,520 --> 04:34:24,070
problem was not necessarily

143
04:34:24,080 --> 04:34:27,029
the agent's reasoning. In some cases,

144
04:34:27,039 --> 04:34:30,629
the agent lacked precise information

145
04:34:30,639 --> 04:34:34,709
about the original state. In uh in

146
04:34:34,719 --> 04:34:38,470
others, the recovery language could not

147
04:34:38,480 --> 04:34:41,349
express the operation needed to reverse

148
04:34:41,359 --> 04:34:44,789
the change. So, we investigated these

149
04:34:44,799 --> 04:34:50,470
two limitations separately.

150
04:34:50,480 --> 04:34:53,109
So, two bottlenecks.

151
04:34:53,119 --> 04:34:55,429
First is state grounding and another one

152
04:34:55,439 --> 04:34:57,830
is recovery expressivity.

153
04:34:57,840 --> 04:35:01,510
The when we provided like precise

154
04:35:01,520 --> 04:35:04,070
information about what had changed and

155
04:35:04,080 --> 04:35:07,590
where then recovery improved from 0 to

156
04:35:07,600 --> 04:35:12,390
38 out of 48 cases. And the second is

157
04:35:12,400 --> 04:35:14,709
recovery expressivity. So when we

158
04:35:14,719 --> 04:35:17,349
introduced richer undo operation

159
04:35:17,359 --> 04:35:20,470
recovery improved significantly from

160
04:35:20,480 --> 04:35:24,709
zero to 142 out of 143 cases. So these

161
04:35:24,719 --> 04:35:27,109
results come from the two different

162
04:35:27,119 --> 04:35:30,230
experimental tubros but together they

163
04:35:30,240 --> 04:35:33,990
show why recovery needs both accurate

164
04:35:34,000 --> 04:35:37,349
state information and ability to perform

165
04:35:37,359 --> 04:35:46,629
the required end to operation.

166
04:35:46,639 --> 04:35:49,510
Now question comes to our mind that can

167
04:35:49,520 --> 04:35:52,150
better undo mechanism rescue failed

168
04:35:52,160 --> 04:35:53,830
mutations.

169
04:35:53,840 --> 04:35:57,189
So what happens like when we give the

170
04:35:57,199 --> 04:35:59,990
agent better recovery capabilities. So

171
04:36:00,000 --> 04:36:03,269
with the original recovery language none

172
04:36:03,279 --> 04:36:06,949
of the 197 recovery defective mutations

173
04:36:06,959 --> 04:36:09,990
could be restored. But after extending

174
04:36:10,000 --> 04:36:12,150
the recovery language, we successfully

175
04:36:12,160 --> 04:36:16,709
recovered 180 out of 197 cases. That is

176
04:36:16,719 --> 04:36:20,709
around 92%. As a result, the number of

177
04:36:20,719 --> 04:36:23,750
admissible improvement increased from

178
04:36:23,760 --> 04:36:26,390
281 to 461

179
04:36:26,400 --> 04:36:30,470
out of 600 task. That is a 30 percentage

180
04:36:30,480 --> 04:36:33,510
point improvement. And

181
04:36:33,520 --> 04:36:36,230
that brings us to the next question. How

182
04:36:36,240 --> 04:36:39,670
do we build these recovery checks into

183
04:36:39,680 --> 04:36:43,189
the agent's evolution process? I will

184
04:36:43,199 --> 04:36:46,150
now hand it over to Dolly to walk

185
04:36:46,160 --> 04:36:51,429
through the EVO undo algorithm.

186
04:36:51,439 --> 04:36:54,949
All right, thanks San. Uh so let's get

187
04:36:54,959 --> 04:36:57,429
into the working on how Evo and actually

188
04:36:57,439 --> 04:36:59,990
works. So first we will start with the

189
04:37:00,000 --> 04:37:02,949
agent pro. Uh first

190
04:37:02,959 --> 04:37:05,750
the agent will start the agent pro

191
04:37:05,760 --> 04:37:08,150
processes the mutation on which state it

192
04:37:08,160 --> 04:37:09,910
is currently in along with the effect

193
04:37:09,920 --> 04:37:12,869
contract and the recovery plan. And

194
04:37:12,879 --> 04:37:15,910
before applying the mutation

195
04:37:15,920 --> 04:37:18,709
we captured the state witness

196
04:37:18,719 --> 04:37:21,429
representing the original state

197
04:37:21,439 --> 04:37:25,029
and even before that the then a testing

198
04:37:25,039 --> 04:37:26,709
in the testing environment agent will

199
04:37:26,719 --> 04:37:28,789
apply the mutation and execute the

200
04:37:28,799 --> 04:37:31,269
recovery plant uh an independent

201
04:37:31,279 --> 04:37:34,310
verifier will also verify the original

202
04:37:34,320 --> 04:37:36,390
if uh verify to check if the original

203
04:37:36,400 --> 04:37:38,709
state has been restored or not. If the

204
04:37:38,719 --> 04:37:41,670
mutation improves uh the capability

205
04:37:41,680 --> 04:37:44,230
based on the userdefined criteria and

206
04:37:44,240 --> 04:37:47,990
passes the recovery um check the agents

207
04:37:48,000 --> 04:37:53,029
admit the changes otherwise well

208
04:37:53,039 --> 04:37:55,429
uh the agent attempt to repair or the

209
04:37:55,439 --> 04:37:57,670
verifier itself rejects the mutation

210
04:37:57,680 --> 04:38:00,150
altogether. This way the recoveryability

211
04:38:00,160 --> 04:38:02,070
becomes a part of the admission decision

212
04:38:02,080 --> 04:38:04,070
rather than something that we strategize

213
04:38:04,080 --> 04:38:08,390
or even plan once we are uh deployed the

214
04:38:08,400 --> 04:38:11,990
deployed the AI agent.

215
04:38:12,000 --> 04:38:16,709
Okay. So with next

216
04:38:16,719 --> 04:38:19,109
we will walk through as as Tan have

217
04:38:19,119 --> 04:38:21,510
already explained in given an example

218
04:38:21,520 --> 04:38:23,990
let let me walk through how the ex uh

219
04:38:24,000 --> 04:38:26,310
how that example of a timer changing

220
04:38:26,320 --> 04:38:29,750
from 30 to 60 would look like in action.

221
04:38:29,760 --> 04:38:32,551
So let's take let's start with the

222
04:38:32,561 --> 04:38:34,869
original state the simple example that

223
04:38:34,879 --> 04:38:37,670
uh simple example of um our agent having

224
04:38:37,680 --> 04:38:40,551
the time value of 30 before changing

225
04:38:40,561 --> 04:38:43,510
anything the evoando would first read

226
04:38:43,520 --> 04:38:45,990
the original value

227
04:38:46,000 --> 04:38:48,230
and then it will save the it will

228
04:38:48,240 --> 04:38:50,150
capture the original value as a state

229
04:38:50,160 --> 04:38:53,510
witness. We know exact we so we know

230
04:38:53,520 --> 04:38:56,070
exactly what need to be restored when

231
04:38:56,080 --> 04:38:58,789
when and when if we have to restore it

232
04:38:58,799 --> 04:39:00,869
we would know what what need to be

233
04:39:00,879 --> 04:39:02,789
restored or which state would be the

234
04:39:02,799 --> 04:39:06,070
true state to return back to. Now the

235
04:39:06,080 --> 04:39:08,551
agent so next come the mutation. Now the

236
04:39:08,561 --> 04:39:11,029
agent changes the timeout from 30 to 60

237
04:39:11,039 --> 04:39:14,150
as we want it to do. And if recovery is

238
04:39:14,160 --> 04:39:17,990
required uh if recovery is required the

239
04:39:18,000 --> 04:39:20,551
undo uses the saved witness to restore

240
04:39:20,561 --> 04:39:22,869
the timeout to 60.

241
04:39:22,879 --> 04:39:25,349
Now independently the verifier finally

242
04:39:25,359 --> 04:39:26,949
an independent verifier would also

243
04:39:26,959 --> 04:39:29,189
confirm that the uh recovered value

244
04:39:29,199 --> 04:39:31,189
matches the original value. So the

245
04:39:31,199 --> 04:39:33,109
recovery isn't considered successful

246
04:39:33,119 --> 04:39:36,709
simply because an undo command was run

247
04:39:36,719 --> 04:39:39,590
but also the verifier would verify the

248
04:39:39,600 --> 04:39:42,150
resulting state to make sure that we are

249
04:39:42,160 --> 04:39:46,070
in fact doing the uh recovery those in

250
04:39:46,080 --> 04:39:49,990
the state we want it to be.

251
04:39:50,000 --> 04:39:53,030
Well now comes the bigger question. So

252
04:39:53,040 --> 04:39:55,590
so we have this capability of recovery

253
04:39:55,600 --> 04:39:57,830
in our arsenal. Should we be using it

254
04:39:57,840 --> 04:39:59,830
for uniformly and applying the recovery

255
04:39:59,840 --> 04:40:02,708
mechanism to all the failures? Well, of

256
04:40:02,718 --> 04:40:05,670
course not. Um, not every failures can

257
04:40:05,680 --> 04:40:07,670
be recovered the same way. If nothing

258
04:40:07,680 --> 04:40:11,030
has changed, then the then simply if if

259
04:40:11,040 --> 04:40:12,628
nothing has changed in the in the

260
04:40:12,638 --> 04:40:16,550
system, we would simply want it to um

261
04:40:16,560 --> 04:40:19,430
recover uh we we would simply want it to

262
04:40:19,440 --> 04:40:22,230
safely retry. we would want the agent to

263
04:40:22,240 --> 04:40:25,190
simply retry if uh and try to achieve

264
04:40:25,200 --> 04:40:27,590
the state that we want to be. However,

265
04:40:27,600 --> 04:40:30,788
if we have the original state uh once we

266
04:40:30,798 --> 04:40:32,470
have the original state, we can restore

267
04:40:32,480 --> 04:40:34,948
it. If the external action has already

268
04:40:34,958 --> 04:40:39,270
been committed and we need uh

269
04:40:39,280 --> 04:40:41,030
it's then when we would need a

270
04:40:41,040 --> 04:40:43,350
compensation mechanism, compensation

271
04:40:43,360 --> 04:40:45,990
action to comp compensating action to

272
04:40:46,000 --> 04:40:48,470
reverse its effect. And if we cannot

273
04:40:48,480 --> 04:40:52,390
recover the recovery uh and we cannot if

274
04:40:52,400 --> 04:40:55,670
if and if we cannot the cannot verify

275
04:40:55,680 --> 04:40:57,910
that recovery is safe or the verifier

276
04:40:57,920 --> 04:41:00,390
have rejected the recovery then the

277
04:41:00,400 --> 04:41:03,110
agent should stop and escalate. The

278
04:41:03,120 --> 04:41:04,788
right response definitely depends on

279
04:41:04,798 --> 04:41:07,270
what actually happened and what we what

280
04:41:07,280 --> 04:41:13,430
can be safely reversed or not.

281
04:41:13,440 --> 04:41:16,070
That brings me

282
04:41:16,080 --> 04:41:18,310
to the almost the last slide which is

283
04:41:18,320 --> 04:41:20,390
about bringing everything together on

284
04:41:20,400 --> 04:41:22,948
our two paper undo bench and ebo undo.

285
04:41:22,958 --> 04:41:26,390
The so the key takeaways here are first

286
04:41:26,400 --> 04:41:29,830
the undo bench shows the successful task

287
04:41:29,840 --> 04:41:31,590
execution.

288
04:41:31,600 --> 04:41:34,708
Uh the undo bench shows that the

289
04:41:34,718 --> 04:41:36,550
successful task execution doesn't

290
04:41:36,560 --> 04:41:38,390
guarantee doesn't always guarantee the

291
04:41:38,400 --> 04:41:41,350
successful recovery of it. Uh secondly,

292
04:41:41,360 --> 04:41:43,830
the EVO undo algorithm provides a

293
04:41:43,840 --> 04:41:45,670
structured process. Capture the original

294
04:41:45,680 --> 04:41:47,590
state, apply the mutation, recover

295
04:41:47,600 --> 04:41:50,070
independently, verify and restoration.

296
04:41:50,080 --> 04:41:53,190
So the EVO undo algorithm does work in

297
04:41:53,200 --> 04:41:55,590
the in the fash in the way we intended

298
04:41:55,600 --> 04:41:58,470
it to. And third, we should admit

299
04:41:58,480 --> 04:42:00,708
self-evolution changes only when the

300
04:42:00,718 --> 04:42:02,310
when both the capability and the

301
04:42:02,320 --> 04:42:06,310
recoverability requirements are met. So

302
04:42:06,320 --> 04:42:08,390
ultimately as we give agent more

303
04:42:08,400 --> 04:42:11,110
autonomy, we should also give them more

304
04:42:11,120 --> 04:42:12,948
uh reliable ways to recover from their

305
04:42:12,958 --> 04:42:16,390
actions.

306
04:42:16,400 --> 04:42:19,510
All right. So and finally I hope you

307
04:42:19,520 --> 04:42:21,750
remember the agent from the beginning.

308
04:42:21,760 --> 04:42:23,990
It was looking for an undo button. Well,

309
04:42:24,000 --> 04:42:26,310
now it has one. We have started building

310
04:42:26,320 --> 04:42:27,910
this capability through our research

311
04:42:27,920 --> 04:42:30,230
project Evo undo. And you can explore

312
04:42:30,240 --> 04:42:32,708
the work through the in in the GitHub

313
04:42:32,718 --> 04:42:36,070
repository. can scan the QR code on the

314
04:42:36,080 --> 04:42:38,948
screen to get to the page. Our goal is

315
04:42:38,958 --> 04:42:40,470
to make the recovery a fundamental

316
04:42:40,480 --> 04:42:43,590
capability of AI agent so that we not

317
04:42:43,600 --> 04:42:46,310
only have the the agent does not even

318
04:42:46,320 --> 04:42:48,310
execute

319
04:42:48,320 --> 04:42:49,990
the action but also recover when

320
04:42:50,000 --> 04:42:52,070
something goes wrong as well. Thank you

321
04:42:52,080 --> 04:42:54,300
so much.