1
06:37:54,320 --> 06:37:57,350
Hey folks, this is Gish, a principal

2
06:37:57,360 --> 06:37:59,510
software engineer at Microsoft. Outside

3
06:37:59,520 --> 06:38:02,628
my job, I build open source tools for AI

4
06:38:02,638 --> 06:38:06,070
agents. One of them is MCP pacemaker, a

5
06:38:06,080 --> 06:38:08,628
supervised bridge that keeps MCP servers

6
06:38:08,638 --> 06:38:10,390
running.

7
06:38:10,400 --> 06:38:13,030
Before we start, a small disclosure. The

8
06:38:13,040 --> 06:38:15,110
views in this talk are my own and

9
06:38:15,120 --> 06:38:17,670
doesn't represent my employer.

10
06:38:17,680 --> 06:38:20,948
So sub agents are a distributed system.

11
06:38:20,958 --> 06:38:22,788
Over the past year, I have been using

12
06:38:22,798 --> 06:38:24,390
agents and sub agents on real

13
06:38:24,400 --> 06:38:27,350
engineering work, dev sessions, code

14
06:38:27,360 --> 06:38:29,990
reviews, instant trashing

15
06:38:30,000 --> 06:38:31,990
what

16
06:38:32,000 --> 06:38:34,708
this talk is about what happens when one

17
06:38:34,718 --> 06:38:38,150
agent hands work to another.

18
06:38:38,160 --> 06:38:41,030
So here's an architecture.

19
06:38:41,040 --> 06:38:43,750
The orchestrator splits a task and hands

20
06:38:43,760 --> 06:38:47,270
the pieces to three different sub aents.

21
06:38:47,280 --> 06:38:50,070
They all run at the same time.

22
06:38:50,080 --> 06:38:52,230
They all read from the same shared

23
06:38:52,240 --> 06:38:56,628
state. Consization, memory files, repo,

24
06:38:56,638 --> 06:39:01,510
working trees, issues, vector stores.

25
06:39:01,520 --> 06:39:04,070
And they don't just think, they make

26
06:39:04,080 --> 06:39:08,070
tool calls to real systems

27
06:39:08,080 --> 06:39:11,590
which touch databases, APIs, CI tools,

28
06:39:11,600 --> 06:39:13,990
deployment pipelines

29
06:39:14,000 --> 06:39:15,670
and

30
06:39:15,680 --> 06:39:18,628
the results come back and orchestrator

31
06:39:18,638 --> 06:39:20,948
merges them into a single answer and

32
06:39:20,958 --> 06:39:23,990
orchestrator workers shared mutable

33
06:39:24,000 --> 06:39:26,708
state and results merge from concurrent

34
06:39:26,718 --> 06:39:29,110
executions. This represents a classic

35
06:39:29,120 --> 06:39:31,270
distributed system.

36
06:39:31,280 --> 06:39:33,110
I have known how to reason with dispute

37
06:39:33,120 --> 06:39:35,350
systems for decades,

38
06:39:35,360 --> 06:39:39,270
but none of the setup came with it.

39
06:39:39,280 --> 06:39:41,670
What you get is a function that takes a

40
06:39:41,680 --> 06:39:44,468
string and returns a string.

41
06:39:44,478 --> 06:39:47,510
Failures we hit are not new, are not

42
06:39:47,520 --> 06:39:50,150
specific to a agents. They're classic

43
06:39:50,160 --> 06:39:53,670
distributed system failures.

44
06:39:53,680 --> 06:39:56,390
partial failure. A normal call that

45
06:39:56,400 --> 06:39:59,190
fails halfway through and returns an

46
06:39:59,200 --> 06:40:02,150
exception error code. In a sub aent

47
06:40:02,160 --> 06:40:04,948
case, it returns text that looks like a

48
06:40:04,958 --> 06:40:06,628
success.

49
06:40:06,638 --> 06:40:10,390
Dirty reads normally we have isolations

50
06:40:10,400 --> 06:40:13,510
versions. Here every worker reads one

51
06:40:13,520 --> 06:40:16,070
shade context including things other

52
06:40:16,080 --> 06:40:18,548
worker only guessed.

53
06:40:18,558 --> 06:40:20,948
Unbounded fan out.

54
06:40:20,958 --> 06:40:25,750
A busy service pushes back with the cues

55
06:40:25,760 --> 06:40:28,310
429's 503s.

56
06:40:28,320 --> 06:40:31,430
In agents, orchestrator does not see any

57
06:40:31,440 --> 06:40:34,548
of this. It just starts in the worker.

58
06:40:34,558 --> 06:40:36,468
Retries.

59
06:40:36,478 --> 06:40:39,110
In normal distute systems, we build

60
06:40:39,120 --> 06:40:42,310
potency to make retries reliable.

61
06:40:42,320 --> 06:40:45,430
in agents a retrate sub agent and the

62
06:40:45,440 --> 06:40:47,510
whole task again with side effects

63
06:40:47,520 --> 06:40:49,030
included

64
06:40:49,040 --> 06:40:51,830
for each of these case

65
06:40:51,840 --> 06:40:54,868
I'll name the concept it already is then

66
06:40:54,878 --> 06:40:56,868
I'll put them together as a consistency

67
06:40:56,878 --> 06:40:59,670
model I'm proposing and walk through why

68
06:40:59,680 --> 06:41:02,070
each part is there

69
06:41:02,080 --> 06:41:03,910
start with the failure model because

70
06:41:03,920 --> 06:41:06,948
everything else depends on it a friendly

71
06:41:06,958 --> 06:41:09,830
case is a crash stop a process it was

72
06:41:09,840 --> 06:41:11,990
running correctly stops and stays is

73
06:41:12,000 --> 06:41:14,628
stopped. The sub agent throws an error

74
06:41:14,638 --> 06:41:18,070
in this box. It's easy to handle.

75
06:41:18,080 --> 06:41:21,350
Next, no reply. The call is lost or it's

76
06:41:21,360 --> 06:41:25,350
very late. We'll come back to this soon.

77
06:41:25,360 --> 06:41:28,788
The hot case is the one from Baantine

78
06:41:28,798 --> 06:41:31,990
general's paper. A component that gives

79
06:41:32,000 --> 06:41:34,230
conflicting information. The process

80
06:41:34,240 --> 06:41:37,270
answers and the answer is wrong.

81
06:41:37,280 --> 06:41:39,350
A half finished sub agent sits closest

82
06:41:39,360 --> 06:41:42,150
to the box. It is not malicious but from

83
06:41:42,160 --> 06:41:44,468
the outside it looks the same. It does

84
06:41:44,478 --> 06:41:47,030
not crash. It answers fluently. Let's

85
06:41:47,040 --> 06:41:50,708
jump into this. A sub agent that half

86
06:41:50,718 --> 06:41:53,510
finished does not raise. It returns

87
06:41:53,520 --> 06:41:55,830
normal looking text with the same type

88
06:41:55,840 --> 06:41:58,070
signature as success.

89
06:41:58,080 --> 06:42:00,788
I talked about one version of this in

90
06:42:00,798 --> 06:42:03,750
one of my talks. We pointed an agent at

91
06:42:03,760 --> 06:42:05,670
an LQ.

92
06:42:05,680 --> 06:42:08,230
Here's an illustration I used there with

93
06:42:08,240 --> 06:42:10,070
three different paths which gave the

94
06:42:10,080 --> 06:42:11,830
same answer.

95
06:42:11,840 --> 06:42:13,830
The first one

96
06:42:13,840 --> 06:42:15,910
it pulled the telemetry and compared

97
06:42:15,920 --> 06:42:18,390
with a known brand set. It actually did

98
06:42:18,400 --> 06:42:21,430
the work. The second one it matched

99
06:42:21,440 --> 06:42:24,708
against the title. It has seen before

100
06:42:24,718 --> 06:42:27,270
and gave the response.

101
06:42:27,280 --> 06:42:29,430
The third alert

102
06:42:29,440 --> 06:42:31,670
it didn't do anything.

103
06:42:31,680 --> 06:42:34,788
It just originated.

104
06:42:34,798 --> 06:42:36,868
All three are categorized as benign with

105
06:42:36,878 --> 06:42:39,350
the same level of confidence

106
06:42:39,360 --> 06:42:41,430
and nothing downstream can dull this

107
06:42:41,440 --> 06:42:43,750
apart. The orchestrator can read the

108
06:42:43,760 --> 06:42:45,910
conclusion. It cannot read how much work

109
06:42:45,920 --> 06:42:48,390
was behind it.

110
06:42:48,400 --> 06:42:51,750
Now the call gets no reply.

111
06:42:51,760 --> 06:42:54,788
The orchestrator send work and a timeout

112
06:42:54,798 --> 06:42:58,310
fired. From here three volts look the

113
06:42:58,320 --> 06:43:01,110
same. The sub agent is still working. It

114
06:43:01,120 --> 06:43:04,548
crashed and the reply was lost.

115
06:43:04,558 --> 06:43:07,110
It's a hall limit, not a tooling gap.

116
06:43:07,120 --> 06:43:10,150
The FLP paper from 1985 showed that in a

117
06:43:10,160 --> 06:43:12,548
synchronous system with even one faulty

118
06:43:12,558 --> 06:43:16,070
process, agreement can fail to finish.

119
06:43:16,080 --> 06:43:18,788
Chundra and 2x answer in 1996 was

120
06:43:18,798 --> 06:43:20,468
failure detectors that can make

121
06:43:20,478 --> 06:43:24,150
mistakes. They suspect did not know. I

122
06:43:24,160 --> 06:43:26,628
hit both directions

123
06:43:26,638 --> 06:43:30,070
in my own open source project MCP

124
06:43:30,080 --> 06:43:33,110
pacemaker supervised bridge.

125
06:43:33,120 --> 06:43:35,590
A server whose first run downloads

126
06:43:35,600 --> 06:43:37,990
itself hit the 30 secondond call time

127
06:43:38,000 --> 06:43:41,430
out and look broken. It was slow not

128
06:43:41,440 --> 06:43:44,868
dead. We fixed it by giving initialize

129
06:43:44,878 --> 06:43:48,070
its own time out. Now the other way the

130
06:43:48,080 --> 06:43:50,868
server crashed during a call. The call

131
06:43:50,878 --> 06:43:53,910
then timed out and the timeout overrode

132
06:43:53,920 --> 06:43:56,628
the exit code that explained the crash.

133
06:43:56,638 --> 06:43:59,910
We fixed it by recording the time out

134
06:43:59,920 --> 06:44:02,310
only when it was alive.

135
06:44:02,320 --> 06:44:03,830
So

136
06:44:03,840 --> 06:44:06,230
time out is a suspicion. Record it as

137
06:44:06,240 --> 06:44:09,830
unknown and keep the real cause.

138
06:44:09,840 --> 06:44:15,030
And now make failure a value.

139
06:44:15,040 --> 06:44:17,510
Let's take this illustration.

140
06:44:17,520 --> 06:44:20,708
A sub agent returns a status. What it

141
06:44:20,718 --> 06:44:24,310
did, what it did not do, the evidence,

142
06:44:24,320 --> 06:44:26,788
and any side effects. Partial is a

143
06:44:26,798 --> 06:44:29,110
normal answer here.

144
06:44:29,120 --> 06:44:31,590
The protocols already have a slot for

145
06:44:31,600 --> 06:44:35,270
this. In A2A, a task can end as

146
06:44:35,280 --> 06:44:39,030
completed, failed, canceled or rejected

147
06:44:39,040 --> 06:44:42,788
or stop and ask for input. In MCP, a

148
06:44:42,798 --> 06:44:46,868
dual result can say is error true. But a

149
06:44:46,878 --> 06:44:50,150
sub agent can still say completed while

150
06:44:50,160 --> 06:44:53,190
its text says something else. And a loop

151
06:44:53,200 --> 06:44:55,030
that only reads the text never sees a

152
06:44:55,040 --> 06:44:56,788
difference.

153
06:44:56,798 --> 06:44:58,788
Another spirit.

154
06:44:58,798 --> 06:45:01,910
Checking a condition and getting no is a

155
06:45:01,920 --> 06:45:04,070
different fact from not being able to

156
06:45:04,080 --> 06:45:06,150
check at all.

157
06:45:06,160 --> 06:45:08,788
My own individual internet draft on

158
06:45:08,798 --> 06:45:11,270
evaluation states

159
06:45:11,280 --> 06:45:13,510
talks about this.

160
06:45:13,520 --> 06:45:15,830
Collapse the two and you change what

161
06:45:15,840 --> 06:45:18,708
gets retrieded and who gets ped and what

162
06:45:18,718 --> 06:45:21,510
the audit says later.

163
06:45:21,520 --> 06:45:24,868
Next dirty reads.

164
06:45:24,878 --> 06:45:26,948
Let's take this madeup example from one

165
06:45:26,958 --> 06:45:28,948
of my talk.

166
06:45:28,958 --> 06:45:32,548
The sub agent the scratch it writes into

167
06:45:32,558 --> 06:45:34,468
a shade context

168
06:45:34,478 --> 06:45:37,590
saying that error rate fell by 80%. It

169
06:45:37,600 --> 06:45:41,030
did not measure that it inferred it.

170
06:45:41,040 --> 06:45:43,670
Now

171
06:45:43,680 --> 06:45:45,750
two other sub aents read from the same

172
06:45:45,760 --> 06:45:48,310
shade context.

173
06:45:48,320 --> 06:45:50,628
The second sub agent writes a review

174
06:45:50,638 --> 06:45:53,430
note and picks the same sentenc.

175
06:45:53,440 --> 06:45:56,230
The third agent writes a runbook step

176
06:45:56,240 --> 06:45:59,350
and uses it as a reason to proceed.

177
06:45:59,360 --> 06:46:01,990
The orchestrator merges all the all

178
06:46:02,000 --> 06:46:05,430
these three three artifacts agree. It is

179
06:46:05,440 --> 06:46:07,910
once unsupported sentence read three

180
06:46:07,920 --> 06:46:09,910
different times.

181
06:46:09,920 --> 06:46:12,310
The 1995 critique of SQL isolation

182
06:46:12,320 --> 06:46:15,270
levels define a dirty read as you read a

183
06:46:15,280 --> 06:46:17,270
value before the writer commits or rolls

184
06:46:17,280 --> 06:46:20,628
back. In a shared context, a guess is

185
06:46:20,638 --> 06:46:22,788
readable the moment it is written and

186
06:46:22,798 --> 06:46:25,830
nothing marks it as unchecked.

187
06:46:25,840 --> 06:46:28,310
A regular fix

188
06:46:28,320 --> 06:46:31,430
to is to ask three sub agents and get a

189
06:46:31,440 --> 06:46:33,430
quorum.

190
06:46:33,440 --> 06:46:36,628
Corums do work for a specific reason. In

191
06:46:36,638 --> 06:46:39,670
weighted voting from 1979, the read

192
06:46:39,680 --> 06:46:42,310
corum plus right corum is greater than

193
06:46:42,320 --> 06:46:43,830
total.

194
06:46:43,840 --> 06:46:46,468
So every read overlaps the latest right.

195
06:46:46,478 --> 06:46:49,030
The overlap is the guarantee.

196
06:46:49,040 --> 06:46:51,910
In sub aent's case, the three sub aents

197
06:46:51,920 --> 06:46:54,070
that read the same shared context does

198
06:46:54,080 --> 06:46:55,830
not give you that. They are not

199
06:46:55,840 --> 06:46:58,070
independent copies. They read the same

200
06:46:58,080 --> 06:47:01,030
guess. Their agreement is one source

201
06:47:01,040 --> 06:47:03,270
counted three different times. Let's

202
06:47:03,280 --> 06:47:05,750
pick a consistency model.

203
06:47:05,760 --> 06:47:07,910
In databases, we have vocabulary for

204
06:47:07,920 --> 06:47:09,510
this.

205
06:47:09,520 --> 06:47:11,270
Linearizable.

206
06:47:11,280 --> 06:47:13,430
Every operation appears to happen at one

207
06:47:13,440 --> 06:47:17,270
point in real time order. Sequential.

208
06:47:17,280 --> 06:47:19,430
One total order that respects each

209
06:47:19,440 --> 06:47:23,190
process one order. Causal

210
06:47:23,200 --> 06:47:25,830
operations that are causally related

211
06:47:25,840 --> 06:47:28,230
appear in the same order everywhere.

212
06:47:28,240 --> 06:47:30,390
Session guarantees.

213
06:47:30,400 --> 06:47:32,868
There are promises to one client like

214
06:47:32,878 --> 06:47:36,310
read your rights and monotonic reads

215
06:47:36,320 --> 06:47:39,510
eventual. If no new updates are made,

216
06:47:39,520 --> 06:47:42,548
reads eventually return the lost value.

217
06:47:42,558 --> 06:47:44,390
A shared context gives you none of

218
06:47:44,400 --> 06:47:46,708
these. There's no stated guarantee at

219
06:47:46,718 --> 06:47:48,310
all.

220
06:47:48,320 --> 06:47:50,948
And you don't need the top consistency

221
06:47:50,958 --> 06:47:52,628
models.

222
06:47:52,638 --> 06:47:54,548
The strongest models cannot stay

223
06:47:54,558 --> 06:47:57,590
available when the network splits.

224
06:47:57,600 --> 06:48:00,070
Causal end session guarantees can as

225
06:48:00,080 --> 06:48:03,350
long as each clients stick to one place

226
06:48:03,360 --> 06:48:05,590
for sub agents that place the

227
06:48:05,600 --> 06:48:08,788
orchestrator. So

228
06:48:08,798 --> 06:48:11,030
here's a consistency model I use for sub

229
06:48:11,040 --> 06:48:16,468
agents. I call it safe handoffs.

230
06:48:16,478 --> 06:48:20,390
S stands for source fees.

231
06:48:20,400 --> 06:48:24,150
Every fact a worker hands on carries its

232
06:48:24,160 --> 06:48:27,990
source version as of time and revision

233
06:48:28,000 --> 06:48:29,990
and decisions read only fresh source

234
06:48:30,000 --> 06:48:31,830
facts.

235
06:48:31,840 --> 06:48:35,190
A at most once effects

236
06:48:35,200 --> 06:48:37,670
every right has identity and a single

237
06:48:37,680 --> 06:48:40,390
use approval for that exact action. An

238
06:48:40,400 --> 06:48:44,948
outcome is read back never replayed.

239
06:48:44,958 --> 06:48:49,430
F fan out budgets. Each downstream gets

240
06:48:49,440 --> 06:48:52,310
a gate and a bounded weight and refusal

241
06:48:52,320 --> 06:48:54,708
counts as a fail.

242
06:48:54,718 --> 06:48:57,910
E stands for explicit outcomes.

243
06:48:57,920 --> 06:49:01,110
Every handoff ends in a name state and a

244
06:49:01,120 --> 06:49:04,788
timeout is unknown with its cost script.

245
06:49:04,798 --> 06:49:07,750
This consistency model is the one I use

246
06:49:07,760 --> 06:49:09,430
for my own sub agents. It's not a

247
06:49:09,440 --> 06:49:12,070
universal model. It's based off my own

248
06:49:12,080 --> 06:49:15,510
public work. Some of it is in my FCP

249
06:49:15,520 --> 06:49:17,990
pacemaker bridge and some are based on

250
06:49:18,000 --> 06:49:24,150
my other docs and my own internet draft.

251
06:49:24,160 --> 06:49:29,430
Now let's see what's behind each letter.

252
06:49:29,440 --> 06:49:32,070
Yes, source reads

253
06:49:32,080 --> 06:49:35,030
it's the answer to that reads where one

254
06:49:35,040 --> 06:49:36,868
worker's guess becomes another worker's

255
06:49:36,878 --> 06:49:40,628
fact. What helps is to treat every fact

256
06:49:40,638 --> 06:49:43,270
as a record not a sentence.

257
06:49:43,280 --> 06:49:46,390
Lampot's 1978 paper define when one

258
06:49:46,400 --> 06:49:48,788
event happened before another. The facts

259
06:49:48,798 --> 06:49:51,270
provenence is that record. What it was

260
06:49:51,280 --> 06:49:53,750
read from and when.

261
06:49:53,760 --> 06:49:57,830
Let's take this madeup example.

262
06:49:57,840 --> 06:50:00,708
the record names, the claim and its

263
06:50:00,718 --> 06:50:04,070
revision, the source and its version,

264
06:50:04,080 --> 06:50:08,070
the scope and three different times,

265
06:50:08,080 --> 06:50:10,628
when the data was current, when we read

266
06:50:10,638 --> 06:50:13,990
it and when we used it.

267
06:50:14,000 --> 06:50:16,310
Freshness is checked with the fact is

268
06:50:16,320 --> 06:50:18,150
used.

269
06:50:18,160 --> 06:50:22,230
In this illustration, the data ends at

270
06:50:22,240 --> 06:50:26,628
9:05. We read it at 9:06 and we decided

271
06:50:26,638 --> 06:50:28,708
at 9:07.

272
06:50:28,718 --> 06:50:30,708
In this case,

273
06:50:30,718 --> 06:50:34,708
we picked a 5 minute limit. So the data

274
06:50:34,718 --> 06:50:37,750
decision at 9:07, it's fine.

275
06:50:37,760 --> 06:50:42,468
Now at 9:15, the second edition came in

276
06:50:42,478 --> 06:50:44,708
because our freshness limit is five.

277
06:50:44,718 --> 06:50:47,270
This is a stale record.

278
06:50:47,280 --> 06:50:49,590
We reject it

279
06:50:49,600 --> 06:50:51,350
and we say that the next station needs

280
06:50:51,360 --> 06:50:53,590
another read.

281
06:50:53,600 --> 06:50:55,830
This is how you separate an observation

282
06:50:55,840 --> 06:50:58,230
from an inference. An observation has a

283
06:50:58,240 --> 06:51:00,868
source and a read time. An inference has

284
06:51:00,878 --> 06:51:02,788
none.

285
06:51:02,798 --> 06:51:06,868
A most effect is about retries.

286
06:51:06,878 --> 06:51:09,510
This is an illustration

287
06:51:09,520 --> 06:51:14,788
from my article about MCPmaker bridge.

288
06:51:14,798 --> 06:51:18,070
The orchestrator delegates a task. The

289
06:51:18,080 --> 06:51:21,110
sub agent calls a tool which does a real

290
06:51:21,120 --> 06:51:22,788
right.

291
06:51:22,798 --> 06:51:24,310
like

292
06:51:24,320 --> 06:51:26,628
creating a PR,

293
06:51:26,638 --> 06:51:29,110
the reply is lost.

294
06:51:29,120 --> 06:51:31,430
The orchestrator knows one thing, the

295
06:51:31,440 --> 06:51:33,430
outcome is unknown.

296
06:51:33,440 --> 06:51:37,350
So it retries the whole task, the right

297
06:51:37,360 --> 06:51:39,430
happens again.

298
06:51:39,440 --> 06:51:43,590
This is a classic problem. In 1984 RPC

299
06:51:43,600 --> 06:51:45,590
paper,

300
06:51:45,600 --> 06:51:48,310
it says that if the call returns, the

301
06:51:48,320 --> 06:51:50,788
procedure ran precisely once. If you get

302
06:51:50,798 --> 06:51:54,468
an exception, it ran once or not at all.

303
06:51:54,478 --> 06:51:57,110
And you are not told which.

304
06:51:57,120 --> 06:52:00,708
So you pick a delivery semantic.

305
06:52:00,718 --> 06:52:03,270
At most once

306
06:52:03,280 --> 06:52:05,590
don't retry and accept the fact that it

307
06:52:05,600 --> 06:52:10,310
may be lost. At least once. Retry and

308
06:52:10,320 --> 06:52:13,590
accept that duplicates are possible.

309
06:52:13,600 --> 06:52:15,830
Exactly once.

310
06:52:15,840 --> 06:52:18,310
It happened at least once. And there's a

311
06:52:18,320 --> 06:52:21,590
way we can dduplicate

312
06:52:21,600 --> 06:52:24,708
maybe by using an item potency key. In

313
06:52:24,718 --> 06:52:27,910
MCP item potent hint is only a hint and

314
06:52:27,920 --> 06:52:30,468
da or false. If you need it, you build

315
06:52:30,478 --> 06:52:34,390
it. Here's a small design from my bridge

316
06:52:34,400 --> 06:52:36,868
on how I handle it.

317
06:52:36,878 --> 06:52:38,468
When a client comes back after a

318
06:52:38,478 --> 06:52:41,030
restart, the bridge replace the client's

319
06:52:41,040 --> 06:52:42,548
record initialize against a fresh

320
06:52:42,558 --> 06:52:44,548
server. That's safe. It's just a

321
06:52:44,558 --> 06:52:47,030
handshake and it does not restore tool

322
06:52:47,040 --> 06:52:49,510
state.

323
06:52:49,520 --> 06:52:51,590
What it never replace is an uncertain

324
06:52:51,600 --> 06:52:53,990
tool call. When a shared server is

325
06:52:54,000 --> 06:52:56,948
recycled, the bridge drains it and calls

326
06:52:56,958 --> 06:52:59,590
till unanswered at the deadline fail as

327
06:52:59,600 --> 06:53:02,070
potentially excluded.

328
06:53:02,080 --> 06:53:04,788
That status is honest. It tells the

329
06:53:04,798 --> 06:53:08,708
caller to go and check.

330
06:53:08,718 --> 06:53:11,190
What helped is giving every side effect

331
06:53:11,200 --> 06:53:14,868
an identity. Approval binds to that

332
06:53:14,878 --> 06:53:18,390
exact action. It's only used once.

333
06:53:18,400 --> 06:53:20,788
After the call, read back by that

334
06:53:20,798 --> 06:53:22,310
identity.

335
06:53:22,320 --> 06:53:24,310
Submitted is not confirmed. Confirmed

336
06:53:24,320 --> 06:53:27,190
means you observe the effect.

337
06:53:27,200 --> 06:53:30,628
The last part of a it shows up when work

338
06:53:30,638 --> 06:53:33,590
passes and resumes. A lease is a lock

339
06:53:33,600 --> 06:53:35,590
with a timeout.

340
06:53:35,600 --> 06:53:37,590
Martin Clipman's post about distributed

341
06:53:37,600 --> 06:53:40,150
locking shows this problem.

342
06:53:40,160 --> 06:53:43,590
A client passes while holding it. The

343
06:53:43,600 --> 06:53:46,628
lease expense. Someone else takes out

344
06:53:46,638 --> 06:53:48,548
and the first client wakes up and writes

345
06:53:48,558 --> 06:53:50,310
anyway.

346
06:53:50,320 --> 06:53:54,548
The fix he proposes a fencing token. A

347
06:53:54,558 --> 06:53:56,628
number that goes up every time the lock

348
06:53:56,638 --> 06:53:58,628
is granted.

349
06:53:58,638 --> 06:54:01,110
The storage remembers the highest token

350
06:54:01,120 --> 06:54:04,230
it has seen and rejects any right with

351
06:54:04,240 --> 06:54:06,628
the old one.

352
06:54:06,638 --> 06:54:09,110
Sub agents pause and resume all the

353
06:54:09,120 --> 06:54:13,830
time. In my other talk, as I put it,

354
06:54:13,840 --> 06:54:16,868
restore context, not side effects. So

355
06:54:16,878 --> 06:54:19,110
treat an approval like a fencing token.

356
06:54:19,120 --> 06:54:21,590
One action, one use. Checked at the

357
06:54:21,600 --> 06:54:23,990
gate, not remembered in the prompt. A

358
06:54:24,000 --> 06:54:26,708
conversation.

359
06:54:26,718 --> 06:54:30,150
My bridge fixed the same class of bug. A

360
06:54:30,160 --> 06:54:33,190
stale exit or a deleted resume can no

361
06:54:33,200 --> 06:54:34,548
longer remove or bring back a

362
06:54:34,558 --> 06:54:37,430
replacement session.

363
06:54:37,440 --> 06:54:40,948
Fan out budgets is about unbounded fan

364
06:54:40,958 --> 06:54:43,750
out. This is flow control or the lack of

365
06:54:43,760 --> 06:54:46,708
it. I hit this issue in my own MCV

366
06:54:46,718 --> 06:54:48,788
bridge before it had a limit on cold

367
06:54:48,798 --> 06:54:50,868
starts.

368
06:54:50,878 --> 06:54:53,990
Many sessions start their service at the

369
06:54:54,000 --> 06:54:56,788
same time through npx.

370
06:54:56,798 --> 06:54:59,990
They share one package cache. Every one

371
06:55:00,000 --> 06:55:01,830
of them failed with npm's lock

372
06:55:01,840 --> 06:55:03,750
compromis.

373
06:55:03,760 --> 06:55:06,070
Nothing in any single request was wrong.

374
06:55:06,080 --> 06:55:09,190
The failures was the concurrency itself.

375
06:55:09,200 --> 06:55:11,830
And when a downstream does push back, it

376
06:55:11,840 --> 06:55:15,350
can arrive as a plain text. This is MCP

377
06:55:15,360 --> 06:55:17,110
specs one example of a tool execution

378
06:55:17,120 --> 06:55:18,230
error.

379
06:55:18,240 --> 06:55:21,750
limit as one sentence. The model reads a

380
06:55:21,760 --> 06:55:23,830
sentence. The orchestrator code may

381
06:55:23,840 --> 06:55:25,830
never look the next worker starts

382
06:55:25,840 --> 06:55:27,670
anyway.

383
06:55:27,680 --> 06:55:30,708
So I introduce a kota budget in my

384
06:55:30,718 --> 06:55:32,468
bridge.

385
06:55:32,478 --> 06:55:35,830
Cole starts that use a package manager

386
06:55:35,840 --> 06:55:38,230
go through a gate two at a time by

387
06:55:38,240 --> 06:55:41,430
default. Everyone else waits and the

388
06:55:41,440 --> 06:55:45,110
weight is born at 15 seconds. Pass that

389
06:55:45,120 --> 06:55:48,150
the start goes ahead without a gate.

390
06:55:48,160 --> 06:55:51,510
And I documented why a corrupted cache

391
06:55:51,520 --> 06:55:54,468
is a risk while a hung client request is

392
06:55:54,478 --> 06:55:56,788
certainity. There is a choice for

393
06:55:56,798 --> 06:55:58,868
availability made on purpose and written

394
06:55:58,878 --> 06:56:00,708
down.

395
06:56:00,718 --> 06:56:04,468
And I also built a cap on sessions.

396
06:56:04,478 --> 06:56:07,430
At the cap, a new a new session waits up

397
06:56:07,440 --> 06:56:10,868
to 10 seconds and then it's refused.

398
06:56:10,878 --> 06:56:12,708
That refusal now counts as a failure in

399
06:56:12,718 --> 06:56:16,150
health. Another book capacity has to be

400
06:56:16,160 --> 06:56:18,468
reserved before the synchronous start

401
06:56:18,478 --> 06:56:21,430
not after otherwise

402
06:56:21,440 --> 06:56:24,310
cued starts or book the cap that is a

403
06:56:24,320 --> 06:56:28,070
check then act race for sub agents give

404
06:56:28,080 --> 06:56:30,628
the orchestrator a number per downstream

405
06:56:30,638 --> 06:56:34,230
a bounded weight and a refusate consent.

406
06:56:34,240 --> 06:56:35,750
Now

407
06:56:35,760 --> 06:56:40,070
e the explicit outcomes every hand of

408
06:56:40,080 --> 06:56:42,708
ends in a name state like stale or

409
06:56:42,718 --> 06:56:46,070
unknown not just text. The last piece is

410
06:56:46,080 --> 06:56:48,868
who decided that state treats the agent

411
06:56:48,878 --> 06:56:51,350
output as untrusted input the same way

412
06:56:51,360 --> 06:56:52,788
you treat a request body from the

413
06:56:52,798 --> 06:56:54,708
internet.

414
06:56:54,718 --> 06:56:57,270
The orchestrator owns the checks in

415
06:56:57,280 --> 06:57:00,310
code. Does the result match the schema?

416
06:57:00,320 --> 06:57:02,390
Is the evidence there? Is it fresh

417
06:57:02,400 --> 06:57:05,030
enough? the numbers hold when you recmp

418
06:57:05,040 --> 06:57:08,548
compute them. If a check fails, the step

419
06:57:08,558 --> 06:57:11,750
returns a refusal with a reason. It does

420
06:57:11,760 --> 06:57:13,350
not pass the text along with a warning

421
06:57:13,360 --> 06:57:16,230
paragraph underneath. If the same model

422
06:57:16,240 --> 06:57:18,708
can skip a check and declare it past,

423
06:57:18,718 --> 06:57:21,750
you have a request, not a boundary.

424
06:57:21,760 --> 06:57:23,990
Here's a madeup example from another

425
06:57:24,000 --> 06:57:28,390
talk. A sub agent says that a rate fell.

426
06:57:28,400 --> 06:57:31,670
Before it was 10 errors in,000 requests.

427
06:57:31,680 --> 06:57:34,230
after it was six errors with 600

428
06:57:34,240 --> 06:57:37,030
requests. It went from 10 to six and it

429
06:57:37,040 --> 06:57:39,990
claimed that our rate fell. But when we

430
06:57:40,000 --> 06:57:43,990
computed the rate, it's both 1%.

431
06:57:44,000 --> 06:57:46,070
We recomputed it and then the claim is

432
06:57:46,080 --> 06:57:47,990
refused.

433
06:57:48,000 --> 06:57:50,468
A supported answer is fine. A deliberate

434
06:57:50,478 --> 06:57:53,030
handoff to a person is fine. A confident

435
06:57:53,040 --> 06:57:55,350
guess it's not.

436
06:57:55,360 --> 06:57:57,910
So limitations because a consistency

437
06:57:57,920 --> 06:57:59,830
model is only useful when you know where

438
06:57:59,840 --> 06:58:01,990
we should start.

439
06:58:02,000 --> 06:58:07,510
So here's what safe model does not do.

440
06:58:07,520 --> 06:58:10,548
It does not make facts true. A source

441
06:58:10,558 --> 06:58:13,910
fact is checkable, not correct. It does

442
06:58:13,920 --> 06:58:17,110
not give you exactly once delivery.

443
06:58:17,120 --> 06:58:20,070
A lost reply stays unknown until you

444
06:58:20,080 --> 06:58:22,708
read it back. It does not roll anything

445
06:58:22,718 --> 06:58:24,390
back.

446
06:58:24,400 --> 06:58:26,788
Undoing a partial task is a compensation

447
06:58:26,798 --> 06:58:29,590
step you design. like the Saga's paper

448
06:58:29,600 --> 06:58:31,510
from 1987.

449
06:58:31,520 --> 06:58:33,670
And finally, there's no global order

450
06:58:33,680 --> 06:58:36,548
across workers, only causal order for a

451
06:58:36,558 --> 06:58:38,548
fact.

452
06:58:38,558 --> 06:58:42,628
And it has a price. CAP theorem says

453
06:58:42,638 --> 06:58:45,430
that when the network splits, you trade

454
06:58:45,440 --> 06:58:49,110
off availability against consistency.

455
06:58:49,120 --> 06:58:51,430
My bridge chose availability for cold

456
06:58:51,440 --> 06:58:55,590
starts on purpose. For rights, safe

457
06:58:55,600 --> 06:58:59,750
picks consistency over availability. It

458
06:58:59,760 --> 06:59:02,628
refuses rather than guesses.

459
06:59:02,638 --> 06:59:04,230
Packlac

460
06:59:04,240 --> 06:59:06,788
adds that even without failures, you

461
06:59:06,798 --> 06:59:09,670
trade latency for consistency.

462
06:59:09,680 --> 06:59:12,628
Every freshness check is another read

463
06:59:12,638 --> 06:59:16,788
and and every refusal needs an order.

464
06:59:16,798 --> 06:59:20,070
So where does not fit?

465
06:59:20,080 --> 06:59:23,430
If a sub agent only drafts text that a

466
06:59:23,440 --> 06:59:25,910
person reads before anything happens,

467
06:59:25,920 --> 06:59:28,468
most of this is a overhead.

468
06:59:28,478 --> 06:59:30,788
It also assumes that there's one single

469
06:59:30,798 --> 06:59:32,390
orchestrator.

470
06:59:32,400 --> 06:59:35,910
If you have more, this does not fit.

471
06:59:35,920 --> 06:59:38,230
Pure agents with no single place to

472
06:59:38,240 --> 06:59:41,350
stick to need something else.

473
06:59:41,360 --> 06:59:43,270
This is the consistency model I use for

474
06:59:43,280 --> 06:59:45,750
my own agents that touch actual real

475
06:59:45,760 --> 06:59:49,190
production systems. It's not universal.

476
06:59:49,200 --> 06:59:51,830
So wrapping up, here's three key

477
06:59:51,840 --> 06:59:53,590
takeaways.

478
06:59:53,600 --> 06:59:55,990
Treat every handoff to a sub agent as a

479
06:59:56,000 --> 06:59:59,350
network call. It can fail halfway. It

480
06:59:59,360 --> 07:00:02,230
can be slow and it can come back twice.

481
07:00:02,240 --> 07:00:05,350
Return a status, not just text. If the

482
07:00:05,360 --> 07:00:07,430
result cannot say partial, stale or

483
07:00:07,440 --> 07:00:09,590
unknown, the orchestrator will read it

484
07:00:09,600 --> 07:00:12,708
as

485
07:00:12,718 --> 07:00:15,110
start with our safe consistency model

486
07:00:15,120 --> 07:00:17,990
and enforce it in code. Keep what fits

487
07:00:18,000 --> 07:00:20,948
and change what does not. We already

488
07:00:20,958 --> 07:00:23,590
know how to build distribute systems. We

489
07:00:23,600 --> 07:00:25,110
just have to notice that we are building

490
07:00:25,120 --> 07:00:26,788
one.

491
07:00:26,798 --> 07:00:29,350
So these are some of the sources behind

492
07:00:29,360 --> 07:00:32,310
this talk.

493
07:00:32,320 --> 07:00:34,788
On the left side, it's my own work, my

494
07:00:34,798 --> 07:00:38,070
open source projects, my articles,

495
07:00:38,080 --> 07:00:40,948
internet drafts and my talks. On the

496
07:00:40,958 --> 07:00:42,708
right side,

497
07:00:42,718 --> 07:00:45,030
specs and standards

498
07:00:45,040 --> 07:00:46,548
and

499
07:00:46,558 --> 07:00:50,628
bunch of papers behind each concept.

500
07:00:50,638 --> 07:00:53,100
Thank you.