# Sub-Agents Are a Distributed System (And Nobody Gave You a Consistency Model) Source: https://www.youtube.com/watch?v=ZZ19zhYnDKY Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session. [06:37:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23874s) Hey folks, this is Gish, a principal [06:37:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23877s) software engineer at Microsoft. Outside [06:37:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23879s) my job, I build open source tools for AI [06:38:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23882s) agents. One of them is MCP pacemaker, a [06:38:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23886s) supervised bridge that keeps MCP servers [06:38:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23888s) running. [06:38:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23890s) Before we start, a small disclosure. The [06:38:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23893s) views in this talk are my own and [06:38:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23895s) doesn't represent my employer. [06:38:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23897s) So sub agents are a distributed system. [06:38:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23900s) Over the past year, I have been using [06:38:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23902s) agents and sub agents on real [06:38:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23904s) engineering work, dev sessions, code [06:38:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23907s) reviews, instant trashing [06:38:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23910s) what [06:38:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23912s) this talk is about what happens when one [06:38:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23914s) agent hands work to another. [06:38:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23918s) So here's an architecture. [06:38:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23921s) The orchestrator splits a task and hands [06:38:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23923s) the pieces to three different sub aents. [06:38:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23927s) They all run at the same time. [06:38:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23930s) They all read from the same shared [06:38:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23932s) state. Consization, memory files, repo, [06:38:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23936s) working trees, issues, vector stores. [06:39:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23941s) And they don't just think, they make [06:39:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23944s) tool calls to real systems [06:39:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23948s) which touch databases, APIs, CI tools, [06:39:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23951s) deployment pipelines [06:39:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23954s) and [06:39:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23955s) the results come back and orchestrator [06:39:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23958s) merges them into a single answer and [06:39:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23960s) orchestrator workers shared mutable [06:39:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23964s) state and results merge from concurrent [06:39:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23966s) executions. This represents a classic [06:39:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23969s) distributed system. [06:39:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23971s) I have known how to reason with dispute [06:39:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23973s) systems for decades, [06:39:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23975s) but none of the setup came with it. [06:39:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23979s) What you get is a function that takes a [06:39:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23981s) string and returns a string. [06:39:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23984s) Failures we hit are not new, are not [06:39:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23987s) specific to a agents. They're classic [06:39:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23990s) distributed system failures. [06:39:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23993s) partial failure. A normal call that [06:39:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23996s) fails halfway through and returns an [06:39:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=23999s) exception error code. In a sub aent [06:40:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24002s) case, it returns text that looks like a [06:40:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24004s) success. [06:40:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24006s) Dirty reads normally we have isolations [06:40:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24010s) versions. Here every worker reads one [06:40:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24013s) shade context including things other [06:40:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24016s) worker only guessed. [06:40:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24018s) Unbounded fan out. [06:40:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24020s) A busy service pushes back with the cues [06:40:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24025s) 429's 503s. [06:40:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24028s) In agents, orchestrator does not see any [06:40:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24031s) of this. It just starts in the worker. [06:40:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24034s) Retries. [06:40:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24036s) In normal distute systems, we build [06:40:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24039s) potency to make retries reliable. [06:40:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24042s) in agents a retrate sub agent and the [06:40:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24045s) whole task again with side effects [06:40:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24047s) included [06:40:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24049s) for each of these case [06:40:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24051s) I'll name the concept it already is then [06:40:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24054s) I'll put them together as a consistency [06:40:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24056s) model I'm proposing and walk through why [06:40:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24059s) each part is there [06:41:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24062s) start with the failure model because [06:41:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24063s) everything else depends on it a friendly [06:41:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24066s) case is a crash stop a process it was [06:41:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24069s) running correctly stops and stays is [06:41:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24072s) stopped. The sub agent throws an error [06:41:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24074s) in this box. It's easy to handle. [06:41:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24078s) Next, no reply. The call is lost or it's [06:41:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24081s) very late. We'll come back to this soon. [06:41:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24085s) The hot case is the one from Baantine [06:41:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24088s) general's paper. A component that gives [06:41:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24092s) conflicting information. The process [06:41:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24094s) answers and the answer is wrong. [06:41:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24097s) A half finished sub agent sits closest [06:41:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24099s) to the box. It is not malicious but from [06:41:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24102s) the outside it looks the same. It does [06:41:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24104s) not crash. It answers fluently. Let's [06:41:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24107s) jump into this. A sub agent that half [06:41:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24110s) finished does not raise. It returns [06:41:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24113s) normal looking text with the same type [06:41:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24115s) signature as success. [06:41:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24118s) I talked about one version of this in [06:42:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24120s) one of my talks. We pointed an agent at [06:42:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24123s) an LQ. [06:42:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24125s) Here's an illustration I used there with [06:42:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24128s) three different paths which gave the [06:42:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24130s) same answer. [06:42:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24131s) The first one [06:42:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24133s) it pulled the telemetry and compared [06:42:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24135s) with a known brand set. It actually did [06:42:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24138s) the work. The second one it matched [06:42:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24141s) against the title. It has seen before [06:42:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24144s) and gave the response. [06:42:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24147s) The third alert [06:42:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24149s) it didn't do anything. [06:42:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24151s) It just originated. [06:42:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24154s) All three are categorized as benign with [06:42:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24156s) the same level of confidence [06:42:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24159s) and nothing downstream can dull this [06:42:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24161s) apart. The orchestrator can read the [06:42:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24163s) conclusion. It cannot read how much work [06:42:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24165s) was behind it. [06:42:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24168s) Now the call gets no reply. [06:42:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24171s) The orchestrator send work and a timeout [06:42:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24174s) fired. From here three volts look the [06:42:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24178s) same. The sub agent is still working. It [06:43:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24181s) crashed and the reply was lost. [06:43:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24184s) It's a hall limit, not a tooling gap. [06:43:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24187s) The FLP paper from 1985 showed that in a [06:43:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24190s) synchronous system with even one faulty [06:43:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24192s) process, agreement can fail to finish. [06:43:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24196s) Chundra and 2x answer in 1996 was [06:43:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24198s) failure detectors that can make [06:43:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24200s) mistakes. They suspect did not know. I [06:43:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24204s) hit both directions [06:43:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24206s) in my own open source project MCP [06:43:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24210s) pacemaker supervised bridge. [06:43:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24213s) A server whose first run downloads [06:43:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24215s) itself hit the 30 secondond call time [06:43:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24218s) out and look broken. It was slow not [06:43:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24221s) dead. We fixed it by giving initialize [06:43:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24224s) its own time out. Now the other way the [06:43:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24228s) server crashed during a call. The call [06:43:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24230s) then timed out and the timeout overrode [06:43:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24233s) the exit code that explained the crash. [06:43:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24236s) We fixed it by recording the time out [06:43:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24239s) only when it was alive. [06:44:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24242s) So [06:44:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24243s) time out is a suspicion. Record it as [06:44:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24246s) unknown and keep the real cause. [06:44:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24249s) And now make failure a value. [06:44:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24255s) Let's take this illustration. [06:44:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24257s) A sub agent returns a status. What it [06:44:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24260s) did, what it did not do, the evidence, [06:44:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24264s) and any side effects. Partial is a [06:44:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24266s) normal answer here. [06:44:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24269s) The protocols already have a slot for [06:44:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24271s) this. In A2A, a task can end as [06:44:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24275s) completed, failed, canceled or rejected [06:44:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24279s) or stop and ask for input. In MCP, a [06:44:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24282s) dual result can say is error true. But a [06:44:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24286s) sub agent can still say completed while [06:44:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24290s) its text says something else. And a loop [06:44:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24293s) that only reads the text never sees a [06:44:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24295s) difference. [06:44:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24296s) Another spirit. [06:44:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24298s) Checking a condition and getting no is a [06:45:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24301s) different fact from not being able to [06:45:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24304s) check at all. [06:45:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24306s) My own individual internet draft on [06:45:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24308s) evaluation states [06:45:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24311s) talks about this. [06:45:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24313s) Collapse the two and you change what [06:45:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24315s) gets retrieded and who gets ped and what [06:45:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24318s) the audit says later. [06:45:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24321s) Next dirty reads. [06:45:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24324s) Let's take this madeup example from one [06:45:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24326s) of my talk. [06:45:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24328s) The sub agent the scratch it writes into [06:45:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24332s) a shade context [06:45:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24334s) saying that error rate fell by 80%. It [06:45:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24337s) did not measure that it inferred it. [06:45:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24341s) Now [06:45:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24343s) two other sub aents read from the same [06:45:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24345s) shade context. [06:45:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24348s) The second sub agent writes a review [06:45:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24350s) note and picks the same sentenc. [06:45:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24353s) The third agent writes a runbook step [06:45:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24356s) and uses it as a reason to proceed. [06:45:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24359s) The orchestrator merges all the all [06:46:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24362s) these three three artifacts agree. It is [06:46:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24365s) once unsupported sentence read three [06:46:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24367s) different times. [06:46:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24369s) The 1995 critique of SQL isolation [06:46:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24372s) levels define a dirty read as you read a [06:46:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24375s) value before the writer commits or rolls [06:46:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24377s) back. In a shared context, a guess is [06:46:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24380s) readable the moment it is written and [06:46:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24382s) nothing marks it as unchecked. [06:46:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24385s) A regular fix [06:46:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24388s) to is to ask three sub agents and get a [06:46:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24391s) quorum. [06:46:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24393s) Corums do work for a specific reason. In [06:46:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24396s) weighted voting from 1979, the read [06:46:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24399s) corum plus right corum is greater than [06:46:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24402s) total. [06:46:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24403s) So every read overlaps the latest right. [06:46:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24406s) The overlap is the guarantee. [06:46:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24409s) In sub aent's case, the three sub aents [06:46:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24411s) that read the same shared context does [06:46:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24414s) not give you that. They are not [06:46:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24415s) independent copies. They read the same [06:46:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24418s) guess. Their agreement is one source [06:47:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24421s) counted three different times. Let's [06:47:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24423s) pick a consistency model. [06:47:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24425s) In databases, we have vocabulary for [06:47:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24427s) this. [06:47:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24429s) Linearizable. [06:47:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24431s) Every operation appears to happen at one [06:47:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24433s) point in real time order. Sequential. [06:47:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24437s) One total order that respects each [06:47:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24439s) process one order. Causal [06:47:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24443s) operations that are causally related [06:47:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24445s) appear in the same order everywhere. [06:47:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24448s) Session guarantees. [06:47:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24450s) There are promises to one client like [06:47:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24452s) read your rights and monotonic reads [06:47:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24456s) eventual. If no new updates are made, [06:47:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24459s) reads eventually return the lost value. [06:47:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24462s) A shared context gives you none of [06:47:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24464s) these. There's no stated guarantee at [06:47:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24466s) all. [06:47:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24468s) And you don't need the top consistency [06:47:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24470s) models. [06:47:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24472s) The strongest models cannot stay [06:47:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24474s) available when the network splits. [06:47:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24477s) Causal end session guarantees can as [06:48:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24480s) long as each clients stick to one place [06:48:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24483s) for sub agents that place the [06:48:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24485s) orchestrator. So [06:48:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24488s) here's a consistency model I use for sub [06:48:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24491s) agents. I call it safe handoffs. [06:48:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24496s) S stands for source fees. [06:48:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24500s) Every fact a worker hands on carries its [06:48:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24504s) source version as of time and revision [06:48:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24508s) and decisions read only fresh source [06:48:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24510s) facts. [06:48:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24511s) A at most once effects [06:48:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24515s) every right has identity and a single [06:48:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24517s) use approval for that exact action. An [06:48:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24520s) outcome is read back never replayed. [06:48:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24524s) F fan out budgets. Each downstream gets [06:48:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24529s) a gate and a bounded weight and refusal [06:48:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24532s) counts as a fail. [06:48:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24534s) E stands for explicit outcomes. [06:48:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24537s) Every handoff ends in a name state and a [06:49:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24541s) timeout is unknown with its cost script. [06:49:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24544s) This consistency model is the one I use [06:49:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24547s) for my own sub agents. It's not a [06:49:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24549s) universal model. It's based off my own [06:49:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24552s) public work. Some of it is in my FCP [06:49:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24555s) pacemaker bridge and some are based on [06:49:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24558s) my other docs and my own internet draft. [06:49:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24564s) Now let's see what's behind each letter. [06:49:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24569s) Yes, source reads [06:49:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24572s) it's the answer to that reads where one [06:49:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24575s) worker's guess becomes another worker's [06:49:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24576s) fact. What helps is to treat every fact [06:49:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24580s) as a record not a sentence. [06:49:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24583s) Lampot's 1978 paper define when one [06:49:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24586s) event happened before another. The facts [06:49:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24588s) provenence is that record. What it was [06:49:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24591s) read from and when. [06:49:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24593s) Let's take this madeup example. [06:49:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24597s) the record names, the claim and its [06:50:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24600s) revision, the source and its version, [06:50:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24604s) the scope and three different times, [06:50:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24608s) when the data was current, when we read [06:50:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24610s) it and when we used it. [06:50:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24614s) Freshness is checked with the fact is [06:50:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24616s) used. [06:50:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24618s) In this illustration, the data ends at [06:50:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24622s) 9:05. We read it at 9:06 and we decided [06:50:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24626s) at 9:07. [06:50:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24628s) In this case, [06:50:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24630s) we picked a 5 minute limit. So the data [06:50:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24634s) decision at 9:07, it's fine. [06:50:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24637s) Now at 9:15, the second edition came in [06:50:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24642s) because our freshness limit is five. [06:50:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24644s) This is a stale record. [06:50:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24647s) We reject it [06:50:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24649s) and we say that the next station needs [06:50:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24651s) another read. [06:50:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24653s) This is how you separate an observation [06:50:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24655s) from an inference. An observation has a [06:50:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24658s) source and a read time. An inference has [06:51:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24660s) none. [06:51:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24662s) A most effect is about retries. [06:51:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24666s) This is an illustration [06:51:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24669s) from my article about MCPmaker bridge. [06:51:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24674s) The orchestrator delegates a task. The [06:51:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24678s) sub agent calls a tool which does a real [06:51:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24681s) right. [06:51:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24682s) like [06:51:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24684s) creating a PR, [06:51:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24686s) the reply is lost. [06:51:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24689s) The orchestrator knows one thing, the [06:51:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24691s) outcome is unknown. [06:51:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24693s) So it retries the whole task, the right [06:51:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24697s) happens again. [06:51:39](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24699s) This is a classic problem. In 1984 RPC [06:51:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24703s) paper, [06:51:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24705s) it says that if the call returns, the [06:51:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24708s) procedure ran precisely once. If you get [06:51:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24710s) an exception, it ran once or not at all. [06:51:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24714s) And you are not told which. [06:51:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24717s) So you pick a delivery semantic. [06:52:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24720s) At most once [06:52:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24723s) don't retry and accept the fact that it [06:52:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24725s) may be lost. At least once. Retry and [06:52:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24730s) accept that duplicates are possible. [06:52:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24733s) Exactly once. [06:52:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24735s) It happened at least once. And there's a [06:52:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24738s) way we can dduplicate [06:52:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24741s) maybe by using an item potency key. In [06:52:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24744s) MCP item potent hint is only a hint and [06:52:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24747s) da or false. If you need it, you build [06:52:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24750s) it. Here's a small design from my bridge [06:52:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24754s) on how I handle it. [06:52:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24756s) When a client comes back after a [06:52:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24758s) restart, the bridge replace the client's [06:52:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24761s) record initialize against a fresh [06:52:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24762s) server. That's safe. It's just a [06:52:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24764s) handshake and it does not restore tool [06:52:47](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24767s) state. [06:52:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24769s) What it never replace is an uncertain [06:52:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24771s) tool call. When a shared server is [06:52:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24774s) recycled, the bridge drains it and calls [06:52:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24776s) till unanswered at the deadline fail as [06:52:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24779s) potentially excluded. [06:53:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24782s) That status is honest. It tells the [06:53:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24784s) caller to go and check. [06:53:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24788s) What helped is giving every side effect [06:53:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24791s) an identity. Approval binds to that [06:53:14](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24794s) exact action. It's only used once. [06:53:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24798s) After the call, read back by that [06:53:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24800s) identity. [06:53:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24802s) Submitted is not confirmed. Confirmed [06:53:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24804s) means you observe the effect. [06:53:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24807s) The last part of a it shows up when work [06:53:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24810s) passes and resumes. A lease is a lock [06:53:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24813s) with a timeout. [06:53:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24815s) Martin Clipman's post about distributed [06:53:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24817s) locking shows this problem. [06:53:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24820s) A client passes while holding it. The [06:53:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24823s) lease expense. Someone else takes out [06:53:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24826s) and the first client wakes up and writes [06:53:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24828s) anyway. [06:53:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24830s) The fix he proposes a fencing token. A [06:53:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24834s) number that goes up every time the lock [06:53:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24836s) is granted. [06:53:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24838s) The storage remembers the highest token [06:54:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24841s) it has seen and rejects any right with [06:54:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24844s) the old one. [06:54:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24846s) Sub agents pause and resume all the [06:54:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24849s) time. In my other talk, as I put it, [06:54:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24853s) restore context, not side effects. So [06:54:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24856s) treat an approval like a fencing token. [06:54:19](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24859s) One action, one use. Checked at the [06:54:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24861s) gate, not remembered in the prompt. A [06:54:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24864s) conversation. [06:54:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24866s) My bridge fixed the same class of bug. A [06:54:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24870s) stale exit or a deleted resume can no [06:54:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24873s) longer remove or bring back a [06:54:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24874s) replacement session. [06:54:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24877s) Fan out budgets is about unbounded fan [06:54:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24880s) out. This is flow control or the lack of [06:54:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24883s) it. I hit this issue in my own MCV [06:54:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24886s) bridge before it had a limit on cold [06:54:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24888s) starts. [06:54:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24890s) Many sessions start their service at the [06:54:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24894s) same time through npx. [06:54:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24896s) They share one package cache. Every one [06:55:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24900s) of them failed with npm's lock [06:55:01](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24901s) compromis. [06:55:03](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24903s) Nothing in any single request was wrong. [06:55:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24906s) The failures was the concurrency itself. [06:55:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24909s) And when a downstream does push back, it [06:55:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24911s) can arrive as a plain text. This is MCP [06:55:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24915s) specs one example of a tool execution [06:55:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24917s) error. [06:55:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24918s) limit as one sentence. The model reads a [06:55:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24921s) sentence. The orchestrator code may [06:55:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24923s) never look the next worker starts [06:55:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24925s) anyway. [06:55:27](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24927s) So I introduce a kota budget in my [06:55:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24930s) bridge. [06:55:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24932s) Cole starts that use a package manager [06:55:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24935s) go through a gate two at a time by [06:55:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24938s) default. Everyone else waits and the [06:55:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24941s) weight is born at 15 seconds. Pass that [06:55:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24945s) the start goes ahead without a gate. [06:55:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24948s) And I documented why a corrupted cache [06:55:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24951s) is a risk while a hung client request is [06:55:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24954s) certainity. There is a choice for [06:55:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24956s) availability made on purpose and written [06:55:58](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24958s) down. [06:56:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24960s) And I also built a cap on sessions. [06:56:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24964s) At the cap, a new a new session waits up [06:56:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24967s) to 10 seconds and then it's refused. [06:56:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24970s) That refusal now counts as a failure in [06:56:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24972s) health. Another book capacity has to be [06:56:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24976s) reserved before the synchronous start [06:56:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24978s) not after otherwise [06:56:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24981s) cued starts or book the cap that is a [06:56:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24984s) check then act race for sub agents give [06:56:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24988s) the orchestrator a number per downstream [06:56:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24990s) a bounded weight and a refusate consent. [06:56:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24994s) Now [06:56:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=24995s) e the explicit outcomes every hand of [06:56:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25000s) ends in a name state like stale or [06:56:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25002s) unknown not just text. The last piece is [06:56:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25006s) who decided that state treats the agent [06:56:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25008s) output as untrusted input the same way [06:56:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25011s) you treat a request body from the [06:56:52](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25012s) internet. [06:56:54](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25014s) The orchestrator owns the checks in [06:56:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25017s) code. Does the result match the schema? [06:57:00](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25020s) Is the evidence there? Is it fresh [06:57:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25022s) enough? the numbers hold when you recmp [06:57:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25025s) compute them. If a check fails, the step [06:57:08](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25028s) returns a refusal with a reason. It does [06:57:11](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25031s) not pass the text along with a warning [06:57:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25033s) paragraph underneath. If the same model [06:57:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25036s) can skip a check and declare it past, [06:57:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25038s) you have a request, not a boundary. [06:57:21](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25041s) Here's a madeup example from another [06:57:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25044s) talk. A sub agent says that a rate fell. [06:57:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25048s) Before it was 10 errors in,000 requests. [06:57:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25051s) after it was six errors with 600 [06:57:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25054s) requests. It went from 10 to six and it [06:57:37](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25057s) claimed that our rate fell. But when we [06:57:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25060s) computed the rate, it's both 1%. [06:57:44](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25064s) We recomputed it and then the claim is [06:57:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25066s) refused. [06:57:48](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25068s) A supported answer is fine. A deliberate [06:57:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25070s) handoff to a person is fine. A confident [06:57:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25073s) guess it's not. [06:57:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25075s) So limitations because a consistency [06:57:57](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25077s) model is only useful when you know where [06:57:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25079s) we should start. [06:58:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25082s) So here's what safe model does not do. [06:58:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25087s) It does not make facts true. A source [06:58:10](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25090s) fact is checkable, not correct. It does [06:58:13](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25093s) not give you exactly once delivery. [06:58:17](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25097s) A lost reply stays unknown until you [06:58:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25100s) read it back. It does not roll anything [06:58:22](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25102s) back. [06:58:24](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25104s) Undoing a partial task is a compensation [06:58:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25106s) step you design. like the Saga's paper [06:58:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25109s) from 1987. [06:58:31](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25111s) And finally, there's no global order [06:58:33](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25113s) across workers, only causal order for a [06:58:36](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25116s) fact. [06:58:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25118s) And it has a price. CAP theorem says [06:58:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25122s) that when the network splits, you trade [06:58:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25125s) off availability against consistency. [06:58:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25129s) My bridge chose availability for cold [06:58:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25131s) starts on purpose. For rights, safe [06:58:55](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25135s) picks consistency over availability. It [06:58:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25139s) refuses rather than guesses. [06:59:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25142s) Packlac [06:59:04](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25144s) adds that even without failures, you [06:59:06](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25146s) trade latency for consistency. [06:59:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25149s) Every freshness check is another read [06:59:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25152s) and and every refusal needs an order. [06:59:16](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25156s) So where does not fit? [06:59:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25160s) If a sub agent only drafts text that a [06:59:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25163s) person reads before anything happens, [06:59:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25165s) most of this is a overhead. [06:59:28](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25168s) It also assumes that there's one single [06:59:30](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25170s) orchestrator. [06:59:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25172s) If you have more, this does not fit. [06:59:35](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25175s) Pure agents with no single place to [06:59:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25178s) stick to need something else. [06:59:41](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25181s) This is the consistency model I use for [06:59:43](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25183s) my own agents that touch actual real [06:59:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25185s) production systems. It's not universal. [06:59:49](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25189s) So wrapping up, here's three key [06:59:51](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25191s) takeaways. [06:59:53](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25193s) Treat every handoff to a sub agent as a [06:59:56](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25196s) network call. It can fail halfway. It [06:59:59](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25199s) can be slow and it can come back twice. [07:00:02](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25202s) Return a status, not just text. If the [07:00:05](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25205s) result cannot say partial, stale or [07:00:07](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25207s) unknown, the orchestrator will read it [07:00:09](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25209s) as [07:00:12](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25212s) start with our safe consistency model [07:00:15](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25215s) and enforce it in code. Keep what fits [07:00:18](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25218s) and change what does not. We already [07:00:20](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25220s) know how to build distribute systems. We [07:00:23](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25223s) just have to notice that we are building [07:00:25](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25225s) one. [07:00:26](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25226s) So these are some of the sources behind [07:00:29](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25229s) this talk. [07:00:32](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25232s) On the left side, it's my own work, my [07:00:34](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25234s) open source projects, my articles, [07:00:38](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25238s) internet drafts and my talks. On the [07:00:40](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25240s) right side, [07:00:42](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25242s) specs and standards [07:00:45](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25245s) and [07:00:46](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25246s) bunch of papers behind each concept. [07:00:50](https://www.youtube.com/watch?v=ZZ19zhYnDKY&t=25250s) Thank you.