# Your Sub-Agents Are a Distributed System. Treat Them Like One. - Girish Konda, Microsoft Girish Konda, Microsoft [Talk page](https://ai.engineer/talks/DxP5aW4VknQ-your-sub-agents-are-distributed-system-treat) · [Video](https://www.youtube.com/watch?v=DxP5aW4VknQ) ## Your Sub-Agents Are a Distributed System. Treat Them Like One. A worker can return a confident verdict after skipping an investigation, or create a pull request and lose the reply. Girish Konda’s SAFE handoffs give the orchestrator a way to decide which evidence it can use, which operations it can repeat, and when it has to stop. ### Three workers behind one function call [0:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=0s) An orchestrator splits a task among three sub-agents. They run concurrently, read shared state from memory files, repository working trees, issues, or vector stores, and call tools that reach databases, APIs, CI systems, and deployment pipelines. The orchestrator merges their replies into one answer. To its caller, the whole operation can look like a function that takes a string and returns a string. Girish Konda, a principal software engineer at Microsoft and the creator of **MCP Pacemaker**, encountered the consequences in engineering work and in his open-source bridge for keeping MCP servers running. Half-finished work can return fluent prose. One worker can read another’s guess. A service can signal overload while the orchestrator keeps starting workers. If a reply goes missing, retrying the worker can repeat every write it made on the first attempt. The string-returning interface leaves all four decisions to the caller. ### Three ways to classify an alert as benign [3:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=188s) In a crash-stop failure, a process that was running correctly stops and stays stopped. A worker that throws an error gives the caller an explicit failure to handle. Silence is harder to interpret. A fluent but wrong reply is harder still. Konda draws an analogy to a Byzantine failure from the caller’s point of view. An entirely well-intentioned worker can report success after an incomplete investigation, using the same return type it would use for a completed one. How much investigation went into a benign alert verdict? In Konda’s illustration, three paths produce the same classification at the same confidence level. - **Telemetry comparison.** The worker retrieved telemetry and performed a comparison before classifying the alert. - **Familiar title.** The worker recognized an alert title it had seen before and returned the associated answer. - **Unsupported verdict.** The worker performed no investigation and generated the answer anyway. The next decision might depend on whether anyone inspected *this alert’s* telemetry. Even retaining the confidence alongside the label gives the orchestrator no way to recover that information. ### What a 30-second timeout can hide [4:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=294s) When a deadline expires, the orchestrator knows how long it waited. The worker might still be running, it might have crashed, or it might have finished and lost the reply. The 1985 FLP result concerns the limits of reaching agreement when a process may fail and messages may be delayed. The practical connection is the difficulty of telling a delayed process from a dead one. A timeout-based failure detector can raise a suspicion and be wrong. MCP Pacemaker needed two different fixes for calls that appeared to time out. - **Slow first start.** A server downloaded itself on its first run and exceeded the ordinary 30-second call timeout. A working installation looked broken. Giving `initialize` its own timeout let startup have a different deadline from routine calls. - **Crash during a call.** A server exited while a call was outstanding. When the call later timed out, the bridge overwrote the exit code that explained the crash. Pacemaker now records the timeout only if the server is still alive. When the process has exited, the caller keeps the more informative cause. A handoff can represent uncertainty directly. The worker returns a status, the work it completed, the work still outstanding, its evidence, and any side effects. `partial` is an ordinary result. An unresolved timeout yields `unknown`, with any underlying diagnostic preserved. A2A already lets a task complete, fail, be canceled or rejected, or pause for input. An MCP tool result can set `isError` to `true`. The orchestrator still has to read those signals. A worker can mark a task complete while its account of the work says otherwise. Konda’s individual Internet-Draft on evaluation states makes a further distinction. A check that ran and returned false establishes something about the condition. A check that could not run leaves the condition open. Collapsing both into a negative result changes which operation gets retried, who gets paged, and what a later audit says was checked. ### How one guessed percentage becomes three agreeing artifacts [7:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=448s) Where did the apparent corroboration come from? In a hypothetical incident, a worker writes into shared context that the error rate fell by 80%. It inferred the change. A second worker picks up the sentence for a review note. A third reads the same context and uses the drop to justify proceeding in a runbook. When the orchestrator merges the three artifacts, the percentage appears in an investigation, a review, and an operational decision. Tracing either downstream branch leads straight back to the first worker. The database analogy is a **dirty read**. A reader sees a value before the writer commits or rolls back. In shared agent context, a sentence becomes readable the moment a worker writes it. Without a marker for unchecked material, another worker can act on a guess before anyone has established where it came from. Would asking three agents for a quorum help? In the 1979 weighted-voting scheme Konda invokes, the read quorum weight plus the write quorum weight exceeds the total voting weight. That inequality forces every read quorum to overlap the latest write quorum. Counting agreeing agent answers supplies no comparable guarantee. Here, all three answers inherit one unsupported sentence. Where do the three accounts of an 80% drop get their evidence? Follow the arrows into the merge. The review and runbook each inherit the first worker’s inference. Neither branch adds a measurement. - First worker’s finding — Writes an inferred 80% error-rate drop into shared context. - Review note — Repeats the sentence. - Runbook step — Uses the drop as a reason to proceed. - Orchestrator’s merge — Receives three agreeing artifacts with one source. - First worker’s finding → Review note (Read from shared context) - First worker’s finding → Runbook step (Read from shared context) - First worker’s finding → Orchestrator’s merge (Original finding) - Review note → Orchestrator’s merge (Review artifact) - Runbook step → Orchestrator’s merge (Runbook artifact) ### Choose what the orchestrator can promise [9:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=549s) A consistency model says what a reader may rely on when operations overlap. **Linearizability** makes each operation appear to occur at one point in real-time order. **Sequential consistency** gives everyone one total order that respects each process’s own order. **Causal consistency** preserves the order of operations that depend on each other. **Session guarantees** can let a client read its own writes and avoid moving backward on successive reads. **Eventual consistency** promises convergence after updates stop. Giving workers a shared context leaves all of these questions open. Requiring a single real-time order has an availability cost when the network splits. Causal and session guarantees leave room for a client to keep working if it stays attached to one place. In Konda’s design, that place is the orchestrator. His **SAFE handoffs** draw on Pacemaker, his other engineering work, and his individual draft. - **S — Source reads.** A worker passes on the source, version, and time behind a fact. The decision that uses it checks whether it is still fresh. - **A — At-most-once effects.** Each write has an identity and a single-use approval for that action. If its outcome is uncertain, the caller reads it back. - **F — Fan-out budgets.** Each downstream dependency gets a limit and a bounded wait. When the system refuses work, the refusal counts as a failure. - **E — Explicit outcomes.** Each handoff ends in a named state. A timeout can leave the outcome `unknown`, and a known cause survives for diagnosis. ### S · Check the source at decision time [11:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=695s) A worker handing on a fact supplies a record of the observation. The record names the claim and its revision, the source and its version, and the scope over which the claim applies. It also distinguishes three times that a single timestamp would conflate. When was the source data current? When did the worker read it? When did a decision use it? How long can the next worker rely on that read? In Konda’s example, the data covers events through 9:05. A worker reads it at 9:06, and a decision uses it at 9:07. The data is two minutes old at the decision, within the chosen five-minute limit. At 9:15 the same data is ten minutes old. The orchestrator rejects the stale record and requires another read before the next decision. Copying the claim into a new note does nothing to advance the time through which the data was current. The same record requirement exposes the problem with the earlier 80% drop. A worker that observed the drop could identify the source and when it read it. The worker that guessed has no such observation to hand on. ### A · Recover a write by its identity [13:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=789s) Suppose a sub-agent calls a tool to create a pull request. The tool makes the write, but the reply is lost. From the orchestrator’s point of view the outcome is `unknown`. If it delegates the whole task again, the worker reaches the creation step a second time. The first pull request can already exist while the orchestrator is still waiting to learn whether anything happened. The 1984 remote procedure call example makes the ambiguity precise. Under its semantics, a normal return means the procedure ran once. An exception leaves two possibilities, zero executions or one. Forgoing a retry preserves at-most-once execution at the cost of possibly losing the operation. Retrying for at-least-once execution admits duplicates. Getting exactly one observable effect requires both eventual execution and a way for the receiver to recognize duplicate attempts, for example with an idempotency key. MCP’s idempotency hint describes a tool property. The tool implementation has to supply the behavior. Pacemaker distinguishes two operations during recovery. When a client reconnects after a restart, the bridge can replay the recorded `initialize` handshake against a fresh server. That reconnects the client without restoring tool state. When recycling a shared server, the bridge gives outstanding calls time to drain. If a tool call is still unanswered at the deadline, Pacemaker reports that it may already have run and leaves the caller to check. Replaying that call could duplicate a write. SAFE gives each side effect an identity and binds a single-use approval to that particular action. After submission, the caller uses the identity to read back the result. For the pull request, confirmation means observing that the intended pull request exists. Remembering that the worker sent the request establishes only submission. Resuming work creates a related problem. Consider a client holding a lease, a lock with an expiry time. The client pauses, the lease expires, and another client takes over. The first client wakes up and tries to write with its old credentials. A **fencing token** lets storage reject that late write. Each lock grant gets a higher number. Storage remembers the highest token it has seen and rejects writes carrying an older one. For a resumed agent, the corresponding check belongs at the point that permits the write. Restoring a conversation can recover the plan. The system still needs to check whether the approval covers this action and whether it has already been used. Pacemaker fixed a related session bug in which an old exit or resume event could interfere with a replacement session. ### F · Budget concurrency at the resource workers share [16:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1003s) Before Pacemaker limited cold starts, many sessions launched servers through `npx` at the same time. They all used one package cache. Every start in that burst failed with npm’s lock-compromised error, even though each individual request was valid. To schedule the next start, the bridge needed to know how many others were already using the cache. A downstream service can also report that it is overloaded. It might return HTTP `429` or `503`. In the MCP tool-error example, the model receives a rate limit as a sentence. If the code that starts workers never inspects that result, the model can read the explanation while the orchestrator sends the next worker straight into the same limit. Pacemaker uses two different admission policies, each with an explicit decision at the deadline. - **Package-manager cold starts.** By default, two starts at a time pass through the gate. Other starts wait up to 15 seconds. Once that wait expires, a start proceeds without the gate. Konda accepted the risk of cache corruption so a stuck gate could not hang a client indefinitely. - **Session capacity.** At the session cap, a new session waits up to 10 seconds for room. If capacity is still unavailable, the bridge refuses the session and counts the refusal as a failure in its health reporting. Enforcing the session cap requires reserving capacity before starting a session. If queued starts each check that space exists and record their use only after startup, several can claim the same remaining slot. A limit of the right size can still fail through this check-then-act race. Reserving the slot makes the next admission decision account for work already committed to start. ### E · Let code decide whether the claim passes [18:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1122s) A worker can propose a status as readily as it can propose a conclusion. To make states such as `stale` and `unknown` meaningful, the orchestrator treats the result like a request body received over the internet. Code checks the schema, looks for the evidence, checks its freshness, and recomputes numbers where possible. If a check fails, the step returns a refusal with a reason. That keeps the failed claim out of the next worker’s inputs, where a warning attached underneath it would be easy to lose. Did the error rate fall when the error count went from ten to six? In Konda’s worked example, the first period had 10 errors per 1,000 requests. The second had 6 errors per 600 requests. Dividing each count by its own request volume gives `10 / 1,000 = 1%` and `6 / 600 = 1%`. The count fell along with traffic. The rate held steady. Recomputing the rates gives the orchestrator a specific reason to refuse the claimed improvement. The worker can return the supported finding that both rates were 1%, or hand the decision to a person. Either outcome lets the investigation continue without passing a spurious improvement into the runbook. ### What SAFE costs, and where it fits [20:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1201s) Provenance makes a claim checkable. The source itself can still be wrong. At-most-once effects leave room for zero executions, and a lost reply leaves the outcome open until read-back. If a multi-step task changed the world before failing, undoing those changes requires a compensation step designed for that action, as in the 1987 Sagas work. Ordering is similarly scoped. A reader can trace the causal history of a particular fact while independent workers’ operations have no single global order. CAP describes the trade between consistency and availability during a network partition. Konda makes the practical choice operation by operation. Pacemaker lets a cold start through after a bounded wait because the client needs progress. For a write whose safety the orchestrator cannot establish, SAFE accepts the availability cost of refusal. Even on a healthy network, checking freshness can add another read to the decision path. Someone also has to own the work a refusal leaves outstanding. The intended setting is one orchestrator delegating work to agents that touch production systems. If a worker only drafts text and a person reads it before anything happens, much of this machinery is overhead. Multiple orchestrators or peer agents need another way to coordinate because they lack the single place SAFE assumes for enforcing decisions. For one orchestrator, a useful starting question is what its code will do when a worker returns `partial`, `stale`, or `unknown`. Each state calls for a different next step. Inspect what remains unfinished, refresh the evidence, or find out whether the write already happened. Giving the caller those choices lets a handoff end honestly while the task itself is still open. ## Key takeaways - A deadline tells the orchestrator that a reply is late. Preserve any known process exit cause, and resolve an uncertain write by reading back its effect. - Trace apparent agreement back to its observations. Three artifacts can repeat one guess, and a once-valid observation can expire before the next decision. - Choose the deadline behavior per operation. Pacemaker lets a package-manager cold start proceed after 15 seconds of waiting, but refuses a new session if the session cap is still full after 10 seconds. ## Slides ### Sub-Agents Are a Distributed System (And Nobody Gave You a… Girish Konda asks what one agent can assume about the state of the work when another hands it a task. In a distributed system, those assumptions depend on a consistency model. [0:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=0s) [AI Engineer Code Summit 2026 title slide naming Girish Konda as a principal software engineer at Microsoft. A contact QR code appears beside the speaker credit.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffabc93035c5b42e0edac78fdaf18e605c4d14d16739698f82da255fd1af6f0e.webp) Sub-Agents Are a Distributed System (And Nobody Gave You a Consistency Model) AI ENGINEER CODE SUMMIT 2026 Girish Konda Principal Software Engineer, Microsoft LET’S CONNECT go.girishkonda.com The views expressed in this talk are my own and do not represent those of my employer, Microsoft Corporation. Interface text: Girish Konda go.girishkonda.com ### Sub-agents are a distributed system Konda introduces the architecture with a handoff. An orchestrator splits one task among three sub-agents. They work in parallel and read the same shared state. [0:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=43s) [The architecture slide contains only the heading “Sub-agents are a distributed system” above an empty diagram area. The words “a distributed system” appear in red.](https://assets.ai.engineer/sites/aiecode2025/sha256/03f96124e3a96a8d71a86d106a828a92c4d157ac8726c2d255a39c7f69b1aca1.webp) 02 · ARCHITECTURE Sub-agents are a distributed system. AI Engineer CODE Summit 2026 · Girish Konda 2 Girish Konda go.girishkonda.com ### Sub-agents are a distributed system An orchestrator assigns tasks to three workers. As they run concurrently, the workers read and write shared state, and their tool calls can change a database or trigger a release. The orchestrator merges their results into one answer. [1:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=79s) [An orchestrator assigns tasks to three workers. They read and write shared state, call external systems, and return results along red arrows. The orchestrator merges the results into one answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/f322da2269e37d2ebc95ad951781c21f674965f7e3b5241eff9099c3f1f2afdb.webp) 02 · ARCHITECTURE Sub-agents are a distributed system. Orchestrator Merge into one answer task result Worker A Worker B Worker C sub-agent sub-agent sub-agent write read Shared state conversation scratchpad or memory file repo working tree issue or ticket vector store tool call tool call tool call REAL SYSTEMS · REAL SIDE EFFECTS Database reads and writes Cloud API rate-limited CI builds and tests Deploy pipeline releases AI Engineer CODE Summit 2026 · Girish Konda 2 Girish Konda go.girishkonda.com ### Plan for worse than crash-stop. A crash-stop gives the caller a clear failure. A lost or late reply leaves it waiting. A half-finished sub-agent comes closest to the hardest case, Byzantine failure, because it can give a fluent but wrong answer. [3:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=210s) [An arrow orders three failure models from easier to harder to handle. A process stops, a reply is lost or late, or a component returns a wrong answer. The last card is highlighted and labeled arbitrary (Byzantine).](https://assets.ai.engineer/sites/aiecode2025/sha256/d204b37c339081dcc111fea4ff6858b53d5d02daa3b8899a906a2784517bfd8b.webp) 04 · FAILURE MODELS Plan for worse than crash-stop. EASIER TO HANDLE → HARDER CRASH-STOP Stops, and stays stopped. A sub-agent that throws. Easy to handle. NO REPLY Lost, or very late. Nothing to branch on. We come back to this one. ARBITRARY (BYZANTINE) Answers, and the answer is wrong. A half-finished sub-agent sits closest to this box. AI Engineer CODE Summit 2026 · Girish Konda 4 Girish Konda go.girishkonda.com ### A half-finished sub-agent does not raise. In this example, the orchestrator sees the same “Benign · 0.94” result whether the sub-agent compared telemetry against known-benign data, matched a familiar title, or read nothing. The score gives no clue which check ran. [4:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=271s) [Comparing telemetry with known-benign data, matching a familiar title, and reading nothing all produce the same green “Benign · 0.94” label. A footnote identifies the score as illustrative.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b143b67d22ea49ddf4b6df24806f338ef6db887fb28829a4de1ce182453e4e2.webp) 05 · PARTIAL FAILURE A half-finished sub-agent does not raise. Benign · 0.94 Pulled the telemetry and compared the known-benign set Benign · 0.94 Matched the title against something seen before Benign · 0.94 Read nothing at all Nothing downstream can tell these apart. Illustrative example · 0.94 is not a measured accuracy AI Engineer CODE Summit 2026 · Girish Konda 5 Girish Konda go.girishkonda.com ### Make failure a value. In Konda’s example, a sub-agent returns a partial result. It read the 09:00–09:05 logs but never ran the comparison with known-benign data. Recording the unfinished check keeps the caller from mistaking it for a negative finding. [6:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=380s) [A partial result records that the logs were read and the comparison is still pending. Beside it are A2A task states and an MCP error flag. A banner distinguishes a negative finding from an inability to check.](https://assets.ai.engineer/sites/aiecode2025/sha256/73944da47954bf3004310059c476e6d7bd1e41c2366db212d9508f8ce38d632f.webp) 07 · FAILURE AS A VALUE Make failure a value. status partial done read the logs, 09:00–09:05 not_done compare with known-benign evidence source, version, as-of effects none A2A task states completed · failed · canceled · rejected input-required · auth-required MCP tool result isError: true Checked and got no ≠ could not check Illustrative format, not a standard · draft-konda-agentproto-evaluation-state-02 AI Engineer CODE Summit 2026 · Girish Konda 7 Girish Konda go.girishkonda.com ### One shared context, no isolation. In this example, a triage agent writes “Error rate fell 80%” into shared context without measuring it. Two agents reuse the claim in a review note and a runbook step. At merge time, all three artifacts agree because they share one source. [7:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=453s) [A triage agent puts an unmeasured claim of an 80% error-rate drop into shared context. Two other agents reuse it in a review note and a runbook step. The merge counts three agreeing artifacts.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3b04c549ff384fb46d7090c93f1bf7031eca23a0a2c19ffa588601ed051280c.webp) 08 · ISOLATION One shared context, no isolation. Sub-agent A triage Shared context “Error rate fell 80%” inferred, never measured Sub-agent B review note Sub-agent C runbook step Merge: three artifacts agree One unsupported sentence, read three times. Illustrative example · Dirty read (P1): Berenson et al. 1995 AI Engineer CODE Summit 2026 · Girish Konda 8 Girish Konda go.girishkonda.com ### Three agents agreeing is not a quorum. With five replicas, reading from three guarantees that at least one took part in the latest write to three. When three agents read the same guess from shared context, their unanimous vote counts one source three times. [8:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=534s) [Among five circles, a read quorum of three and a write quorum of three overlap at one circle. Alongside, three sub-agents vote unanimously using the same guess from shared context.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3bfa2dc9ca53bf61986982e68ad53debb2e2add1816d7b300d3a4c39b7bc7a7.webp) 09 · QUORUMS Three agents agreeing is not a quorum. WEIGHTED VOTING write quorum w = 3 read quorum r = 3 r + w > N → 3 + 3 > 5 Every read overlaps the latest write. THREE SUB-AGENTS Sub-agent A Sub-agent B Sub-agent C Shared context: the same guess Vote: 3 of 3 agree One source, counted three times. Weighted voting: Gifford 1979 AI Engineer CODE Summit 2026 · Girish Konda 9 Girish Konda go.girishkonda.com ### Pick a consistency model on purpose. Shared context gives sub-agents no stated consistency guarantee. Konda builds SAFE on causal consistency and session guarantees because both can stay available during a network split when each client sticks to one orchestrator. [9:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=554s) [Five rows describe linearizable, sequential, causal, session and eventual guarantees. A bracket marks causal consistency and session guarantees as the basis for SAFE. A separate box says shared context has no stated guarantee.](https://assets.ai.engineer/sites/aiecode2025/sha256/32b871c2352e43f1073cef819d111803de046da7022cc6780ad0ac8b669770dc.webp) 10 · CONSISTENCY MODELS Pick a consistency model on purpose. ↑ STRONGER Linearizable one point in time per operation, real-time order stalls in a partition Sequential one total order, each process's order kept stalls in a partition Causal causally related operations in the same order sticky available Session guarantees read your writes; reads never go backwards sticky available Eventual no new updates: reads reach the last value A shared context no stated guarantee What SAFE builds on Definitions: Jepsen, Consistency Models (jepsen.io) AI Engineer CODE Summit 2026 · Girish Konda 10 Girish Konda go.girishkonda.com ### SAFE hand-offs: the model I use. Konda’s sub-agents use fresh, sourced facts, require single-use approval for each write and reserve capacity before dispatching work. If a write times out, they record the outcome as unknown, keep the cause and read back the result. [11:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=660s) [Four stacked rows spell SAFE: sourced reads, at-most-once effects, fan-out budgets and explicit outcomes. The final row classifies a timeout as unknown and retains its cause.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd587eb7aa97716709fd8ef555250cad6b01d0527d03752ce73bd2e1b611eebb.webp) 11 · SAFE HAND-OFFS SAFE hand-offs: the model I use. S Sourced reads Every fact carries source, version, as-of time and revision. Decisions read only fresh, sourced facts, never a replaced revision. BORROWS causal metadata · monotonic reads · read your writes A At-most-once effects Every write has an identity and a single-use approval for that exact action. Unknown outcome: read back, never replay. BORROWS at-most-once RPC · leases · fencing tokens F Fan-out budgets A gate per downstream, a bounded wait, capacity reserved before async work. A refusal counts as a failure. BORROWS back-pressure · admission control E Explicit outcomes positive · negative · not established · stale · unknown · escalated A timeout is unknown, cause kept. A guess is not an outcome. BORROWS unreliable failure detectors The consistency model I use for sub-agents. Not universal. Built from MCP Pacemaker, my earlier talks and my evaluation-state draft AI Engineer CODE Summit 2026 · Girish Konda 11 Girish Konda go.girishkonda.com ### Provenance is causal metadata. One agent can mistake another agent’s guess for a fact. With sourced reads, the agent passing along a claim records where and when it read it, so the next agent can trace the claim to its source. [11:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=694s) [Slide 12 shows the title “Provenance is causal metadata.” and a sourced reads label above an empty example area.](https://assets.ai.engineer/sites/aiecode2025/sha256/31c8605d13b33e7ec22294d6e0e9e17fbc8d80b5e2516bbd43cd62ba040f94ea.webp) 12 · CAUSALITY Provenance is causal metadata. S · SOURCED READS Illustrative example · Happened-before: Lamport 1978 AI Engineer CODE Summit 2026 · Girish Konda 12 Girish Konda go.girishkonda.com ### Provenance is causal metadata. The 09:06 telemetry read covers data through 09:05. Under the example’s five-minute limit, the data is fresh enough for the 09:07 decision. By 09:15, even an agent repeating the summary needs to read the source again before deciding. [12:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=720s) [A telemetry claim is current as of 09:05, read at 09:06 and used for a decision at 09:07. With a five-minute freshness limit, the timeline marks it stale at 09:15.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb3751214355ece8a3c3cd5d0b2a321a3837703fc58bd15c2e9c8d9e4e191a51.webp) Provenance is causal metadata. 12 · CAUSALITY S · SOURCED READS claim error-rate change, rev 2 source telemetry query, pinned scope service, config, window as_of 09:05 read_at 09:06 used_at 09:07 state established-negative Timeline: 09:05 as-of · 09:06 read · 09:07 decide ✓ · 09:15 stale × Limit in this example: 5 minutes A repeated summary is not a new read. Read again before the next decision. Illustrative example · Happened-before: Lamport 1978 AI Engineer CODE Summit 2026 · Girish Konda 12 Interface text: Girish Konda go.girishkonda.com ### A retry runs the whole task again. The tool writes, but its reply gets lost. When the orchestrator retries the task, the sub-agent calls the tool again. An idempotency key lets the tool recognize the repeated call and skip the second write. [14:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=843s) [A lost reply leads to a retry and a second write. Three cards explain that skipping retries can lose an effect, retrying can duplicate it, and an idempotency key can deduplicate repeated attempts.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dbbc221d2edd97ac476f159102ec12e7d41059305f8ced637895b11af3ed8a8.webp) A retry runs the whole task again. 13 · DELIVERY SEMANTICS A · AT-MOST-ONCE EFFECTS Orchestrator Sub-agent Tool delegate the task tool call write happens reply lost × outcome unknown retry the whole task same tool call second write AT-MOST-ONCE Never retry. An effect may be lost. AT-LEAST-ONCE Retry. Duplicates are possible. EXACTLY-ONCE EFFECTS At-least-once + dedup by an idempotency key Illustrative example, adapted from the MCP Pacemaker article · Birrell & Nelson 1984 AI Engineer CODE Summit 2026 · Girish Konda 13 Girish Konda go.girishkonda.com ### Replaying the handshake and confirming side effects After a restart, the bridge can replay the initialize handshake. An unanswered write may already have happened. Give each effect an ID, approve that action once, and use the ID to check whether the write took effect. [14:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=893s) [The initialize handshake is safe to replay. An uncertain tool call may already have executed, and resuming can repeat a write. The flow assigns the effect an identity, approves the exact action once and reads it back by identity to confirm it.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d0f37636a47bd8ddd31f88c734ce9be53c668bcc3f4735778efecbca3409125.webp) Replay the handshake. Not the side effect. 14 · SIDE EFFECTS A · AT-MOST-ONCE EFFECTS SAFE TO REPLAY ✓ The initialize handshake does not restore tool state NEVER REPLAY BLIND × An uncertain tool call fail it as potentially executed × A write after a resume restore context, not side effects Give the effect an identity → Approve that exact action, once → Read back by identity → Confirmed MCP Pacemaker CHANGELOG 1.0.0, 1.3.0 AI Engineer CODE Summit 2026 · Girish Konda 14 Girish Konda go.girishkonda.com ### A resumed worker can be a stale worker. Client 1’s lease expires while it’s paused. Storage accepts Client 2’s write with fencing token 34 and remembers that number. When Client 1 resumes and writes with token 33, storage rejects the write because it has already seen 34. [15:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=932s) [Client 1 pauses past its lease expiry. Client 2 acquires a new lease, and storage accepts its write with fencing token 34. When client 1 resumes, storage rejects its write carrying the older token 33.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e0e1f59095018e6434bc729e038298305cad8d1dc7ed6c8acd2cbfc7d82142a.webp) A resumed worker can be a stale worker. 15 · LEASES AND FENCING A · AT-MOST-ONCE EFFECTS Client 1 token 33 lease long pause lease expires wakes up, writes anyway Client 2 token 34 lease Storage write 34 ✓ accepts 34 write 33 × rejects 33: saw 34 Example from Kleppmann 2016, fencing tokens AI Engineer CODE Summit 2026 · Girish Konda 15 Interface text: Girish Konda go.girishkonda.com ### The orchestrator cannot see the limits. Concurrent npx sessions shared one package cache and all failed on its lock. A tool can report an API rate limit in text the model reads. The orchestrator may miss that signal and start the next worker anyway. [17:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1022s) [Arrows from many sessions launched via npx converge on one shared package cache. A callout beneath the cache says its lock was compromised and every session failed.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d9b5fe4ac0a4b54f41fbd794f02bec3b0d8db043907828308b79e46eb87cd12.webp) 16 · BACK-PRESSURE F · FAN-OUT BUDGETS The orchestrator cannot see the limits. Many sessions, each via npx One shared package cache Lock compromised every session failed MCP Pacemaker CHANGELOG 1.2.0 · MCP spec 2025-06-18, tool execution error example AI Engineer CODE Summit 2026 · Girish Konda 16 Girish Konda go.girishkonda.com [17:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1034s) [Arrows from many npx sessions converge on one shared package cache. A lock error caused every session to fail. A separate tool result reports an API rate limit in text that the model reads and the orchestrator may overlook.](https://assets.ai.engineer/sites/aiecode2025/sha256/be170a2ccbe9131128d47fe7af2da84ea1e63989055c5071ad75f2343de81a7b.webp) 16 · BACK-PRESSURE F · FAN-OUT BUDGETS The orchestrator cannot see the limits. Many sessions, each via npx One shared package cache Lock compromised every session failed A rate limit, as a tool result "isError": true "text": "Failed to fetch weather data: API rate limit exceeded" The model reads the sentence. The orchestrator may never look. MCP Pacemaker CHANGELOG 1.2.0 · MCP spec 2025-06-18, tool execution error example AI Engineer CODE Summit 2026 · Girish Konda 16 Girish Konda go.girishkonda.com ### What SAFE does not do. SAFE assumes one orchestrator. It favors consistency over write availability, and each freshness check costs a read. Teams still have to design rollback. For drafts a person reviews before any action, most of SAFE is overhead. [21:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1282s) [Three columns list the limits of SAFE’s guarantees, the costs of consistency and freshness checks, and two cases where SAFE is a poor fit: human-reviewed drafts and peer agents without an orchestrator.](https://assets.ai.engineer/sites/aiecode2025/sha256/694a9b27c0fb93765cd2013c4840e75095038b1c782ddd4bac8e3d3af85e9671.webp) 19 · LIMITS What SAFE does not do. DOES NOT GUARANTEE True facts Exactly-once delivery Rollback: design a saga A global order WHAT IT COSTS Writes: consistency over availability (CAP) Freshness: a read per check (PACELC) Every refusal needs an owner WHERE IT DOES NOT FIT Drafts a person reads before anything happens Peer agents with no orchestrator The model I use for agents that touch real systems. Not universal. CAP: Gilbert & Lynch 2002 · PACELC: Abadi 2012 AI Engineer CODE Summit 2026 · Girish Konda 19 Girish Konda go.girishkonda.com ### Sources — code, writing, talks and standards Konda lists his own work on the left, including MCP Pacemaker and an Internet-Draft. The right column points to MCP for tool errors, A2A for task states and idempotency, and the RFC sections for HTTP 429 and 503. [22:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1351s) [The left column lists MCP Pacemaker, an article, an Internet-Draft and two talks. The right column cites the MCP and A2A specifications and the RFC sections covering HTTP 429 and 503.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8540ccacd4c960c5d5a584113b08b9d2f71ab036b7df4945bb26c8adbdfde46.webp) 21 · SOURCES Sources MY CODE, WRITING AND TALKS MCP Pacemaker CHANGELOG 1.0.0–1.3.0 github.com/girishkvs/mcp-pacemaker “Your MCP Servers Die Every Time the CLI Exits” Medium and Substack, 8 Oct 2026 draft-konda-agentproto-evaluation-state-02 Individual Internet-Draft, work in progress SREday Dallas 2026, 23 Oct “A Thousand Alerts a Day, and One Real Incident…” Conf42 Feature Flags 2026 “Your Agent Is Confidently Wrong” SPECS AND STANDARDS MCP specification 2025-06-18 Tools: isError, outputSchema, annotations A2A specification Task states; section 3.3.1, idempotency RFC 6585, section 4 HTTP 429 Too Many Requests RFC 9110, section 15.6.4 HTTP 503 Service Unavailable AI Engineer CODE Summit 2026 · Girish Konda 21 Girish Konda go.girishkonda.com ### Sources — distributed systems Konda’s reading list starts with Lamport’s 1978 paper on how to order events across a distributed system. Later entries address what a session can guarantee when replicas disagree and how to coordinate access with distributed locks. [22:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1370s) [Two columns cite work on event ordering, replication, remote procedure calls, consensus, sagas, locking and consistency models. Entries include authors and titles.](https://assets.ai.engineer/sites/aiecode2025/sha256/8dbde05561ebcfabb688e2b6d6cfb15ce163fa2d1ef0a4e38571b5c615e20420.webp) 22 · SOURCES Sources PAPERS Lamport 1978 · CACM Time, Clocks, and the Ordering of Events in a Distributed System Lamport 1979 · IEEE Trans. Computers How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs Gifford 1979 · SOSP Weighted Voting for Replicated Data Lamport, Shostak & Pease 1982 · TOPLAS The Byzantine Generals Problem Birrell & Nelson 1984 · TOCS Implementing Remote Procedure Calls Fischer, Lynch & Paterson 1985 · JACM Impossibility of Distributed Consensus with One Faulty Process Garcia-Molina & Salem 1987 · SIGMOD Sagas Gray & Cheriton 1989 · SOSP Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency Herlihy & Wing 1990 · TOPLAS Linearizability: A Correctness Condition for Concurrent Objects Terry et al. 1994 · PDIS Session Guarantees for Weakly Consistent Replicated Data Ahamad et al. 1995 · Distributed Computing Causal Memory: Definitions, Implementation, and Programming Berenson et al. 1995 · SIGMOD A Critique of ANSI SQL Isolation Levels Chandra & Toueg 1996 · JACM Unreliable Failure Detectors for Reliable Distributed Systems Gilbert & Lynch 2002 · SIGACT News Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services Vogels 2009 · CACM Eventually Consistent Abadi 2012 · IEEE Computer Consistency Tradeoffs in Modern Distributed Database System Design Kleppmann 2016 · blog How to do distributed locking Jepsen · jepsen.io Consistency Models AI Engineer CODE Summit 2026 · Girish Konda 22 Girish Konda go.girishkonda.com ### Thank you Girish Konda closes with a rule for sub-agents: “Treat every hand-off as a network call.” He lists his GitHub and LinkedIn profiles and displays go.girishkonda.com beneath a contact QR code. [22:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1375s) [A large thank-you heading appears above the reminder to treat every hand-off as a network call. The speaker’s profile links appear on the left and a QR code labeled “Let’s connect” appears on the right.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0e3abb858d147b80fa6ed108639585b095e67baba290a53af18000eddbb70ce.webp) 23 · THANK YOU Thank you Treat every hand-off as a network call. Girish Konda github.com/girishkvs · linkedin.com/in/venkatasaigirishkonda LET’S CONNECT go.girishkonda.com The views expressed in this talk are my own and do not represent those of my employer, Microsoft Corporation. Nothing here is an endorsement, a commitment, or a statement of product direction. Examples are public or made up, and contain no customer data. AI Engineer CODE Summit 2026 · Girish Konda 23 Interface text: Girish Konda go.girishkonda.com ## Full transcript [0:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=0s) Hey folks, this is Gish, a principal [0:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=3s) Hey folks, this is Gish, a principal [0:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=3s) Hey folks, this is Gish, a principal software engineer at Microsoft. Outside [0:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=5s) software engineer at Microsoft. Outside [0:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=6s) software engineer at Microsoft. Outside my job, I build open source tools for AI [0:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=9s) my job, I build open source tools for AI [0:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=9s) my job, I build open source tools for AI agents. One of them is MCP pacemaker, a [0:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=12s) agents. One of them is MCP pacemaker, a [0:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=12s) agents. One of them is MCP pacemaker, a supervised bridge that keeps MCP servers [0:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=15s) supervised bridge that keeps MCP servers [0:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=15s) supervised bridge that keeps MCP servers running. [0:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=16s) running. [0:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=16s) running. Before we start, a small disclosure. The [0:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=19s) Before we start, a small disclosure. The [0:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=19s) Before we start, a small disclosure. The views in this talk are my own and [0:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=21s) views in this talk are my own and [0:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=21s) views in this talk are my own and doesn't represent my employer. [0:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=24s) doesn't represent my employer. [0:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=24s) doesn't represent my employer. So, sub agents are a distributed system. [0:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=27s) So, sub agents are a distributed system. [0:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=27s) So, sub agents are a distributed system. Over the past year, I have been using [0:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=29s) Over the past year, I have been using [0:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=29s) Over the past year, I have been using agents and sub agents on real [0:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=30s) agents and sub agents on real [0:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=30s) agents and sub agents on real engineering work, dev sessions, code [0:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=33s) engineering work, dev sessions, code [0:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=33s) engineering work, dev sessions, code reviews, instant trashing, [0:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=36s) reviews, instant trashing, [0:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=36s) reviews, instant trashing, whatnot. [0:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=38s) whatnot. [0:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=38s) whatnot. This talk is about what happens when one [0:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=41s) This talk is about what happens when one [0:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=41s) This talk is about what happens when one agent hands work to another. [0:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=44s) agent hands work to another. [0:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=44s) agent hands work to another. So, here's an architecture. [0:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=47s) So, here's an architecture. [0:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=47s) So, here's an architecture. The orchestrator splits a task and hands [0:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=50s) The orchestrator splits a task and hands [0:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=50s) The orchestrator splits a task and hands the pieces to three different sub aents. [0:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=53s) the pieces to three different sub aents. [0:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=53s) the pieces to three different sub aents. They all run at the same time. [0:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=56s) They all run at the same time. [0:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=56s) They all run at the same time. They all read from the same shared [0:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=58s) They all read from the same shared [0:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=58s) They all read from the same shared state. Consization, memory files, repo, [1:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=63s) state. Consization, memory files, repo, [1:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=63s) state. Consization, memory files, repo, working trees, issues, vector stores. [1:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=67s) working trees, issues, vector stores. [1:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=68s) working trees, issues, vector stores. And they don't just think, they make [1:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=70s) And they don't just think, they make [1:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=70s) And they don't just think, they make tool calls to real systems [1:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=74s) tool calls to real systems [1:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=74s) tool calls to real systems which touch databases, APIs, CI tools, [1:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=78s) which touch databases, APIs, CI tools, [1:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=78s) which touch databases, APIs, CI tools, deployment pipelines [1:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=80s) deployment pipelines [1:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=80s) deployment pipelines and [1:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=82s) and [1:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=82s) and the results come back and orchestrator [1:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=85s) the results come back and orchestrator [1:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=85s) the results come back and orchestrator merges them into a single answer. An [1:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=87s) merges them into a single answer. An [1:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=87s) merges them into a single answer. An orchestrator workers shared mutable [1:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=90s) orchestrator workers shared mutable [1:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=90s) orchestrator workers shared mutable state and results merge from concurrent [1:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=93s) state and results merge from concurrent [1:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=93s) state and results merge from concurrent executions. This represents a classic [1:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=95s) executions. This represents a classic [1:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=95s) executions. This represents a classic distributed system. [1:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=97s) distributed system. [1:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=97s) distributed system. I have known how to reason with dispute [1:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=99s) I have known how to reason with dispute [1:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=99s) I have known how to reason with dispute systems for decades, [1:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=101s) systems for decades, [1:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=101s) systems for decades, but none of the setup came with it. [1:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=105s) but none of the setup came with it. [1:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=105s) but none of the setup came with it. What you get is a function that takes a [1:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=108s) What you get is a function that takes a [1:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=108s) What you get is a function that takes a string and returns a string. [1:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=110s) string and returns a string. [1:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=110s) string and returns a string. Failures we hit are not new, are not [1:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=113s) Failures we hit are not new, are not [1:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=114s) Failures we hit are not new, are not specific to a agents. They're classic [1:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=116s) specific to a agents. They're classic [1:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=116s) specific to a agents. They're classic distributed system failures. [2:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=120s) distributed system failures. [2:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=120s) distributed system failures. partial failure, a normal call that [2:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=122s) partial failure, a normal call that [2:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=122s) partial failure, a normal call that fails halfway through and returns an [2:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=125s) fails halfway through and returns an [2:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=125s) fails halfway through and returns an exception error code. In a sub agent [2:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=128s) exception error code. In a sub agent [2:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=128s) exception error code. In a sub agent case, it returns text that looks like a [2:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=131s) case, it returns text that looks like a [2:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=131s) case, it returns text that looks like a success. [2:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=133s) success. [2:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=133s) success. Dirty reads normally we have isolations [2:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=136s) Dirty reads normally we have isolations [2:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=136s) Dirty reads normally we have isolations versions. Here every worker reads one [2:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=139s) versions. Here every worker reads one [2:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=140s) versions. Here every worker reads one shade context including things other [2:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=142s) shade context including things other [2:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=142s) shade context including things other worker only guessed. [2:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=145s) worker only guessed. [2:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=145s) worker only guessed. Unbounded fan out. [2:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=147s) Unbounded fan out. [2:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=147s) Unbounded fan out. A busy service pushes [clears throat] [2:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=149s) A busy service pushes [clears throat] [2:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=150s) A busy service pushes [clears throat] back with the cues 429's 503s. [2:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=154s) back with the cues 429's 503s. [2:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=154s) back with the cues 429's 503s. In agents, orchestrator does not see any [2:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=157s) In agents, orchestrator does not see any [2:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=157s) In agents, orchestrator does not see any of this. It just starts in the worker. [2:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=161s) of this. It just starts in the worker. [2:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=161s) of this. It just starts in the worker. Retries. [2:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=162s) Retries. [2:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=162s) Retries. In normal distute systems, we build [2:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=165s) In normal distute systems, we build [2:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=165s) In normal distute systems, we build potency to make retries reliable. [2:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=168s) potency to make retries reliable. [2:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=168s) potency to make retries reliable. in agents a reiterate sub agent runs the [2:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=171s) in agents a reiterate sub agent runs the [2:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=171s) in agents a reiterate sub agent runs the whole task again with side effects [2:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=173s) whole task again with side effects [2:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=174s) whole task again with side effects included [2:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=175s) included [2:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=175s) included for each of these case [2:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=178s) for each of these case [2:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=178s) for each of these case I'll name the concept it already is then [3:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=181s) I'll name the concept it already is then [3:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=181s) I'll name the concept it already is then I'll put them together as a [3:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=182s) I'll put them together as a [3:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=182s) I'll put them together as a [clears throat] consistency model I'm [3:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=183s) [clears throat] consistency model I'm [3:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=184s) [clears throat] consistency model I'm proposing and walk through why each part [3:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=186s) proposing and walk through why each part [3:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=186s) proposing and walk through why each part is there [3:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=188s) is there [3:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=188s) is there start with the failure model because [3:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=190s) start with the failure model because [3:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=190s) start with the failure model because everything else depends on it a friendly [3:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=193s) everything else depends on it a friendly [3:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=193s) everything else depends on it a friendly case is a crash stop a process it was [3:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=196s) case is a crash stop a process it was [3:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=196s) case is a crash stop a process it was running correctly stops and stays is [3:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=198s) running correctly stops and stays is [3:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=198s) running correctly stops and stays is stopped. The subway agent that throws an [3:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=200s) stopped. The subway agent that throws an [3:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=200s) stopped. The subway agent that throws an error in this box. It's easy to handle. [3:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=204s) error in this box. It's easy to handle. [3:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=204s) error in this box. It's easy to handle. Next, no reply. The call is lost or it's [3:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=207s) Next, no reply. The call is lost or it's [3:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=207s) Next, no reply. The call is lost or it's very late. We'll come back to this soon. [3:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=211s) very late. We'll come back to this soon. [3:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=211s) very late. We'll come back to this soon. The hot case is the one from Baantine [3:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=215s) The hot case is the one from Baantine [3:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=215s) The hot case is the one from Baantine general sniper. A component that gives [3:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=218s) general sniper. A component that gives [3:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=218s) general sniper. A component that gives conflicting information. The process [3:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=220s) conflicting information. The process [3:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=220s) conflicting information. The process answers and the answer is wrong. [3:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=223s) answers and the answer is wrong. [3:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=223s) answers and the answer is wrong. A half finished sub agent sits closest [3:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=225s) A half finished sub agent sits closest [3:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=225s) A half finished sub agent sits closest to the box. It is not malicious but from [3:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=228s) to the box. It is not malicious but from [3:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=228s) to the box. It is not malicious but from the outside it looks the same. It does [3:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=230s) the outside it looks the same. It does [3:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=230s) the outside it looks the same. It does not crash. It answers fluently. Let's [3:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=233s) not crash. It answers fluently. Let's [3:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=233s) not crash. It answers fluently. Let's jump into this. A sub agent that half [3:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=237s) jump into this. A sub agent that half [3:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=237s) jump into this. A sub agent that half finished does not raise. It returns [3:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=239s) finished does not raise. It returns [4:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=240s) finished does not raise. It returns normal looking text with the same type [4:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=242s) normal looking text with the same type [4:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=242s) normal looking text with the same type signature as success. [4:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=244s) signature as success. [4:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=244s) signature as success. I talked about one version of this in [4:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=247s) I talked about one version of this in [4:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=247s) I talked about one version of this in one of my talks. We pointed an agent at [4:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=250s) one of my talks. We pointed an agent at [4:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=250s) one of my talks. We pointed an agent at an LQ. [4:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=252s) an LQ. [4:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=252s) an LQ. Here's an illustration I used there with [4:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=254s) Here's an illustration I used there with [4:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=254s) Here's an illustration I used there with three different paths which gave the [4:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=256s) three different paths which gave the [4:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=256s) three different paths which gave the same answer. [4:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=258s) same answer. [4:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=258s) same answer. The first one [4:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=260s) The first one [4:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=260s) The first one it pulled the telemetry and compared [4:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=262s) it pulled the telemetry and compared [4:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=262s) it pulled the telemetry and compared with a nonbrand set. It actually did the [4:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=265s) with a nonbrand set. It actually did the [4:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=265s) with a nonbrand set. It actually did the work. The second one it matched against [4:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=268s) work. The second one it matched against [4:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=268s) work. The second one it matched against the title. It it has seen before and [4:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=271s) the title. It it has seen before and [4:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=271s) the title. It it has seen before and gave the response. [4:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=273s) gave the response. [4:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=273s) gave the response. The third alert [4:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=275s) The third alert [4:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=275s) The third alert it didn't do anything. [4:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=278s) it didn't do anything. [4:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=278s) it didn't do anything. It just hallated. [4:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=281s) It just hallated. [4:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=281s) It just hallated. All three are categorized as benign with [4:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=283s) All three are categorized as benign with [4:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=283s) All three are categorized as benign with the same level of confidence [4:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=285s) the same level of confidence [4:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=285s) the same level of confidence and nothing downstream can tell these [4:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=287s) and nothing downstream can tell these [4:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=287s) and nothing downstream can tell these apart. The orchestrator can read the [4:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=290s) apart. The orchestrator can read the [4:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=290s) apart. The orchestrator can read the conclusion. It cannot read how much work [4:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=292s) conclusion. It cannot read how much work [4:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=292s) conclusion. It cannot read how much work was behind it. [4:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=294s) was behind it. [4:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=294s) was behind it. Now the call gets no reply. [4:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=298s) Now the call gets no reply. [4:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=298s) Now the call gets no reply. The orchestrator send work and the [5:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=300s) The orchestrator send work and the [5:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=300s) The orchestrator send work and the timeout fired. From here three volts [5:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=304s) timeout fired. From here three volts [5:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=304s) timeout fired. From here three volts look the same. The sub agent is still [5:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=306s) look the same. The sub agent is still [5:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=306s) look the same. The sub agent is still working. It crashed and the reply was [5:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=309s) working. It crashed and the reply was [5:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=309s) working. It crashed and the reply was lost. [5:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=311s) lost. [5:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=311s) lost. It's a hall limit, not a tooling gap. [5:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=313s) It's a hall limit, not a tooling gap. [5:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=313s) It's a hall limit, not a tooling gap. The FLP paper from 1985 showed that in a [5:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=316s) The FLP paper from 1985 showed that in a [5:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=316s) The FLP paper from 1985 showed that in a synchronous system with even one faulty [5:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=319s) synchronous system with even one faulty [5:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=319s) synchronous system with even one faulty process, agreement can fail to finish. [5:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=322s) process, agreement can fail to finish. [5:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=322s) process, agreement can fail to finish. Chundra and 2x answer in 1996 was [5:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=325s) Chundra and 2x answer in 1996 was [5:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=325s) Chundra and 2x answer in 1996 was failure detectors that can make [5:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=326s) failure detectors that can make [5:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=326s) failure detectors that can make mistakes. They suspect did not know. I [5:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=330s) mistakes. They suspect did not know. I [5:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=330s) mistakes. They suspect did not know. I hit both directions [5:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=333s) hit both directions [5:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=333s) hit both directions in my own open source project MCP [5:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=336s) in my own open source project MCP [5:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=336s) in my own open source project MCP pacemaker supervised bridge [5:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=339s) pacemaker supervised bridge [5:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=339s) pacemaker supervised bridge a server whose first run downloads [5:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=342s) a server whose first run downloads [5:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=342s) a server whose first run downloads itself hit the 30 secondond call timeout [5:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=345s) itself hit the 30 secondond call timeout [5:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=345s) itself hit the 30 secondond call timeout and look broken it was slow not dead we [5:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=349s) and look broken it was slow not dead we [5:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=349s) and look broken it was slow not dead we fixed it by giving initialize its own [5:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=351s) fixed it by giving initialize its own [5:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=351s) fixed it by giving initialize its own timeout now the other way the server [5:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=354s) timeout now the other way the server [5:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=354s) timeout now the other way the server crashed during a call the call then [5:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=357s) crashed during a call the call then [5:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=357s) crashed during a call the call then timed out and the timeout overrode the [6:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=360s) timed out and the timeout overrode the [6:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=360s) timed out and the timeout overrode the exit code that explained the crash. We [6:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=363s) exit code that explained the crash. We [6:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=363s) exit code that explained the crash. We fixed it by recording the time out only [6:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=366s) fixed it by recording the time out only [6:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=366s) fixed it by recording the time out only when it was alive. [6:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=368s) when it was alive. [6:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=368s) when it was alive. So [6:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=370s) So [6:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=370s) So time out is a suspicion. Record it as [6:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=372s) time out is a suspicion. Record it as [6:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=372s) time out is a suspicion. Record it as unknown and keep the real cause. [6:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=376s) unknown and keep the real cause. [6:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=376s) unknown and keep the real cause. And now make failure a value. [6:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=381s) And now make failure a value. [6:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=381s) And now make failure a value. Let's take this illustration. [6:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=383s) Let's take this illustration. [6:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=384s) Let's take this illustration. A sub agent returns a status. what it [6:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=387s) A sub agent returns a status. what it [6:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=387s) A sub agent returns a status. what it did, what it did not do, the evidence, [6:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=390s) did, what it did not do, the evidence, [6:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=390s) did, what it did not do, the evidence, and any side effects. Partial is a [6:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=393s) and any side effects. Partial is a [6:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=393s) and any side effects. Partial is a normal answer here. [6:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=395s) normal answer here. [6:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=395s) normal answer here. The protocols already have a slot for [6:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=398s) The protocols already have a slot for [6:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=398s) The protocols already have a slot for this. In A2A, a task can end as [6:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=401s) this. In A2A, a task can end as [6:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=401s) this. In A2A, a task can end as completed, failed, cancelled, or [6:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=404s) completed, failed, cancelled, or [6:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=404s) completed, failed, cancelled, or rejected or stop and ask for input. In [6:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=407s) rejected or stop and ask for input. In [6:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=407s) rejected or stop and ask for input. In MCP, a dual result can say is error [6:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=411s) MCP, a dual result can say is error [6:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=411s) MCP, a dual result can say is error true. But a sub agent can still say [6:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=415s) true. But a sub agent can still say [6:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=415s) true. But a sub agent can still say completed while its text says something [6:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=417s) completed while its text says something [6:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=417s) completed while its text says something else. And a loop that only reads the [7:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=420s) else. And a loop that only reads the [7:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=420s) else. And a loop that only reads the text never sees the difference. [7:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=423s) text never sees the difference. [7:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=423s) text never sees the difference. Another split. [7:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=425s) Another split. [7:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=425s) Another split. Checking a condition and getting no is a [7:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=428s) Checking a condition and getting no is a [7:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=428s) Checking a condition and getting no is a different fact from not being able to [7:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=430s) different fact from not being able to [7:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=430s) different fact from not being able to check at all. [7:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=432s) check at all. [7:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=432s) check at all. My own individual internet draft on [7:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=435s) My own individual internet draft on [7:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=435s) My own individual internet draft on evaluation states [7:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=437s) evaluation states [7:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=437s) evaluation states talks about this. [7:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=439s) talks about this. [7:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=440s) talks about this. Collapse the two and you change what [7:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=442s) Collapse the two and you change what [7:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=442s) Collapse the two and you change what gets retrieded and who gets ped and what [7:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=445s) gets retrieded and who gets ped and what [7:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=445s) gets retrieded and who gets ped and what the audit says later. [7:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=447s) the audit says later. [7:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=448s) the audit says later. Next dirty reads. [7:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=451s) Next dirty reads. [7:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=451s) Next dirty reads. Let's take this madeup example from one [7:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=453s) Let's take this madeup example from one [7:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=453s) Let's take this madeup example from one of my talk. [7:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=455s) of my talk. [7:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=455s) of my talk. The sub agent the trash it writes into a [7:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=459s) The sub agent the trash it writes into a [7:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=459s) The sub agent the trash it writes into a shade context [7:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=460s) shade context [7:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=460s) shade context saying that error rate fell by 80%. It [7:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=464s) saying that error rate fell by 80%. It [7:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=464s) saying that error rate fell by 80%. It did not measure that it inferred it. [7:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=467s) did not measure that it inferred it. [7:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=467s) did not measure that it inferred it. Now [7:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=470s) Now [7:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=470s) Now two other sub aents read from the same [7:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=472s) two other sub aents read from the same [7:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=472s) two other sub aents read from the same shade context. [7:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=474s) shade context. [7:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=474s) shade context. The second sub aent writes a review note [7:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=477s) The second sub aent writes a review note [7:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=477s) The second sub aent writes a review note and picks the same sentence up. The [8:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=480s) and picks the same sentence up. The [8:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=480s) and picks the same sentence up. The third agent writes a runbook step and [8:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=482s) third agent writes a runbook step and [8:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=482s) third agent writes a runbook step and uses it as a reason to proceed. [8:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=485s) uses it as a reason to proceed. [8:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=485s) uses it as a reason to proceed. The orchestrator merges all the all [8:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=488s) The orchestrator merges all the all [8:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=488s) The orchestrator merges all the all these three three artifacts agree. It is [8:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=491s) these three three artifacts agree. It is [8:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=491s) these three three artifacts agree. It is once unsupported sentence read three [8:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=494s) once unsupported sentence read three [8:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=494s) once unsupported sentence read three different times. [8:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=496s) different times. [8:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=496s) different times. The 1995 critique of SQL isolation [8:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=498s) The 1995 critique of SQL isolation [8:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=498s) The 1995 critique of SQL isolation levels define a dirty read as you read a [8:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=501s) levels define a dirty read as you read a [8:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=501s) levels define a dirty read as you read a value before the writer commits or rolls [8:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=503s) value before the writer commits or rolls [8:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=503s) value before the writer commits or rolls back. In a shared context, a guess is [8:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=507s) back. In a shared context, a guess is [8:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=507s) back. In a shared context, a guess is readable the moment it is written and [8:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=509s) readable the moment it is written and [8:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=509s) readable the moment it is written and nothing marks it as unchecked. [8:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=512s) nothing marks it as unchecked. [8:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=512s) nothing marks it as unchecked. A regular fix [8:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=514s) A regular fix [8:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=514s) A regular fix to is to ask three sub agents and get a [8:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=517s) to is to ask three sub agents and get a [8:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=517s) to is to ask three sub agents and get a quorum. [8:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=519s) quorum. [8:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=519s) quorum. Corums do work for a specific reason. In [8:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=523s) Corums do work for a specific reason. In [8:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=523s) Corums do work for a specific reason. In weighted voting from 1979, the read [8:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=526s) weighted voting from 1979, the read [8:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=526s) weighted voting from 1979, the read corum plus right corum is greater than [8:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=528s) corum plus right corum is greater than [8:48](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=528s) corum plus right corum is greater than total. [8:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=530s) total. [8:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=530s) total. So every read overlaps the latest right. [8:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=532s) So every read overlaps the latest right. [8:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=532s) So every read overlaps the latest right. The overlap is the guarantee. [8:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=535s) The overlap is the guarantee. [8:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=535s) The overlap is the guarantee. In sub aent's case, the three sub aents [8:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=538s) In sub aent's case, the three sub aents [8:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=538s) In sub aent's case, the three sub aents that read the same shared context does [9:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=540s) that read the same shared context does [9:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=540s) that read the same shared context does not give you that. They are not [9:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=542s) not give you that. They are not [9:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=542s) not give you that. They are not independent copies. They read the same [9:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=544s) independent copies. They read the same [9:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=544s) independent copies. They read the same guess. Their agreement is one source [9:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=547s) guess. Their agreement is one source [9:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=547s) guess. Their agreement is one source counted three different times. Let's [9:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=549s) counted three different times. Let's [9:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=549s) counted three different times. Let's pick a consistency model. [9:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=552s) pick a consistency model. [9:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=552s) pick a consistency model. In databases, we have vocabulary for [9:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=554s) In databases, we have vocabulary for [9:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=554s) In databases, we have vocabulary for this. [9:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=555s) this. [9:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=556s) this. Linearizable. [9:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=557s) Linearizable. [9:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=557s) Linearizable. Every operation appears to happen at one [9:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=559s) Every operation appears to happen at one [9:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=559s) Every operation appears to happen at one point in real time order. Sequential. [9:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=563s) point in real time order. Sequential. [9:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=563s) point in real time order. Sequential. One total order that respects each [9:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=565s) One total order that respects each [9:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=565s) One total order that respects each process own order. Causal [9:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=569s) process own order. Causal [9:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=569s) process own order. Causal operations that are causally related [9:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=572s) operations that are causally related [9:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=572s) operations that are causally related appear in the same order everywhere. [9:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=574s) appear in the same order everywhere. [9:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=574s) appear in the same order everywhere. Session guarantees. [9:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=576s) Session guarantees. [9:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=576s) Session guarantees. There are promises to one client like [9:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=579s) There are promises to one client like [9:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=579s) There are promises to one client like read your rights and monotonic reads [9:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=582s) read your rights and monotonic reads [9:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=582s) read your rights and monotonic reads eventual. If no new updates are made, [9:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=585s) eventual. If no new updates are made, [9:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=586s) eventual. If no new updates are made, reads eventually return the last value. [9:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=589s) reads eventually return the last value. [9:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=589s) reads eventually return the last value. A shared context gives you none of [9:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=590s) A shared context gives you none of [9:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=590s) A shared context gives you none of these. There's no stated guarantee at [9:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=593s) these. There's no stated guarantee at [9:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=593s) these. There's no stated guarantee at all. [9:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=594s) all. [9:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=594s) all. And you don't need the top consistency [9:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=597s) And you don't need the top consistency [9:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=597s) And you don't need the top consistency models. [9:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=599s) models. [9:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=599s) models. The strongest models cannot stay [10:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=601s) The strongest models cannot stay [10:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=601s) The strongest models cannot stay available when the network splits. [10:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=604s) available when the network splits. [10:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=604s) available when the network splits. Causal end session guarantees can as [10:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=606s) Causal end session guarantees can as [10:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=606s) Causal end session guarantees can as long as each clients stick to one place [10:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=609s) long as each clients stick to one place [10:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=609s) long as each clients stick to one place for sub agents that place the [10:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=612s) for sub agents that place the [10:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=612s) for sub agents that place the orchestrator. So [10:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=615s) orchestrator. So [10:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=615s) orchestrator. So here's a consistency model I use for sub [10:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=617s) here's a consistency model I use for sub [10:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=617s) here's a consistency model I use for sub agents. I call it safe handoffs. [10:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=622s) agents. I call it safe handoffs. [10:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=622s) agents. I call it safe handoffs. S stands for source feeds. [10:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=626s) S stands for source feeds. [10:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=626s) S stands for source feeds. Every fact a worker hands on carries its [10:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=630s) Every fact a worker hands on carries its [10:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=630s) Every fact a worker hands on carries its source version as of time and revision [10:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=634s) source version as of time and revision [10:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=634s) source version as of time and revision and decisions read only fresh source [10:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=636s) and decisions read only fresh source [10:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=636s) and decisions read only fresh source facts. [10:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=638s) facts. [10:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=638s) facts. A at most once effects [10:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=641s) A at most once effects [10:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=641s) A at most once effects every right has an identity and a single [10:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=644s) every right has an identity and a single [10:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=644s) every right has an identity and a single use approval for that exact action. An [10:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=646s) use approval for that exact action. An [10:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=646s) use approval for that exact action. An outcome [clears throat] is read back [10:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=649s) outcome [clears throat] is read back [10:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=649s) outcome [clears throat] is read back never replayed. [10:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=651s) never replayed. [10:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=651s) never replayed. F fan out budgets. Each downstream gets [10:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=655s) F fan out budgets. Each downstream gets [10:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=656s) F fan out budgets. Each downstream gets a gate and a bounded weight and refusal [10:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=658s) a gate and a bounded weight and refusal [10:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=658s) a gate and a bounded weight and refusal counts as a failure. [11:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=661s) counts as a failure. [11:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=661s) counts as a failure. E stands for explicit outcomes. [11:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=664s) E stands for explicit outcomes. [11:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=664s) E stands for explicit outcomes. Every handoff ends in a named state and [11:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=667s) Every handoff ends in a named state and [11:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=667s) Every handoff ends in a named state and a timeout is unknown with its cost kept. [11:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=671s) a timeout is unknown with its cost kept. [11:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=671s) a timeout is unknown with its cost kept. This consistency model is the one I use [11:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=674s) This consistency model is the one I use [11:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=674s) This consistency model is the one I use for my own sub agents. It's not a [11:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=675s) for my own sub agents. It's not a [11:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=675s) for my own sub agents. It's not a universal model. It's based off my own [11:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=678s) universal model. It's based off my own [11:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=678s) universal model. It's based off my own public work. Some of it is in my FCP [11:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=681s) public work. Some of it is in my FCP [11:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=682s) public work. Some of it is in my FCP pacemaker bridge and some are based on [11:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=684s) pacemaker bridge and some are based on [11:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=684s) pacemaker bridge and some are based on my other docs and my own internet draft. [11:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=690s) my other docs and my own internet draft. [11:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=690s) my other docs and my own internet draft. Now let's see what's behind each letter. [11:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=695s) Now let's see what's behind each letter. [11:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=695s) Now let's see what's behind each letter. Yes, source reads. [11:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=698s) Yes, source reads. [11:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=698s) Yes, source reads. It's the answer to that reads where one [11:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=701s) It's the answer to that reads where one [11:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=701s) It's the answer to that reads where one worker's guess becomes another worker's [11:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=703s) worker's guess becomes another worker's [11:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=703s) worker's guess becomes another worker's fact. What helps is to treat every fact [11:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=707s) fact. What helps is to treat every fact [11:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=707s) fact. What helps is to treat every fact as a record, not a sentence. [11:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=709s) as a record, not a sentence. [11:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=709s) as a record, not a sentence. Lampot's 1978 paper define when one [11:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=712s) Lampot's 1978 paper define when one [11:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=712s) Lampot's 1978 paper define when one event happened before another. A fact's [11:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=715s) event happened before another. A fact's [11:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=715s) event happened before another. A fact's provenence is that record. What it was [11:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=717s) provenence is that record. What it was [11:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=717s) provenence is that record. What it was read from and when. [12:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=720s) read from and when. [12:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=720s) read from and when. Let's take this madeup example. [12:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=724s) Let's take this madeup example. [12:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=724s) Let's take this madeup example. the record names, the claim and its [12:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=727s) the record names, the claim and its [12:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=727s) the record names, the claim and its revision, the source and its version, [12:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=730s) revision, the source and its version, [12:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=730s) revision, the source and its version, the scope and three different times, [12:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=734s) the scope and three different times, [12:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=734s) the scope and three different times, when the data was current, when we read [12:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=737s) when the data was current, when we read [12:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=737s) when the data was current, when we read it and when we used it. [12:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=740s) it and when we used it. [12:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=740s) it and when we used it. Freshness is checked where the fact is [12:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=742s) Freshness is checked where the fact is [12:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=742s) Freshness is checked where the fact is used. [12:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=744s) used. [12:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=744s) used. In this illustration, the data ends at [12:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=748s) In this illustration, the data ends at [12:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=748s) In this illustration, the data ends at 9:05. We read it at 9:06 and we decided [12:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=753s) 9:05. We read it at 9:06 and we decided [12:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=753s) 9:05. We read it at 9:06 and we decided at 9:07. [12:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=755s) at 9:07. [12:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=755s) at 9:07. In this case, [12:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=757s) In this case, [12:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=757s) In this case, we picked a 5 minute limit. So the data [12:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=761s) we picked a 5 minute limit. So the data [12:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=761s) we picked a 5 minute limit. So the data decision at 9:07, it's fine. Now at [12:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=765s) decision at 9:07, it's fine. Now at [12:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=765s) decision at 9:07, it's fine. Now at 9:15, the second edition came in [12:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=769s) 9:15, the second edition came in [12:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=769s) 9:15, the second edition came in because our freshness limit is five. [12:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=771s) because our freshness limit is five. [12:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=771s) because our freshness limit is five. This is a stale record. [12:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=773s) This is a stale record. [12:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=773s) This is a stale record. We reject it [12:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=776s) We reject it [12:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=776s) We reject it and we say that the next station needs [12:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=777s) and we say that the next station needs [12:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=777s) and we say that the next station needs another read. [13:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=780s) another read. [13:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=780s) another read. This is how you separate an observation [13:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=782s) This is how you separate an observation [13:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=782s) This is how you separate an observation from an inference. An observation has a [13:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=784s) from an inference. An observation has a [13:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=784s) from an inference. An observation has a source and a read time. An inference has [13:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=787s) source and a read time. An inference has [13:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=787s) source and a read time. An inference has none. [13:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=789s) none. [13:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=789s) none. A most of it is about retries. [13:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=793s) A most of it is about retries. [13:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=793s) A most of it is about retries. This is an illustration [13:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=795s) This is an illustration [13:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=796s) This is an illustration from my article about MCPmaker bridge. [13:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=801s) from my article about MCPmaker bridge. [13:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=801s) from my article about MCPmaker bridge. The orchestrator delegates a task. The [13:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=804s) The orchestrator delegates a task. The [13:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=804s) The orchestrator delegates a task. The sub agent calls a tool which does a real [13:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=807s) sub agent calls a tool which does a real [13:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=807s) sub agent calls a tool which does a real right. [13:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=809s) right. [13:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=809s) right. like [13:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=810s) like [13:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=810s) like creating a PR [13:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=813s) creating a PR [13:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=813s) creating a PR the reply is lost. [13:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=815s) the reply is lost. [13:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=815s) the reply is lost. The orchestrator knows one thing the [13:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=817s) The orchestrator knows one thing the [13:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=817s) The orchestrator knows one thing the outcome is unknown. [13:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=819s) outcome is unknown. [13:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=820s) outcome is unknown. So it retries the whole task the right [13:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=823s) So it retries the whole task the right [13:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=823s) So it retries the whole task the right happens again. [13:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=825s) happens again. [13:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=825s) happens again. This is a classic problem. In 1984 RPC [13:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=830s) This is a classic problem. In 1984 RPC [13:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=830s) This is a classic problem. In 1984 RPC paper [13:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=832s) paper [13:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=832s) paper it says that if the call returns [13:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=834s) it says that if the call returns [13:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=834s) it says that if the call returns [snorts] the procedure ran precisely [13:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=835s) [snorts] the procedure ran precisely [13:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=836s) [snorts] the procedure ran precisely once. If you get an exception, it ran [13:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=839s) once. If you get an exception, it ran [13:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=839s) once. If you get an exception, it ran once or not at all. And you are not told [14:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=841s) once or not at all. And you are not told [14:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=841s) once or not at all. And you are not told which. [14:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=843s) which. [14:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=843s) which. So you pick a delivery semantic. [14:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=847s) So you pick a delivery semantic. [14:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=847s) So you pick a delivery semantic. At most once [14:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=849s) At most once [14:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=849s) At most once don't retry and accept the fact that it [14:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=852s) don't retry and accept the fact that it [14:12](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=852s) don't retry and accept the fact that it may be lost. At least once. Retry and [14:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=856s) may be lost. At least once. Retry and [14:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=856s) may be lost. At least once. Retry and accept that duplicates are possible. [14:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=860s) accept that duplicates are possible. [14:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=860s) accept that duplicates are possible. Exactly once. [14:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=862s) Exactly once. [14:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=862s) Exactly once. It happened at least once. And there's a [14:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=864s) It happened at least once. And there's a [14:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=864s) It happened at least once. And there's a way we can dduplicate [14:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=868s) way we can dduplicate [14:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=868s) way we can dduplicate maybe by using an item potency key. In [14:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=871s) maybe by using an item potency key. In [14:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=871s) maybe by using an item potency key. In MCP item potent hint is only a hint and [14:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=874s) MCP item potent hint is only a hint and [14:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=874s) MCP item potent hint is only a hint and da or false. If you need it, you build [14:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=876s) da or false. If you need it, you build [14:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=876s) da or false. If you need it, you build it. Here's a small design from my bridge [14:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=880s) it. Here's a small design from my bridge [14:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=880s) it. Here's a small design from my bridge on how I handle it. [14:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=883s) on how I handle it. [14:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=883s) on how I handle it. When a client comes back after restart, [14:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=886s) When a client comes back after restart, [14:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=886s) When a client comes back after restart, the bridge replace the client's record [14:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=887s) the bridge replace the client's record [14:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=887s) the bridge replace the client's record initialize against a fresh server. [14:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=889s) initialize against a fresh server. [14:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=889s) initialize against a fresh server. That's safe. It's just a handshake and [14:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=892s) That's safe. It's just a handshake and [14:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=892s) That's safe. It's just a handshake and it does not restore tool state. [14:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=895s) it does not restore tool state. [14:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=896s) it does not restore tool state. What it never replace is an uncertain [14:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=898s) What it never replace is an uncertain [14:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=898s) What it never replace is an uncertain tool call. When a shared server is [15:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=900s) tool call. When a shared server is [15:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=900s) tool call. When a shared server is recycled, the bridge drains it and calls [15:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=903s) recycled, the bridge drains it and calls [15:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=903s) recycled, the bridge drains it and calls till unanswered at the deadline fail as [15:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=906s) till unanswered at the deadline fail as [15:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=906s) till unanswered at the deadline fail as potentially excluded. [15:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=908s) potentially excluded. [15:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=908s) potentially excluded. That status is honest. It tells the [15:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=911s) That status is honest. It tells the [15:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=911s) That status is honest. It tells the caller to go and check. [15:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=915s) caller to go and check. [15:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=915s) caller to go and check. What helped is giving every side effect [15:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=917s) What helped is giving every side effect [15:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=917s) What helped is giving every side effect an identity. Approval binds to that [15:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=921s) an identity. Approval binds to that [15:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=921s) an identity. Approval binds to that exact action. It's only used once. [15:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=924s) exact action. It's only used once. [15:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=924s) exact action. It's only used once. After the call, read back by that [15:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=927s) After the call, read back by that [15:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=927s) After the call, read back by that identity. [15:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=928s) identity. [15:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=928s) identity. Submitted is not confirmed. Confirmed [15:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=930s) Submitted is not confirmed. Confirmed [15:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=930s) Submitted is not confirmed. Confirmed means you observe the effect. [15:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=933s) means you observe the effect. [15:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=933s) means you observe the effect. The last part of a it shows up when work [15:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=937s) The last part of a it shows up when work [15:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=937s) The last part of a it shows up when work passes and resumes. A lease is a lock [15:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=940s) passes and resumes. A lease is a lock [15:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=940s) passes and resumes. A lease is a lock with a timeout. [15:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=942s) with a timeout. [15:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=942s) with a timeout. Martin Clipman's post about distributed [15:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=944s) Martin Clipman's post about distributed [15:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=944s) Martin Clipman's post about distributed locking shows this problem. A client [15:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=947s) locking shows this problem. A client [15:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=947s) locking shows this problem. A client passes while holding it. The lease [15:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=950s) passes while holding it. The lease [15:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=950s) passes while holding it. The lease expense. Someone else takes over and the [15:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=953s) expense. Someone else takes over and the [15:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=953s) expense. Someone else takes over and the first client wakes up and writes anyway. [15:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=956s) first client wakes up and writes anyway. [15:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=956s) first client wakes up and writes anyway. The fix he proposed is a fencing token. [16:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=960s) The fix he proposed is a fencing token. [16:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=960s) The fix he proposed is a fencing token. A number that goes up every time the [16:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=962s) A number that goes up every time the [16:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=962s) A number that goes up every time the lock is granted. [16:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=965s) lock is granted. [16:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=965s) lock is granted. The storage remembers the highest token [16:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=967s) The storage remembers the highest token [16:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=967s) The storage remembers the highest token it has seen and rejects any right with [16:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=970s) it has seen and rejects any right with [16:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=970s) it has seen and rejects any right with the old one. [16:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=973s) the old one. [16:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=973s) the old one. Sub agents pause and resume all the [16:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=975s) Sub agents pause and resume all the [16:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=975s) Sub agents pause and resume all the time. In my other talk, as I put it, [16:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=980s) time. In my other talk, as I put it, [16:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=980s) time. In my other talk, as I put it, restore context, not side effects. So [16:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=983s) restore context, not side effects. So [16:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=983s) restore context, not side effects. So treat an approval like a fencing token. [16:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=985s) treat an approval like a fencing token. [16:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=985s) treat an approval like a fencing token. One action, one use. Checked at the [16:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=988s) One action, one use. Checked at the [16:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=988s) One action, one use. Checked at the gate, not remembered in the prompt or [16:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=990s) gate, not remembered in the prompt or [16:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=990s) gate, not remembered in the prompt or conversation. [16:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=993s) conversation. [16:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=993s) conversation. My bridge fixed the same class of bug. A [16:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=996s) My bridge fixed the same class of bug. A [16:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=996s) My bridge fixed the same class of bug. A stale exit or a deleted resume can no [16:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=999s) stale exit or a deleted resume can no [16:39](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=999s) stale exit or a deleted resume can no longer remove or bring back a [16:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1001s) longer remove or bring back a [16:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1001s) longer remove or bring back a replacement session. [16:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1003s) replacement session. [16:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1003s) replacement session. Fan out budgets is about unbounded fan [16:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1007s) Fan out budgets is about unbounded fan [16:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1007s) Fan out budgets is about unbounded fan out. This is flow control or the lack of [16:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1010s) out. This is flow control or the lack of [16:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1010s) out. This is flow control or the lack of it. I hit this issue in my own MCP [16:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1013s) it. I hit this issue in my own MCP [16:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1013s) it. I hit this issue in my own MCP bridge before it had a limit on cold [16:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1015s) bridge before it had a limit on cold [16:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1015s) bridge before it had a limit on cold starts. [16:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1017s) starts. [16:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1017s) starts. Many sessions start their service at the [17:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1020s) Many sessions start their service at the [17:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1020s) Many sessions start their service at the same time through npx. [17:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1023s) same time through npx. [17:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1023s) same time through npx. They share one package cache. Every one [17:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1026s) They share one package cache. Every one [17:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1026s) They share one package cache. Every one of them failed with npm's lock [17:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1028s) of them failed with npm's lock [17:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1028s) of them failed with npm's lock compromised error. Nothing in any single [17:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1031s) compromised error. Nothing in any single [17:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1031s) compromised error. Nothing in any single request was wrong. The failures was the [17:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1033s) request was wrong. The failures was the [17:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1033s) request was wrong. The failures was the concurrency itself. [17:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1035s) concurrency itself. [17:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1035s) concurrency itself. And when a downstream does push back, it [17:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1038s) And when a downstream does push back, it [17:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1038s) And when a downstream does push back, it can arrive as a plain text. This is MCP [17:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1041s) can arrive as a plain text. This is MCP [17:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1041s) can arrive as a plain text. This is MCP specs one example of a tool execution [17:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1043s) specs one example of a tool execution [17:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1043s) specs one example of a tool execution error. [17:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1044s) error. [17:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1044s) error. limit as one sentence. The model reads a [17:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1048s) limit as one sentence. The model reads a [17:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1048s) limit as one sentence. The model reads a sentence. The orchestrator code may [17:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1050s) sentence. The orchestrator code may [17:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1050s) sentence. The orchestrator code may never look and the next worker starts [17:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1052s) never look and the next worker starts [17:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1052s) never look and the next worker starts anyway. [17:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1054s) anyway. [17:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1054s) anyway. So I introduce a kota budget in my [17:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1057s) So I introduce a kota budget in my [17:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1057s) So I introduce a kota budget in my bridge. [17:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1058s) bridge. [17:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1058s) bridge. Coal starts that use a package manager [17:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1062s) Coal starts that use a package manager [17:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1062s) Coal starts that use a package manager go through a gate two at a time by [17:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1064s) go through a gate two at a time by [17:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1064s) go through a gate two at a time by default. Everyone else waits and the [17:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1067s) default. Everyone else waits and the [17:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1067s) default. Everyone else waits and the weight is born at 15 seconds. Past that [17:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1071s) weight is born at 15 seconds. Past that [17:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1071s) weight is born at 15 seconds. Past that the start goes ahead without a gate. [17:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1074s) the start goes ahead without a gate. [17:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1074s) the start goes ahead without a gate. And I documented why a corrupted cache [17:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1077s) And I documented why a corrupted cache [17:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1078s) And I documented why a corrupted cache is a risk while a hung client request is [18:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1080s) is a risk while a hung client request is [18:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1080s) is a risk while a hung client request is certainity. There is a choice for [18:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1083s) certainity. There is a choice for [18:03](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1083s) certainity. There is a choice for availability made on purpose and written [18:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1085s) availability made on purpose and written [18:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1085s) availability made on purpose and written down. [18:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1087s) down. [18:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1087s) down. And I also built a cap on sessions. [18:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1090s) And I also built a cap on sessions. [18:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1090s) And I also built a cap on sessions. At the cap, a new a new session waits up [18:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1093s) At the cap, a new a new session waits up [18:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1093s) At the cap, a new a new session waits up to 10 seconds and then it's refused. [18:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1097s) to 10 seconds and then it's refused. [18:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1097s) to 10 seconds and then it's refused. That refusal now counts as a failure in [18:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1099s) That refusal now counts as a failure in [18:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1099s) That refusal now counts as a failure in health. Another book capacity has to be [18:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1102s) health. Another book capacity has to be [18:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1102s) health. Another book capacity has to be reserved before the synchronous start [18:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1104s) reserved before the synchronous start [18:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1104s) reserved before the synchronous start not after otherwise [18:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1107s) not after otherwise [18:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1107s) not after otherwise cued starts or book the cap that is a [18:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1110s) cued starts or book the cap that is a [18:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1110s) cued starts or book the cap that is a check then act race for sub agents give [18:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1114s) check then act race for sub agents give [18:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1114s) check then act race for sub agents give the orchestrator a number per downstream [18:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1117s) the orchestrator a number per downstream [18:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1117s) the orchestrator a number per downstream a bounded weight and a refused cancel. [18:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1120s) a bounded weight and a refused cancel. [18:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1120s) a bounded weight and a refused cancel. Now [18:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1122s) Now [18:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1122s) Now e the an explicit outcomes every hand of [18:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1126s) e the an explicit outcomes every hand of [18:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1126s) e the an explicit outcomes every hand of ends in a named state like stale or [18:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1129s) ends in a named state like stale or [18:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1129s) ends in a named state like stale or unknown not just text. The last piece is [18:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1132s) unknown not just text. The last piece is [18:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1132s) unknown not just text. The last piece is who decided that state treats agent [18:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1135s) who decided that state treats agent [18:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1135s) who decided that state treats agent output as untrusted input the same way [18:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1137s) output as untrusted input the same way [18:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1137s) output as untrusted input the same way you treat request body from the [18:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1139s) you treat request body from the [18:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1139s) you treat request body from the internet. [19:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1141s) internet. [19:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1141s) internet. The orchestrator also checks in code. [19:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1144s) The orchestrator also checks in code. [19:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1144s) The orchestrator also checks in code. Does the result match the schema? Is the [19:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1147s) Does the result match the schema? Is the [19:07](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1147s) Does the result match the schema? Is the evidence there? Is it fresh enough? to [19:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1149s) evidence there? Is it fresh enough? to [19:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1149s) evidence there? Is it fresh enough? to the numbers hold when you recmp compute [19:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1151s) the numbers hold when you recmp compute [19:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1151s) the numbers hold when you recmp compute them. If a check fails, the step returns [19:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1155s) them. If a check fails, the step returns [19:15](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1155s) them. If a check fails, the step returns a refusal with a reason. It does not [19:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1158s) a refusal with a reason. It does not [19:18](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1158s) a refusal with a reason. It does not pass the text along with a warning [19:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1159s) pass the text along with a warning [19:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1159s) pass the text along with a warning paragraph underneath. If the same model [19:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1162s) paragraph underneath. If the same model [19:22](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1162s) paragraph underneath. If the same model can skip a check and declare it past, [19:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1165s) can skip a check and declare it past, [19:25](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1165s) can skip a check and declare it past, you have a request, not a boundary. [19:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1168s) you have a request, not a boundary. [19:28](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1168s) you have a request, not a boundary. Here's a madeup example from another [19:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1170s) Here's a madeup example from another [19:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1170s) Here's a madeup example from another talk. A sub agent says that a rate fell. [19:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1174s) talk. A sub agent says that a rate fell. [19:34](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1174s) talk. A sub agent says that a rate fell. Before it was 10 errors in,000 requests. [19:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1178s) Before it was 10 errors in,000 requests. [19:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1178s) Before it was 10 errors in,000 requests. after it was six errors with 600 [19:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1180s) after it was six errors with 600 [19:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1180s) after it was six errors with 600 requests. It went from 10 to six and it [19:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1183s) requests. It went from 10 to six and it [19:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1183s) requests. It went from 10 to six and it claimed that our rate fell. But when we [19:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1186s) claimed that our rate fell. But when we [19:46](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1186s) claimed that our rate fell. But when we computed the rate, it's both 1%. [19:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1190s) computed the rate, it's both 1%. [19:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1190s) computed the rate, it's both 1%. We recomputed it and then the claim is [19:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1192s) We recomputed it and then the claim is [19:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1192s) We recomputed it and then the claim is refused. [19:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1194s) refused. [19:54](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1194s) refused. A supported answer is fine. A deliberate [19:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1196s) A supported answer is fine. A deliberate [19:56](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1196s) A supported answer is fine. A deliberate handoff to a person is fine. A confident [19:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1199s) handoff to a person is fine. A confident [19:59](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1199s) handoff to a person is fine. A confident guess it's not. [20:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1201s) guess it's not. [20:01](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1201s) guess it's not. So limitations because a consistency [20:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1204s) So limitations because a consistency [20:04](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1204s) So limitations because a consistency model is only useful when you know where [20:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1206s) model is only useful when you know where [20:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1206s) model is only useful when you know where we should stop. [20:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1208s) we should stop. [20:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1208s) we should stop. So here's what safe model does not do. [20:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1213s) So here's what safe model does not do. [20:14](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1214s) So here's what safe model does not do. It does not make facts true. A source [20:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1217s) It does not make facts true. A source [20:17](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1217s) It does not make facts true. A source fact is checkable, not correct. It does [20:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1220s) fact is checkable, not correct. It does [20:20](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1220s) fact is checkable, not correct. It does not give you exactly once delivery. [20:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1223s) not give you exactly once delivery. [20:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1223s) not give you exactly once delivery. A lost reply stays unknown until you [20:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1226s) A lost reply stays unknown until you [20:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1226s) A lost reply stays unknown until you read back. It does not roll anything [20:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1229s) read back. It does not roll anything [20:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1229s) read back. It does not roll anything back. [20:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1230s) back. [20:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1230s) back. Undoing a partial task is a compensation [20:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1233s) Undoing a partial task is a compensation [20:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1233s) Undoing a partial task is a compensation step you design. like the Saga paper [20:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1236s) step you design. like the Saga paper [20:36](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1236s) step you design. like the Saga paper from 1987. [20:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1237s) from 1987. [20:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1238s) from 1987. And finally, there's no global order [20:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1240s) And finally, there's no global order [20:40](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1240s) And finally, there's no global order across workers, only causal order per [20:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1243s) across workers, only causal order per [20:43](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1243s) across workers, only causal order per fact. [20:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1245s) fact. [20:45](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1245s) fact. And it has a price. [20:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1247s) And it has a price. [20:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1247s) And it has a price. CAP theorem says that when the network [20:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1250s) CAP theorem says that when the network [20:50](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1250s) CAP theorem says that when the network splits, you trade off availability [20:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1253s) splits, you trade off availability [20:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1253s) splits, you trade off availability against consistency. [20:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1255s) against consistency. [20:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1255s) against consistency. My bridge chose availability for cold [20:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1257s) My bridge chose availability for cold [20:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1257s) My bridge chose availability for cold starts on purpose. For rights, safe [21:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1262s) starts on purpose. For rights, safe [21:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1262s) starts on purpose. For rights, safe picks consistency over availability. It [21:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1266s) picks consistency over availability. It [21:06](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1266s) picks consistency over availability. It refuses rather than guesses. [21:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1269s) refuses rather than guesses. [21:09](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1269s) refuses rather than guesses. Packlac [21:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1270s) Packlac [21:10](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1270s) Packlac adds that even without failures, you [21:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1273s) adds that even without failures, you [21:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1273s) adds that even without failures, you trade latency for consistency. [21:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1276s) trade latency for consistency. [21:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1276s) trade latency for consistency. Every freshness check is another read [21:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1279s) Every freshness check is another read [21:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1279s) Every freshness check is another read and and every refusal needs an owner. [21:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1283s) and and every refusal needs an owner. [21:23](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1283s) and and every refusal needs an owner. So where does not fit? [21:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1286s) So where does not fit? [21:26](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1286s) So where does not fit? If a sub agent only drafts text that a [21:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1289s) If a sub agent only drafts text that a [21:29](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1289s) If a sub agent only drafts text that a person reads before anything happens, [21:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1292s) person reads before anything happens, [21:32](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1292s) person reads before anything happens, most of this is is a overhead. [21:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1295s) most of this is is a overhead. [21:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1295s) most of this is is a overhead. It also assumes that there's one single [21:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1297s) It also assumes that there's one single [21:37](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1297s) It also assumes that there's one single orchestrator. [21:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1298s) orchestrator. [21:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1298s) orchestrator. If you have more, this does not fit. [21:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1302s) If you have more, this does not fit. [21:42](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1302s) If you have more, this does not fit. Pure agents with no single place to [21:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1304s) Pure agents with no single place to [21:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1304s) Pure agents with no single place to stick to need something else. [21:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1307s) stick to need something else. [21:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1307s) stick to need something else. This is the consistency model I use for [21:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1309s) This is the consistency model I use for [21:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1309s) This is the consistency model I use for my own agents that touch actual real [21:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1312s) my own agents that touch actual real [21:52](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1312s) my own agents that touch actual real production systems. It's not universal. [21:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1315s) production systems. It's not universal. [21:55](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1315s) production systems. It's not universal. So wrapping up, here's three key [21:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1318s) So wrapping up, here's three key [21:58](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1318s) So wrapping up, here's three key takeaways. [22:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1320s) takeaways. [22:00](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1320s) takeaways. Treat every handoff to a sub agent as a [22:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1322s) Treat every handoff to a sub agent as a [22:02](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1322s) Treat every handoff to a sub agent as a network call. It can fail halfway. It [22:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1325s) network call. It can fail halfway. It [22:05](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1325s) network call. It can fail halfway. It can be slow and it can come back twice. [22:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1328s) can be slow and it can come back twice. [22:08](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1328s) can be slow and it can come back twice. Written a status, not just text. If the [22:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1331s) Written a status, not just text. If the [22:11](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1331s) Written a status, not just text. If the result cannot say partial, stale or [22:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1333s) result cannot say partial, stale or [22:13](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1333s) result cannot say partial, stale or unknown, the orchestrator will read it [22:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1336s) unknown, the orchestrator will read it [22:16](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1336s) unknown, the orchestrator will read it as done and [22:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1339s) as done and [22:19](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1339s) as done and start with our safe consistency model [22:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1341s) start with our safe consistency model [22:21](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1341s) start with our safe consistency model and enforce it in code. Keep what fits [22:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1344s) and enforce it in code. Keep what fits [22:24](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1344s) and enforce it in code. Keep what fits and change what does not. We already [22:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1347s) and change what does not. We already [22:27](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1347s) and change what does not. We already know how to build distribute systems. We [22:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1350s) know how to build distribute systems. We [22:30](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1350s) know how to build distribute systems. We just have to notice that we are building [22:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1351s) just have to notice that we are building [22:31](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1351s) just have to notice that we are building one. [22:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1353s) one. [22:33](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1353s) one. So these are some of the sources behind [22:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1355s) So these are some of the sources behind [22:35](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1355s) So these are some of the sources behind this talk. [22:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1358s) this talk. [22:38](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1358s) this talk. On the left side it's my own work, my [22:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1361s) On the left side it's my own work, my [22:41](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1361s) On the left side it's my own work, my open source projects, my articles, [22:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1364s) open source projects, my articles, [22:44](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1364s) open source projects, my articles, internet drafts and my talks. On the [22:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1367s) internet drafts and my talks. On the [22:47](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1367s) internet drafts and my talks. On the right side [22:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1369s) right side [22:49](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1369s) right side specs and standards [22:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1371s) specs and standards [22:51](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1371s) specs and standards and [22:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1373s) and [22:53](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1373s) and bunch of papers behind each concept. [22:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1377s) bunch of papers behind each concept. [22:57](https://www.youtube.com/watch?v=DxP5aW4VknQ&t=1377s) bunch of papers behind each concept. Thank you.