Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
6:37:54Hey folks, this is Gish, a principal
6:37:57software engineer at Microsoft. Outside
6:37:59my job, I build open source tools for AI
6:38:02agents. One of them is MCP pacemaker, a
6:38:06supervised bridge that keeps MCP servers
6:38:08running.
6:38:10Before we start, a small disclosure. The
6:38:13views in this talk are my own and
6:38:15doesn't represent my employer.
6:38:17So sub agents are a distributed system.
6:38:20Over the past year, I have been using
6:38:22agents and sub agents on real
6:38:24engineering work, dev sessions, code
6:38:27reviews, instant trashing
6:38:30what
6:38:32this talk is about what happens when one
6:38:34agent hands work to another.
6:38:38So here's an architecture.
6:38:41The orchestrator splits a task and hands
6:38:43the pieces to three different sub aents.
6:38:47They all run at the same time.
6:38:50They all read from the same shared
6:38:52state. Consization, memory files, repo,
6:38:56working trees, issues, vector stores.
6:39:01And they don't just think, they make
6:39:04tool calls to real systems
6:39:08which touch databases, APIs, CI tools,
6:39:11deployment pipelines
6:39:14and
6:39:15the results come back and orchestrator
6:39:18merges them into a single answer and
6:39:20orchestrator workers shared mutable
6:39:24state and results merge from concurrent
6:39:26executions. This represents a classic
6:39:29distributed system.
6:39:31I have known how to reason with dispute
6:39:33systems for decades,
6:39:35but none of the setup came with it.
6:39:39What you get is a function that takes a
6:39:41string and returns a string.
6:39:44Failures we hit are not new, are not
6:39:47specific to a agents. They're classic
6:39:50distributed system failures.
6:39:53partial failure. A normal call that
6:39:56fails halfway through and returns an
6:39:59exception error code. In a sub aent
6:40:02case, it returns text that looks like a
6:40:04success.
6:40:06Dirty reads normally we have isolations
6:40:10versions. Here every worker reads one
6:40:13shade context including things other
6:40:16worker only guessed.
6:40:18Unbounded fan out.
6:40:20A busy service pushes back with the cues
6:40:25429's 503s.
6:40:28In agents, orchestrator does not see any
6:40:31of this. It just starts in the worker.
6:40:34Retries.
6:40:36In normal distute systems, we build
6:40:39potency to make retries reliable.
6:40:42in agents a retrate sub agent and the
6:40:45whole task again with side effects
6:40:47included
6:40:49for each of these case
6:40:51I'll name the concept it already is then
6:40:54I'll put them together as a consistency
6:40:56model I'm proposing and walk through why
6:40:59each part is there
6:41:02start with the failure model because
6:41:03everything else depends on it a friendly
6:41:06case is a crash stop a process it was
6:41:09running correctly stops and stays is
6:41:12stopped. The sub agent throws an error
6:41:14in this box. It's easy to handle.
6:41:18Next, no reply. The call is lost or it's
6:41:21very late. We'll come back to this soon.
6:41:25The hot case is the one from Baantine
6:41:28general's paper. A component that gives
6:41:32conflicting information. The process
6:41:34answers and the answer is wrong.
6:41:37A half finished sub agent sits closest
6:41:39to the box. It is not malicious but from
6:41:42the outside it looks the same. It does
6:41:44not crash. It answers fluently. Let's
6:41:47jump into this. A sub agent that half
6:41:50finished does not raise. It returns
6:41:53normal looking text with the same type
6:41:55signature as success.
6:41:58I talked about one version of this in
6:42:00one of my talks. We pointed an agent at
6:42:03an LQ.
6:42:05Here's an illustration I used there with
6:42:08three different paths which gave the
6:42:10same answer.
6:42:11The first one
6:42:13it pulled the telemetry and compared
6:42:15with a known brand set. It actually did
6:42:18the work. The second one it matched
6:42:21against the title. It has seen before
6:42:24and gave the response.
6:42:27The third alert
6:42:29it didn't do anything.
6:42:31It just originated.
6:42:34All three are categorized as benign with
6:42:36the same level of confidence
6:42:39and nothing downstream can dull this
6:42:41apart. The orchestrator can read the
6:42:43conclusion. It cannot read how much work
6:42:45was behind it.
6:42:48Now the call gets no reply.
6:42:51The orchestrator send work and a timeout
6:42:54fired. From here three volts look the
6:42:58same. The sub agent is still working. It
6:43:01crashed and the reply was lost.
6:43:04It's a hall limit, not a tooling gap.
6:43:07The FLP paper from 1985 showed that in a
6:43:10synchronous system with even one faulty
6:43:12process, agreement can fail to finish.
6:43:16Chundra and 2x answer in 1996 was
6:43:18failure detectors that can make
6:43:20mistakes. They suspect did not know. I
6:43:24hit both directions
6:43:26in my own open source project MCP
6:43:30pacemaker supervised bridge.
6:43:33A server whose first run downloads
6:43:35itself hit the 30 secondond call time
6:43:38out and look broken. It was slow not
6:43:41dead. We fixed it by giving initialize
6:43:44its own time out. Now the other way the
6:43:48server crashed during a call. The call
6:43:50then timed out and the timeout overrode
6:43:53the exit code that explained the crash.
6:43:56We fixed it by recording the time out
6:43:59only when it was alive.
6:44:02So
6:44:03time out is a suspicion. Record it as
6:44:06unknown and keep the real cause.
6:44:09And now make failure a value.
6:44:15Let's take this illustration.
6:44:17A sub agent returns a status. What it
6:44:20did, what it did not do, the evidence,
6:44:24and any side effects. Partial is a
6:44:26normal answer here.
6:44:29The protocols already have a slot for
6:44:31this. In A2A, a task can end as
6:44:35completed, failed, canceled or rejected
6:44:39or stop and ask for input. In MCP, a
6:44:42dual result can say is error true. But a
6:44:46sub agent can still say completed while
6:44:50its text says something else. And a loop
6:44:53that only reads the text never sees a
6:44:55difference.
6:44:56Another spirit.
6:44:58Checking a condition and getting no is a
6:45:01different fact from not being able to
6:45:04check at all.
6:45:06My own individual internet draft on
6:45:08evaluation states
6:45:11talks about this.
6:45:13Collapse the two and you change what
6:45:15gets retrieded and who gets ped and what
6:45:18the audit says later.
6:45:21Next dirty reads.
6:45:24Let's take this madeup example from one
6:45:26of my talk.
6:45:28The sub agent the scratch it writes into
6:45:32a shade context
6:45:34saying that error rate fell by 80%. It
6:45:37did not measure that it inferred it.
6:45:41Now
6:45:43two other sub aents read from the same
6:45:45shade context.
6:45:48The second sub agent writes a review
6:45:50note and picks the same sentenc.
6:45:53The third agent writes a runbook step
6:45:56and uses it as a reason to proceed.
6:45:59The orchestrator merges all the all
6:46:02these three three artifacts agree. It is
6:46:05once unsupported sentence read three
6:46:07different times.
6:46:09The 1995 critique of SQL isolation
6:46:12levels define a dirty read as you read a
6:46:15value before the writer commits or rolls
6:46:17back. In a shared context, a guess is
6:46:20readable the moment it is written and
6:46:22nothing marks it as unchecked.
6:46:25A regular fix
6:46:28to is to ask three sub agents and get a
6:46:31quorum.
6:46:33Corums do work for a specific reason. In
6:46:36weighted voting from 1979, the read
6:46:39corum plus right corum is greater than
6:46:42total.
6:46:43So every read overlaps the latest right.
6:46:46The overlap is the guarantee.
6:46:49In sub aent's case, the three sub aents
6:46:51that read the same shared context does
6:46:54not give you that. They are not
6:46:55independent copies. They read the same
6:46:58guess. Their agreement is one source
6:47:01counted three different times. Let's
6:47:03pick a consistency model.
6:47:05In databases, we have vocabulary for
6:47:07this.
6:47:09Linearizable.
6:47:11Every operation appears to happen at one
6:47:13point in real time order. Sequential.
6:47:17One total order that respects each
6:47:19process one order. Causal
6:47:23operations that are causally related
6:47:25appear in the same order everywhere.
6:47:28Session guarantees.
6:47:30There are promises to one client like
6:47:32read your rights and monotonic reads
6:47:36eventual. If no new updates are made,
6:47:39reads eventually return the lost value.
6:47:42A shared context gives you none of
6:47:44these. There's no stated guarantee at
6:47:46all.
6:47:48And you don't need the top consistency
6:47:50models.
6:47:52The strongest models cannot stay
6:47:54available when the network splits.
6:47:57Causal end session guarantees can as
6:48:00long as each clients stick to one place
6:48:03for sub agents that place the
6:48:05orchestrator. So
6:48:08here's a consistency model I use for sub
6:48:11agents. I call it safe handoffs.
6:48:16S stands for source fees.
6:48:20Every fact a worker hands on carries its
6:48:24source version as of time and revision
6:48:28and decisions read only fresh source
6:48:30facts.
6:48:31A at most once effects
6:48:35every right has identity and a single
6:48:37use approval for that exact action. An
6:48:40outcome is read back never replayed.
6:48:44F fan out budgets. Each downstream gets
6:48:49a gate and a bounded weight and refusal
6:48:52counts as a fail.
6:48:54E stands for explicit outcomes.
6:48:57Every handoff ends in a name state and a
6:49:01timeout is unknown with its cost script.
6:49:04This consistency model is the one I use
6:49:07for my own sub agents. It's not a
6:49:09universal model. It's based off my own
6:49:12public work. Some of it is in my FCP
6:49:15pacemaker bridge and some are based on
6:49:18my other docs and my own internet draft.
6:49:24Now let's see what's behind each letter.
6:49:29Yes, source reads
6:49:32it's the answer to that reads where one
6:49:35worker's guess becomes another worker's
6:49:36fact. What helps is to treat every fact
6:49:40as a record not a sentence.
6:49:43Lampot's 1978 paper define when one
6:49:46event happened before another. The facts
6:49:48provenence is that record. What it was
6:49:51read from and when.
6:49:53Let's take this madeup example.
6:49:57the record names, the claim and its
6:50:00revision, the source and its version,
6:50:04the scope and three different times,
6:50:08when the data was current, when we read
6:50:10it and when we used it.
6:50:14Freshness is checked with the fact is
6:50:16used.
6:50:18In this illustration, the data ends at
6:50:229:05. We read it at 9:06 and we decided
6:50:26at 9:07.
6:50:28In this case,
6:50:30we picked a 5 minute limit. So the data
6:50:34decision at 9:07, it's fine.
6:50:37Now at 9:15, the second edition came in
6:50:42because our freshness limit is five.
6:50:44This is a stale record.
6:50:47We reject it
6:50:49and we say that the next station needs
6:50:51another read.
6:50:53This is how you separate an observation
6:50:55from an inference. An observation has a
6:50:58source and a read time. An inference has
6:51:00none.
6:51:02A most effect is about retries.
6:51:06This is an illustration
6:51:09from my article about MCPmaker bridge.
6:51:14The orchestrator delegates a task. The
6:51:18sub agent calls a tool which does a real
6:51:21right.
6:51:22like
6:51:24creating a PR,
6:51:26the reply is lost.
6:51:29The orchestrator knows one thing, the
6:51:31outcome is unknown.
6:51:33So it retries the whole task, the right
6:51:37happens again.
6:51:39This is a classic problem. In 1984 RPC
6:51:43paper,
6:51:45it says that if the call returns, the
6:51:48procedure ran precisely once. If you get
6:51:50an exception, it ran once or not at all.
6:51:54And you are not told which.
6:51:57So you pick a delivery semantic.
6:52:00At most once
6:52:03don't retry and accept the fact that it
6:52:05may be lost. At least once. Retry and
6:52:10accept that duplicates are possible.
6:52:13Exactly once.
6:52:15It happened at least once. And there's a
6:52:18way we can dduplicate
6:52:21maybe by using an item potency key. In
6:52:24MCP item potent hint is only a hint and
6:52:27da or false. If you need it, you build
6:52:30it. Here's a small design from my bridge
6:52:34on how I handle it.
6:52:36When a client comes back after a
6:52:38restart, the bridge replace the client's
6:52:41record initialize against a fresh
6:52:42server. That's safe. It's just a
6:52:44handshake and it does not restore tool
6:52:47state.
6:52:49What it never replace is an uncertain
6:52:51tool call. When a shared server is
6:52:54recycled, the bridge drains it and calls
6:52:56till unanswered at the deadline fail as
6:52:59potentially excluded.
6:53:02That status is honest. It tells the
6:53:04caller to go and check.
6:53:08What helped is giving every side effect
6:53:11an identity. Approval binds to that
6:53:14exact action. It's only used once.
6:53:18After the call, read back by that
6:53:20identity.
6:53:22Submitted is not confirmed. Confirmed
6:53:24means you observe the effect.
6:53:27The last part of a it shows up when work
6:53:30passes and resumes. A lease is a lock
6:53:33with a timeout.
6:53:35Martin Clipman's post about distributed
6:53:37locking shows this problem.
6:53:40A client passes while holding it. The
6:53:43lease expense. Someone else takes out
6:53:46and the first client wakes up and writes
6:53:48anyway.
6:53:50The fix he proposes a fencing token. A
6:53:54number that goes up every time the lock
6:53:56is granted.
6:53:58The storage remembers the highest token
6:54:01it has seen and rejects any right with
6:54:04the old one.
6:54:06Sub agents pause and resume all the
6:54:09time. In my other talk, as I put it,
6:54:13restore context, not side effects. So
6:54:16treat an approval like a fencing token.
6:54:19One action, one use. Checked at the
6:54:21gate, not remembered in the prompt. A
6:54:24conversation.
6:54:26My bridge fixed the same class of bug. A
6:54:30stale exit or a deleted resume can no
6:54:33longer remove or bring back a
6:54:34replacement session.
6:54:37Fan out budgets is about unbounded fan
6:54:40out. This is flow control or the lack of
6:54:43it. I hit this issue in my own MCV
6:54:46bridge before it had a limit on cold
6:54:48starts.
6:54:50Many sessions start their service at the
6:54:54same time through npx.
6:54:56They share one package cache. Every one
6:55:00of them failed with npm's lock
6:55:01compromis.
6:55:03Nothing in any single request was wrong.
6:55:06The failures was the concurrency itself.
6:55:09And when a downstream does push back, it
6:55:11can arrive as a plain text. This is MCP
6:55:15specs one example of a tool execution
6:55:17error.
6:55:18limit as one sentence. The model reads a
6:55:21sentence. The orchestrator code may
6:55:23never look the next worker starts
6:55:25anyway.
6:55:27So I introduce a kota budget in my
6:55:30bridge.
6:55:32Cole starts that use a package manager
6:55:35go through a gate two at a time by
6:55:38default. Everyone else waits and the
6:55:41weight is born at 15 seconds. Pass that
6:55:45the start goes ahead without a gate.
6:55:48And I documented why a corrupted cache
6:55:51is a risk while a hung client request is
6:55:54certainity. There is a choice for
6:55:56availability made on purpose and written
6:55:58down.
6:56:00And I also built a cap on sessions.
6:56:04At the cap, a new a new session waits up
6:56:07to 10 seconds and then it's refused.
6:56:10That refusal now counts as a failure in
6:56:12health. Another book capacity has to be
6:56:16reserved before the synchronous start
6:56:18not after otherwise
6:56:21cued starts or book the cap that is a
6:56:24check then act race for sub agents give
6:56:28the orchestrator a number per downstream
6:56:30a bounded weight and a refusate consent.
6:56:34Now
6:56:35e the explicit outcomes every hand of
6:56:40ends in a name state like stale or
6:56:42unknown not just text. The last piece is
6:56:46who decided that state treats the agent
6:56:48output as untrusted input the same way
6:56:51you treat a request body from the
6:56:52internet.
6:56:54The orchestrator owns the checks in
6:56:57code. Does the result match the schema?
6:57:00Is the evidence there? Is it fresh
6:57:02enough? the numbers hold when you recmp
6:57:05compute them. If a check fails, the step
6:57:08returns a refusal with a reason. It does
6:57:11not pass the text along with a warning
6:57:13paragraph underneath. If the same model
6:57:16can skip a check and declare it past,
6:57:18you have a request, not a boundary.
6:57:21Here's a madeup example from another
6:57:24talk. A sub agent says that a rate fell.
6:57:28Before it was 10 errors in,000 requests.
6:57:31after it was six errors with 600
6:57:34requests. It went from 10 to six and it
6:57:37claimed that our rate fell. But when we
6:57:40computed the rate, it's both 1%.
6:57:44We recomputed it and then the claim is
6:57:46refused.
6:57:48A supported answer is fine. A deliberate
6:57:50handoff to a person is fine. A confident
6:57:53guess it's not.
6:57:55So limitations because a consistency
6:57:57model is only useful when you know where
6:57:59we should start.
6:58:02So here's what safe model does not do.
6:58:07It does not make facts true. A source
6:58:10fact is checkable, not correct. It does
6:58:13not give you exactly once delivery.
6:58:17A lost reply stays unknown until you
6:58:20read it back. It does not roll anything
6:58:22back.
6:58:24Undoing a partial task is a compensation
6:58:26step you design. like the Saga's paper
6:58:29from 1987.
6:58:31And finally, there's no global order
6:58:33across workers, only causal order for a
6:58:36fact.
6:58:38And it has a price. CAP theorem says
6:58:42that when the network splits, you trade
6:58:45off availability against consistency.
6:58:49My bridge chose availability for cold
6:58:51starts on purpose. For rights, safe
6:58:55picks consistency over availability. It
6:58:59refuses rather than guesses.
6:59:02Packlac
6:59:04adds that even without failures, you
6:59:06trade latency for consistency.
6:59:09Every freshness check is another read
6:59:12and and every refusal needs an order.
6:59:16So where does not fit?
6:59:20If a sub agent only drafts text that a
6:59:23person reads before anything happens,
6:59:25most of this is a overhead.
6:59:28It also assumes that there's one single
6:59:30orchestrator.
6:59:32If you have more, this does not fit.
6:59:35Pure agents with no single place to
6:59:38stick to need something else.
6:59:41This is the consistency model I use for
6:59:43my own agents that touch actual real
6:59:45production systems. It's not universal.
6:59:49So wrapping up, here's three key
6:59:51takeaways.
6:59:53Treat every handoff to a sub agent as a
6:59:56network call. It can fail halfway. It
6:59:59can be slow and it can come back twice.
7:00:02Return a status, not just text. If the
7:00:05result cannot say partial, stale or
7:00:07unknown, the orchestrator will read it
7:00:09as
7:00:12start with our safe consistency model
7:00:15and enforce it in code. Keep what fits
7:00:18and change what does not. We already
7:00:20know how to build distribute systems. We
7:00:23just have to notice that we are building
7:00:25one.
7:00:26So these are some of the sources behind
7:00:29this talk.
7:00:32On the left side, it's my own work, my
7:00:34open source projects, my articles,
7:00:38internet drafts and my talks. On the
7:00:40right side,
7:00:42specs and standards
7:00:45and
7:00:46bunch of papers behind each concept.
7:00:50Thank you.
Original stream, 6:48:20. The slide specifies source, version, as-of time, and revision, and explicitly labels this as Konda's model rather than a universal one.