Your Sub-Agents Are a Distributed System. Treat Them Like One. - Girish Konda, Microsoft

An orchestrator splits a task among three sub-agents. They run concurrently, read shared state from memory files, repository working trees, issues, or vector stores, and call tools that reach databases, APIs, CI systems, and deployment pipelines. The orchestrator merges their replies into one answer. To its caller, the whole operation can look like a function that takes a string and returns a string.

Girish Konda, a principal software engineer at Microsoft and the creator of MCP Pacemaker, encountered the consequences in engineering work and in his open-source bridge for keeping MCP servers running. Half-finished work can return fluent prose. One worker can read another’s guess. A service can signal overload while the orchestrator keeps starting workers. If a reply goes missing, retrying the worker can repeat every write it made on the first attempt. The string-returning interface leaves all four decisions to the caller.

In a crash-stop failure, a process that was running correctly stops and stays stopped. A worker that throws an error gives the caller an explicit failure to handle. Silence is harder to interpret. A fluent but wrong reply is harder still. Konda draws an analogy to a Byzantine failure from the caller’s point of view. An entirely well-intentioned worker can report success after an incomplete investigation, using the same return type it would use for a completed one.

How much investigation went into a benign alert verdict? In Konda’s illustration, three paths produce the same classification at the same confidence level.

• Telemetry comparison. The worker retrieved telemetry and performed a comparison before classifying the alert.
• Familiar title. The worker recognized an alert title it had seen before and returned the associated answer.
• Unsupported verdict. The worker performed no investigation and generated the answer anyway.

The next decision might depend on whether anyone inspected this alert’s telemetry. Even retaining the confidence alongside the label gives the orchestrator no way to recover that information.

When a deadline expires, the orchestrator knows how long it waited. The worker might still be running, it might have crashed, or it might have finished and lost the reply. The 1985 FLP result concerns the limits of reaching agreement when a process may fail and messages may be delayed. The practical connection is the difficulty of telling a delayed process from a dead one. A timeout-based failure detector can raise a suspicion and be wrong.

MCP Pacemaker needed two different fixes for calls that appeared to time out.

• Slow first start. A server downloaded itself on its first run and exceeded the ordinary 30-second call timeout. A working installation looked broken. Giving initialize its own timeout let startup have a different deadline from routine calls.
• Crash during a call. A server exited while a call was outstanding. When the call later timed out, the bridge overwrote the exit code that explained the crash. Pacemaker now records the timeout only if the server is still alive. When the process has exited, the caller keeps the more informative cause.

A handoff can represent uncertainty directly. The worker returns a status, the work it completed, the work still outstanding, its evidence, and any side effects. partial is an ordinary result. An unresolved timeout yields unknown, with any underlying diagnostic preserved. A2A already lets a task complete, fail, be canceled or rejected, or pause for input. An MCP tool result can set isError to true. The orchestrator still has to read those signals. A worker can mark a task complete while its account of the work says otherwise.

Konda’s individual Internet-Draft on evaluation states makes a further distinction. A check that ran and returned false establishes something about the condition. A check that could not run leaves the condition open. Collapsing both into a negative result changes which operation gets retried, who gets paged, and what a later audit says was checked.

Where did the apparent corroboration come from? In a hypothetical incident, a worker writes into shared context that the error rate fell by 80%. It inferred the change. A second worker picks up the sentence for a review note. A third reads the same context and uses the drop to justify proceeding in a runbook. When the orchestrator merges the three artifacts, the percentage appears in an investigation, a review, and an operational decision. Tracing either downstream branch leads straight back to the first worker.

The database analogy is a dirty read. A reader sees a value before the writer commits or rolls back. In shared agent context, a sentence becomes readable the moment a worker writes it. Without a marker for unchecked material, another worker can act on a guess before anyone has established where it came from.

Would asking three agents for a quorum help? In the 1979 weighted-voting scheme Konda invokes, the read quorum weight plus the write quorum weight exceeds the total voting weight. That inequality forces every read quorum to overlap the latest write quorum. Counting agreeing agent answers supplies no comparable guarantee. Here, all three answers inherit one unsupported sentence.

A consistency model says what a reader may rely on when operations overlap. Linearizability makes each operation appear to occur at one point in real-time order. Sequential consistency gives everyone one total order that respects each process’s own order. Causal consistency preserves the order of operations that depend on each other. Session guarantees can let a client read its own writes and avoid moving backward on successive reads. Eventual consistency promises convergence after updates stop. Giving workers a shared context leaves all of these questions open.

Requiring a single real-time order has an availability cost when the network splits. Causal and session guarantees leave room for a client to keep working if it stays attached to one place. In Konda’s design, that place is the orchestrator. His SAFE handoffs draw on Pacemaker, his other engineering work, and his individual draft.

• S — Source reads. A worker passes on the source, version, and time behind a fact. The decision that uses it checks whether it is still fresh.
• A — At-most-once effects. Each write has an identity and a single-use approval for that action. If its outcome is uncertain, the caller reads it back.
• F — Fan-out budgets. Each downstream dependency gets a limit and a bounded wait. When the system refuses work, the refusal counts as a failure.
• E — Explicit outcomes. Each handoff ends in a named state. A timeout can leave the outcome unknown, and a known cause survives for diagnosis.

A worker handing on a fact supplies a record of the observation. The record names the claim and its revision, the source and its version, and the scope over which the claim applies. It also distinguishes three times that a single timestamp would conflate. When was the source data current? When did the worker read it? When did a decision use it?

How long can the next worker rely on that read? In Konda’s example, the data covers events through 9:05. A worker reads it at 9:06, and a decision uses it at 9:07. The data is two minutes old at the decision, within the chosen five-minute limit. At 9:15 the same data is ten minutes old. The orchestrator rejects the stale record and requires another read before the next decision.

Copying the claim into a new note does nothing to advance the time through which the data was current. The same record requirement exposes the problem with the earlier 80% drop. A worker that observed the drop could identify the source and when it read it. The worker that guessed has no such observation to hand on.

Suppose a sub-agent calls a tool to create a pull request. The tool makes the write, but the reply is lost. From the orchestrator’s point of view the outcome is unknown. If it delegates the whole task again, the worker reaches the creation step a second time. The first pull request can already exist while the orchestrator is still waiting to learn whether anything happened.

The 1984 remote procedure call example makes the ambiguity precise. Under its semantics, a normal return means the procedure ran once. An exception leaves two possibilities, zero executions or one. Forgoing a retry preserves at-most-once execution at the cost of possibly losing the operation. Retrying for at-least-once execution admits duplicates. Getting exactly one observable effect requires both eventual execution and a way for the receiver to recognize duplicate attempts, for example with an idempotency key. MCP’s idempotency hint describes a tool property. The tool implementation has to supply the behavior.

Pacemaker distinguishes two operations during recovery. When a client reconnects after a restart, the bridge can replay the recorded initialize handshake against a fresh server. That reconnects the client without restoring tool state. When recycling a shared server, the bridge gives outstanding calls time to drain. If a tool call is still unanswered at the deadline, Pacemaker reports that it may already have run and leaves the caller to check. Replaying that call could duplicate a write.

SAFE gives each side effect an identity and binds a single-use approval to that particular action. After submission, the caller uses the identity to read back the result. For the pull request, confirmation means observing that the intended pull request exists. Remembering that the worker sent the request establishes only submission.

Resuming work creates a related problem. Consider a client holding a lease, a lock with an expiry time. The client pauses, the lease expires, and another client takes over. The first client wakes up and tries to write with its old credentials. A fencing token lets storage reject that late write. Each lock grant gets a higher number. Storage remembers the highest token it has seen and rejects writes carrying an older one.

For a resumed agent, the corresponding check belongs at the point that permits the write. Restoring a conversation can recover the plan. The system still needs to check whether the approval covers this action and whether it has already been used. Pacemaker fixed a related session bug in which an old exit or resume event could interfere with a replacement session.

Before Pacemaker limited cold starts, many sessions launched servers through npx at the same time. They all used one package cache. Every start in that burst failed with npm’s lock-compromised error, even though each individual request was valid. To schedule the next start, the bridge needed to know how many others were already using the cache.

A downstream service can also report that it is overloaded. It might return HTTP 429 or 503. In the MCP tool-error example, the model receives a rate limit as a sentence. If the code that starts workers never inspects that result, the model can read the explanation while the orchestrator sends the next worker straight into the same limit.

Pacemaker uses two different admission policies, each with an explicit decision at the deadline.

• Package-manager cold starts. By default, two starts at a time pass through the gate. Other starts wait up to 15 seconds. Once that wait expires, a start proceeds without the gate. Konda accepted the risk of cache corruption so a stuck gate could not hang a client indefinitely.
• Session capacity. At the session cap, a new session waits up to 10 seconds for room. If capacity is still unavailable, the bridge refuses the session and counts the refusal as a failure in its health reporting.

Enforcing the session cap requires reserving capacity before starting a session. If queued starts each check that space exists and record their use only after startup, several can claim the same remaining slot. A limit of the right size can still fail through this check-then-act race. Reserving the slot makes the next admission decision account for work already committed to start.

A worker can propose a status as readily as it can propose a conclusion. To make states such as stale and unknown meaningful, the orchestrator treats the result like a request body received over the internet. Code checks the schema, looks for the evidence, checks its freshness, and recomputes numbers where possible. If a check fails, the step returns a refusal with a reason. That keeps the failed claim out of the next worker’s inputs, where a warning attached underneath it would be easy to lose.

Did the error rate fall when the error count went from ten to six? In Konda’s worked example, the first period had 10 errors per 1,000 requests. The second had 6 errors per 600 requests. Dividing each count by its own request volume gives 10 / 1,000 = 1% and 6 / 600 = 1%. The count fell along with traffic. The rate held steady.

Recomputing the rates gives the orchestrator a specific reason to refuse the claimed improvement. The worker can return the supported finding that both rates were 1%, or hand the decision to a person. Either outcome lets the investigation continue without passing a spurious improvement into the runbook.

Provenance makes a claim checkable. The source itself can still be wrong. At-most-once effects leave room for zero executions, and a lost reply leaves the outcome open until read-back. If a multi-step task changed the world before failing, undoing those changes requires a compensation step designed for that action, as in the 1987 Sagas work. Ordering is similarly scoped. A reader can trace the causal history of a particular fact while independent workers’ operations have no single global order.

CAP describes the trade between consistency and availability during a network partition. Konda makes the practical choice operation by operation. Pacemaker lets a cold start through after a bounded wait because the client needs progress. For a write whose safety the orchestrator cannot establish, SAFE accepts the availability cost of refusal. Even on a healthy network, checking freshness can add another read to the decision path. Someone also has to own the work a refusal leaves outstanding.

The intended setting is one orchestrator delegating work to agents that touch production systems. If a worker only drafts text and a person reads it before anything happens, much of this machinery is overhead. Multiple orchestrators or peer agents need another way to coordinate because they lack the single place SAFE assumes for enforcing decisions.

For one orchestrator, a useful starting question is what its code will do when a worker returns partial, stale, or unknown. Each state calls for a different next step. Inspect what remains unfinished, refresh the evidence, or find out whether the write already happened. Giving the caller those choices lets a handoff end honestly while the task itself is still open.