From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik

One agent works well: an LLM, prompts, perhaps a retrieval pipeline and some tool calls. The demo succeeds. Then product asks for five more agents. Now A produces data B needs, C waits for both A and B, D changes the state B is reading, and E crashes midway through the workflow. Adding agents introduces a distributed system, with dependencies and failure modes that model quality alone cannot resolve.

Sandipan Bhaumik describes a financial-services deployment that began with a credit-score agent running for two weeks without issues. The team then added income verification, risk assessment, fraud detection, and final approval. Bhaumik reports that within three days of deploying all five agents, 20% of decisions had incorrect risk ratings. Customers who should have been flagged were receiving approvals, and diagnosis took two days.

The credit-score agent wrote 750 to PostgreSQL; 500 milliseconds later, the risk agent read 680 for the same customer from the cache. The database write had succeeded. The missing operation was cache invalidation, so the risk agent made its decision from an older customer record.

The defect sat between the agents and the database: several agents shared a cache without coordinated invalidation. Fixing the prompt would not have repaired that read path. The investigation delayed delivery because the team had built a distributed system without explicitly designing its consistency behavior.

The number of relationships explains why the transition feels abrupt. For a fully connected set of agents, the undirected connection count is:

That gives zero connections for one agent, one for two agents, and ten for five agents. Bhaumik describes the increase as exponential and invokes a 25-fold complexity increase, but these connection counts establish quadratic growth, not a measured complexity multiplier. Each potential relationship still introduces another place for failures, races, and state synchronization problems. The response starts with choosing a coordination pattern, then specifying state and recovery behavior.

Choreography coordinates agents through events. A research agent finishes and publishes a research-completed event to a message bus. An analysis agent subscribes to that event type, processes the research, and publishes analysis-ready. A report agent consumes that event and generates the report. No central component calls each participant in sequence; each agent reacts to the events it understands.

That loose coupling makes it easy to add another subscriber without changing a central workflow. It also moves the difficulty into operations. When a report never appears, did research fail to publish, did analysis fail to consume, or was an event consumed twice? Choreography requires end-to-end event tracing and deliberate delivery guarantees. It fits naturally event-driven workflows with independent agents and frequent additions, provided the team can reconstruct event propagation when something fails. Choosing it simply because autonomy feels more agentic leaves that operational work undone.

Orchestration puts those decisions in a coordinator. It calls A and waits for the result, starts B and C in parallel, waits for both, then calls D with their combined outputs. The agents do not call one another. They accept inputs, perform work, and return results; the orchestrator owns the execution graph, state, retries, and step logs. Bhaumik names LangGraph wired into Databricks AI Agent Framework as one implementation, while emphasizing that a workflow engine with suitable dependency graphs and retries can fill the same role.

Central control is useful when dependencies are complex, partial work needs compensation, and operators need one view of the workflow. It is especially attractive when the graph changes less often than the agents' internal implementations. Bhaumik reports using orchestration almost exclusively in financial-services work because a failed credit decision must be reconstructed: which agent acted, in which order, and with which data. Choreography can support such reconstruction through strong tracing, but orchestration makes that execution record a direct responsibility of the coordinator.

The decision framework uses workflow complexity and required autonomy:

The hybrid case preserves independent event-driven agents while adding a way to compensate for partially completed work. Bhaumik uses this matrix in customer discussions and points to Agent Bricks as a way to package common coordination patterns. Choosing the pattern establishes who coordinates execution; it does not yet establish how agents safely exchange state.

A stale cache is one consistency failure. A lost update is another. Suppose A and B both read a credit score of 680. A computes and writes 750; B then writes 720. Under a last-write-wins approach, A's update disappears. Database protections only help when the application uses an appropriate transaction and concurrency strategy. Bhaumik calls out explicit transactions, row locks, serializable isolation, and SELECT FOR UPDATE; these are mechanisms to choose and apply correctly, not protections to assume from default settings.

The alternative is immutable state snapshots with versioned handoffs. A produces version one and seals it. The orchestrator stores it as a new row in an append-only log. B receives that specific version, validates its schema, processes it, and inserts version two rather than updating version one. C then receives version two. If C fails, the workflow can return to that known input without reconstructing a record that other agents have overwritten.

The history records what each agent received and produced. Appending a new version removes concurrent modification of the same snapshot; passing an explicit version also avoids asking an ambiguous shared record for the latest value. The snapshots serve as inputs and audit records, not shared mutable working memory. This addresses the demonstrated handoff failures, while selection of the correct version and consistency of other data sources remain responsibilities of the workflow.

In Python, the demonstrated AgentState carries a version, a payload, and a creator. The handoff validates the contract, creates the next version, and invokes the next agent with immutable input. One compact implementation is:

from collections.abc import Callable
from dataclasses import dataclass


@dataclass(frozen=True)
class CreditData:
    customer_id: str
    credit_score: int


@dataclass(frozen=True)
class AgentState:
    version: int
    data: CreditData
    created_by: str


def validate_credit(data: CreditData) -> None:
    if not isinstance(data, CreditData):
        raise TypeError("Expected CreditData")
    if not data.customer_id or type(data.credit_score) is not int:
        raise ValueError("Invalid credit record")


def handoff(
    state: AgentState,
    next_agent: Callable[[AgentState], AgentState],
    validate: Callable[[CreditData], None],
) -> AgentState:
    validate(state.data)
    next_input = AgentState(
        version=state.version + 1,
        data=state.data,
        created_by=state.created_by,
    )
    return next_agent(next_input)

Python's frozen=True blocks ordinary attribute reassignment; it does not recursively freeze a dictionary or list inside an object. Here, the payload is also a frozen dataclass containing immutable scalar values. An agent returns a new state with its output and creator rather than modifying next_input. Persistence of those versions belongs to the orchestrator.

Version history also narrows debugging. If version seven contains a bad result, inspect the version-six input and then the earlier versions to locate where the state first diverged. Bhaumik suggests binary search through the history; that is useful when a reproducible check can distinguish a good prefix from the bad states that follow it. The underlying benefit is retained evidence at every handoff.

Immutability preserves an input; a data contract determines whether the next agent should accept it. The research agent promises findings, a confidence score, sources, and a timestamp. The analysis agent declares the corresponding input requirements and validates them before doing its work. In the example, it rejects research with confidence below 0.7. That threshold is an acceptance rule, not evidence that the confidence score is calibrated.

The rule belongs at the boundary, before analysis runs:

from dataclasses import dataclass
from datetime import datetime
from math import isfinite


@dataclass(frozen=True)
class ResearchOutput:
    findings: tuple[str, ...]
    confidence: float
    sources: tuple[str, ...]
    timestamp: datetime


def validate_research(output: ResearchOutput) -> None:
    if not isinstance(output, ResearchOutput):
        raise TypeError("Expected ResearchOutput")
    for values in (output.findings, output.sources):
        if not isinstance(values, tuple):
            raise TypeError("Expected tuple of strings")
        if not all(isinstance(value, str) for value in values):
            raise TypeError("Expected string entries")
    if not isinstance(output.timestamp, datetime):
        raise TypeError("Expected datetime timestamp")
    if type(output.confidence) not in (int, float):
        raise TypeError("Expected numeric confidence")
    if not isfinite(output.confidence) or not 0 <= output.confidence <= 1:
        raise ValueError("Confidence must be between 0 and 1")
    if output.confidence < 0.7:
        raise ValueError("Research confidence below acceptance threshold")

A rejected handoff is easier to diagnose than a bad report three agents downstream. Bhaumik proposes registering input and output schemas in Unity Catalog so the organization can govern and version the contracts centrally. The receiving agent still needs to enforce its contract during execution.

Even with explicit coordination and valid inputs, an LLM can time out, an API can rate-limit requests, and an agent can crash. A circuit breaker prevents a repeatedly failing dependency from consuming the workflow's resources on every subsequent call. In Bhaumik's example, five consecutive failures open the circuit. New calls then fail immediately instead of waiting for another timeout.

Recovery is a state machine, not an unlimited retry loop:

The example waits 60 seconds before permitting the half-open probe. A failed probe resets the cooldown; a successful probe restores normal operation. This gives the dependency room to recover instead of bombarding it with requests.

Opening the circuit still leaves a business decision to make:

• Reduced functionality: Skip an optional agent and continue with a smaller result.
• Cached results: Use a previous result where its age and meaning are acceptable for the task.
• Human intervention: Alert an operator when continuing automatically would be unsafe.

Bhaumik recommends protection around every agent call and places these policies at the serving boundary, through Databricks Model Serving or AI Gateway. The linked Databricks documentation reflects later 2026 APIs rather than an archived snapshot of the talk. In particular, documented rate limits and per-request fallbacks do not by themselves implement the failure counter, cooldown, and half-open state machine described here. That breaker behavior needs an explicit implementation.

The call wrapper first checks whether the circuit permits a request. A successful closed-state call resets the failure count; a failed call increments it. Reaching the threshold opens the circuit, and the recovery probe determines when it closes again. Record those transitions as operational events. Bhaumik suggests logging them in MLflow so an investigator can see when an agent began failing, alongside the individual calls that led to the transition.

A circuit breaker contains repeated calls, but it does not remove side effects from work already completed. For that, the workflow needs Saga compensation. Each participating agent exposes execute and compensate, and the orchestrator records which steps succeeded. In the example, research caches its findings, analysis writes a draft recommendation, and the execution agent fails. Recovery first deletes the analysis draft, then clears the cached research.

Those two compensations return this example to its initial state because its side effects are reversible. In a general compensating transaction, recovery is application-specific: an external action may be irreversible, concurrent changes may need to survive, and compensation itself can fail. A Saga therefore needs defined recovery actions; it is not a database rollback spanning arbitrary services.

The demonstrated execution order is straightforward:

1. Execute A; record A as successfully completed.
2. Execute B; record B as successfully completed.
3. Execute C; if C fails, enter recovery.
4. Compensate B, then compensate A.

The successful-step list is the basis of recovery. For this workflow, reversing that list also reverses the dependencies between the side effects. The compensate methods must encode the actual undo operations rather than merely mark a step as failed. Bhaumik emphasizes this planning particularly for financial-services workflows, where partial completion cannot simply be ignored.

The combined architecture places a workflow engine and a state store holding versions zero through N inside the orchestrator's responsibility, with an observability layer recording execution. A returns version one. The orchestrator gives that same snapshot to B and C in parallel, stores their results as versions two and three, and calls D with the combined outputs. Those version numbers identify retained results; because B and C share a parent input, they do not imply that C consumed B's output.

All coordination passes through the orchestrator, giving operators one place to inspect execution and initiate recovery. Bhaumik says this architecture runs around the clock across billions of transactions, but supplies no workload, measurement period, or reliability metric for that scale claim. The concrete design to carry forward is the ownership boundary: agents perform work; the orchestrator decides when they run, what state they receive, and what happens after failure.

In the proposed Databricks implementation, LangGraph wired into Mosaic AI Agent Framework manages the graph and call order. Agent capabilities are represented by Unity Catalog functions written in SQL or Python, or by registered models. Central registration makes these assets discoverable and governable. The serving boundary—described as Model Serving or Function Serving—exposes those capabilities and is where Bhaumik places retries, timeouts, rate limits, and call protection.

Delta Lake holds customer data, other workflow data, and the state history. Each agent result becomes a new state row. Append-only state is an application policy: existing snapshot rows are never changed by this workflow. A Delta table itself is not inherently immutable; Delta Lake supports updates, deletes, and merges. Keeping state versions intact requires the application and its access controls to preserve that policy.

Each agent run is associated with its state version so an investigator can follow the data through execution. Unity Catalog supplies the governance layer for access control, lineage, and audit trails. MLflow tracing supplies per-agent execution records, while its evaluation capabilities include LLM judges and call metrics. Agent Bricks sits above these components as a packaged route to common multi-agent use cases.

The final walkthrough follows one execution path:

1. LangGraph calls A, a governed function or model, and writes its result as state version one in Delta.
2. It calls B with version one and appends version two.
3. Serving-layer protection guards calls while telemetry records latency, inputs, outputs, and token usage.
4. If C fails, the workflow invokes the compensation methods of previously successful steps in reverse order.

Tracing requires instrumentation, such as mlflow.langchain.autolog() for the LangGraph integration, plus application context linking calls to state versions and breaker transitions. Likewise, the compensation path must be implemented as workflow logic; LangGraph checkpoint recovery does not automatically undo external side effects. Together, these pieces connect execution, retained state, diagnosis, and recovery.

Scaling past one agent introduces coordination, consistency, and partial-failure problems. How the architecture handles them is a choice. Orchestration or choreography establishes coordination; immutable state and contracts establish handoffs; circuit breakers and compensation establish behavior when execution fails. These mechanisms need to be designed into the workflow, not added only after an impressive demo encounters production traffic.

The work is ordinary infrastructure engineering, and that is why it matters. A circuit breaker may earn little applause, but preventing a dependency failure from becoming a late-night incident is visible over time. The production achievement is a system that continues to behave predictably when its agents do not.