AI Engineer Europe 2026
From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik
Read the talk
From One Agent to a Reliable Distributed System
A stale credit score shows why multi-agent workflows need explicit coordination, immutable handoffs, circuit breakers, and compensation—not just more capable agents.
From a talk by Sandipan Bhaumik
Before you start: Familiarity with agent tool calls, database reads and writes, and basic Python will help you follow the examples.
When the next agent changes the problem
One agent works well: an LLM, prompts, perhaps a retrieval pipeline and some tool calls. The demo succeeds. Then product asks for five more agents. Now A produces data B needs, C waits for both A and B, D changes the state B is reading, and E crashes midway through the workflow. Adding agents introduces a distributed system, with dependencies and failure modes that model quality alone cannot resolve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A successful database write, an incorrect decision
Sandipan Bhaumik describes a financial-services deployment that began with a credit-score agent running for two weeks without issues. The team then added income verification, risk assessment, fraud detection, and final approval. Bhaumik reports that within three days of deploying all five agents, 20% of decisions had incorrect risk ratings. Customers who should have been flagged were receiving approvals, and diagnosis took two days.
The credit-score agent wrote 750 to PostgreSQL; 500 milliseconds later, the risk agent read 680 for the same customer from the cache. The database write had succeeded. The missing operation was cache invalidation, so the risk agent made its decision from an older customer record.
The defect sat between the agents and the database: several agents shared a cache without coordinated invalidation. Fixing the prompt would not have repaired that read path. The investigation delayed delivery because the team had built a distributed system without explicitly designing its consistency behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose who owns coordination
The number of relationships explains why the transition feels abrupt. For a fully connected set of agents, the undirected connection count is:
That gives zero connections for one agent, one for two agents, and ten for five agents. Bhaumik describes the increase as exponential and invokes a 25-fold complexity increase, but these connection counts establish quadratic growth, not a measured complexity multiplier. Each potential relationship still introduces another place for failures, races, and state synchronization problems. The response starts with choosing a coordination pattern, then specifying state and recovery behavior.
Choreography coordinates agents through events. A research agent finishes and publishes a research-completed event to a message bus. An analysis agent subscribes to that event type, processes the research, and publishes analysis-ready. A report agent consumes that event and generates the report. No central component calls each participant in sequence; each agent reacts to the events it understands.
That loose coupling makes it easy to add another subscriber without changing a central workflow. It also moves the difficulty into operations. When a report never appears, did research fail to publish, did analysis fail to consume, or was an event consumed twice? Choreography requires end-to-end event tracing and deliberate delivery guarantees. It fits naturally event-driven workflows with independent agents and frequent additions, provided the team can reconstruct event propagation when something fails. Choosing it simply because autonomy feels more agentic leaves that operational work undone.
Orchestration puts those decisions in a coordinator. It calls A and waits for the result, starts B and C in parallel, waits for both, then calls D with their combined outputs. The agents do not call one another. They accept inputs, perform work, and return results; the orchestrator owns the execution graph, state, retries, and step logs. Bhaumik names LangGraph wired into Databricks AI Agent Framework as one implementation, while emphasizing that a workflow engine with suitable dependency graphs and retries can fill the same role.
Central control is useful when dependencies are complex, partial work needs compensation, and operators need one view of the workflow. It is especially attractive when the graph changes less often than the agents' internal implementations. Bhaumik reports using orchestration almost exclusively in financial-services work because a failed credit decision must be reconstructed: which agent acted, in which order, and with which data. Choreography can support such reconstruction through strong tracing, but orchestration makes that execution record a direct responsibility of the coordinator.
The decision framework uses workflow complexity and required autonomy:
| Workflow | Autonomy requirement | Suggested pattern |
|---|---|---|
| Simple | High | Choreography |
| Complex | Low | Orchestration |
| Complex | High | Choreography with Saga compensation |
The hybrid case preserves independent event-driven agents while adding a way to compensate for partially completed work. Bhaumik uses this matrix in customer discussions and points to Agent Bricks as a way to package common coordination patterns. Choosing the pattern establishes who coordinates execution; it does not yet establish how agents safely exchange state.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pass versions instead of overwriting shared records
A stale cache is one consistency failure. A lost update is another. Suppose A and B both read a credit score of 680. A computes and writes 750; B then writes 720. Under a last-write-wins approach, A's update disappears. Database protections only help when the application uses an appropriate transaction and concurrency strategy. Bhaumik calls out explicit transactions, row locks, serializable isolation, and SELECT FOR UPDATE; these are mechanisms to choose and apply correctly, not protections to assume from default settings.
The alternative is immutable state snapshots with versioned handoffs. A produces version one and seals it. The orchestrator stores it as a new row in an append-only log. B receives that specific version, validates its schema, processes it, and inserts version two rather than updating version one. C then receives version two. If C fails, the workflow can return to that known input without reconstructing a record that other agents have overwritten.
The history records what each agent received and produced. Appending a new version removes concurrent modification of the same snapshot; passing an explicit version also avoids asking an ambiguous shared record for the latest value. The snapshots serve as inputs and audit records, not shared mutable working memory. This addresses the demonstrated handoff failures, while selection of the correct version and consistency of other data sources remain responsibilities of the workflow.
In Python, the demonstrated AgentState carries a version, a payload, and a creator. The handoff validates the contract, creates the next version, and invokes the next agent with immutable input. One compact implementation is:
python
from collections.abc import Callable
from dataclasses import dataclass
@dataclass(frozen=True)
class CreditData:
customer_id: str
credit_score: int
@dataclass(frozen=True)
class AgentState:
version: int
data: CreditData
created_by: str
def validate_credit(data: CreditData) -> None:
if not isinstance(data, CreditData):
raise TypeError("Expected CreditData")
if not data.customer_id or type(data.credit_score) is not int:
raise ValueError("Invalid credit record")
def handoff(
state: AgentState,
next_agent: Callable[[AgentState], AgentState],
validate: Callable[[CreditData], None],
) -> AgentState:
validate(state.data)
next_input = AgentState(
version=state.version + 1,
data=state.data,
created_by=state.created_by,
)
return next_agent(next_input)
Python's frozen=True blocks ordinary attribute reassignment; it does not recursively freeze a dictionary or list inside an object. Here, the payload is also a frozen dataclass containing immutable scalar values. An agent returns a new state with its output and creator rather than modifying next_input. Persistence of those versions belongs to the orchestrator.
Version history also narrows debugging. If version seven contains a bad result, inspect the version-six input and then the earlier versions to locate where the state first diverged. Bhaumik suggests binary search through the history; that is useful when a reproducible check can distinguish a good prefix from the bad states that follow it. The underlying benefit is retained evidence at every handoff.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reject unacceptable inputs at the handoff
Immutability preserves an input; a data contract determines whether the next agent should accept it. The research agent promises findings, a confidence score, sources, and a timestamp. The analysis agent declares the corresponding input requirements and validates them before doing its work. In the example, it rejects research with confidence below 0.7. That threshold is an acceptance rule, not evidence that the confidence score is calibrated.
The rule belongs at the boundary, before analysis runs:
python
from dataclasses import dataclass
from datetime import datetime
from math import isfinite
@dataclass(frozen=True)
class ResearchOutput:
findings: tuple[str, ...]
confidence: float
sources: tuple[str, ...]
timestamp: datetime
def validate_research(output: ResearchOutput) -> None:
if not isinstance(output, ResearchOutput):
raise TypeError("Expected ResearchOutput")
for values in (output.findings, output.sources):
if not isinstance(values, tuple):
raise TypeError("Expected tuple of strings")
if not all(isinstance(value, str) for value in values):
raise TypeError("Expected string entries")
if not isinstance(output.timestamp, datetime):
raise TypeError("Expected datetime timestamp")
if type(output.confidence) not in (int, float):
raise TypeError("Expected numeric confidence")
if not isfinite(output.confidence) or not 0 <= output.confidence <= 1:
raise ValueError("Confidence must be between 0 and 1")
if output.confidence < 0.7:
raise ValueError("Research confidence below acceptance threshold")
A rejected handoff is easier to diagnose than a bad report three agents downstream. Bhaumik proposes registering input and output schemas in Unity Catalog so the organization can govern and version the contracts centrally. The receiving agent still needs to enforce its contract during execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop calling a dependency that keeps failing
Even with explicit coordination and valid inputs, an LLM can time out, an API can rate-limit requests, and an agent can crash. A circuit breaker prevents a repeatedly failing dependency from consuming the workflow's resources on every subsequent call. In Bhaumik's example, five consecutive failures open the circuit. New calls then fail immediately instead of waiting for another timeout.
Recovery is a state machine, not an unlimited retry loop:
| State | Call behavior | Transition |
|---|---|---|
| Closed | Call the agent normally | Success resets failures; threshold opens circuit |
| Open | Reject calls immediately | Cooldown permits a probe |
| Half-open | Allow one test request | Success closes; failure reopens |
The example waits 60 seconds before permitting the half-open probe. A failed probe resets the cooldown; a successful probe restores normal operation. This gives the dependency room to recover instead of bombarding it with requests.
Opening the circuit still leaves a business decision to make:
- Reduced functionality: Skip an optional agent and continue with a smaller result.
- Cached results: Use a previous result where its age and meaning are acceptable for the task.
- Human intervention: Alert an operator when continuing automatically would be unsafe.
Bhaumik recommends protection around every agent call and places these policies at the serving boundary, through Databricks Model Serving or AI Gateway. The linked Databricks documentation reflects later 2026 APIs rather than an archived snapshot of the talk. In particular, documented rate limits and per-request fallbacks do not by themselves implement the failure counter, cooldown, and half-open state machine described here. That breaker behavior needs an explicit implementation.
The call wrapper first checks whether the circuit permits a request. A successful closed-state call resets the failure count; a failed call increments it. Reaching the threshold opens the circuit, and the recovery probe determines when it closes again. Record those transitions as operational events. Bhaumik suggests logging them in MLflow so an investigator can see when an agent began failing, alongside the individual calls that led to the transition.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Undo completed work when a later step fails
A circuit breaker contains repeated calls, but it does not remove side effects from work already completed. For that, the workflow needs Saga compensation. Each participating agent exposes execute and compensate, and the orchestrator records which steps succeeded. In the example, research caches its findings, analysis writes a draft recommendation, and the execution agent fails. Recovery first deletes the analysis draft, then clears the cached research.
Those two compensations return this example to its initial state because its side effects are reversible. In a general compensating transaction, recovery is application-specific: an external action may be irreversible, concurrent changes may need to survive, and compensation itself can fail. A Saga therefore needs defined recovery actions; it is not a database rollback spanning arbitrary services.
The demonstrated execution order is straightforward:
- Execute A; record A as successfully completed.
- Execute B; record B as successfully completed.
- Execute C; if C fails, enter recovery.
- Compensate B, then compensate A.
The successful-step list is the basis of recovery. For this workflow, reversing that list also reverses the dependencies between the side effects. The compensate methods must encode the actual undo operations rather than merely mark a step as failed. Bhaumik emphasizes this planning particularly for financial-services workflows, where partial completion cannot simply be ignored.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Assemble the workflow, state store, and telemetry
The combined architecture places a workflow engine and a state store holding versions zero through N inside the orchestrator's responsibility, with an observability layer recording execution. A returns version one. The orchestrator gives that same snapshot to B and C in parallel, stores their results as versions two and three, and calls D with the combined outputs. Those version numbers identify retained results; because B and C share a parent input, they do not imply that C consumed B's output.
All coordination passes through the orchestrator, giving operators one place to inspect execution and initiate recovery. Bhaumik says this architecture runs around the clock across billions of transactions, but supplies no workload, measurement period, or reliability metric for that scale claim. The concrete design to carry forward is the ownership boundary: agents perform work; the orchestrator decides when they run, what state they receive, and what happens after failure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Map the architecture onto Databricks
In the proposed Databricks implementation, LangGraph wired into Mosaic AI Agent Framework manages the graph and call order. Agent capabilities are represented by Unity Catalog functions written in SQL or Python, or by registered models. Central registration makes these assets discoverable and governable. The serving boundary—described as Model Serving or Function Serving—exposes those capabilities and is where Bhaumik places retries, timeouts, rate limits, and call protection.
Delta Lake holds customer data, other workflow data, and the state history. Each agent result becomes a new state row. Append-only state is an application policy: existing snapshot rows are never changed by this workflow. A Delta table itself is not inherently immutable; Delta Lake supports updates, deletes, and merges. Keeping state versions intact requires the application and its access controls to preserve that policy.
Each agent run is associated with its state version so an investigator can follow the data through execution. Unity Catalog supplies the governance layer for access control, lineage, and audit trails. MLflow tracing supplies per-agent execution records, while its evaluation capabilities include LLM judges and call metrics. Agent Bricks sits above these components as a packaged route to common multi-agent use cases.
The final walkthrough follows one execution path:
- LangGraph calls A, a governed function or model, and writes its result as state version one in Delta.
- It calls B with version one and appends version two.
- Serving-layer protection guards calls while telemetry records latency, inputs, outputs, and token usage.
- If C fails, the workflow invokes the compensation methods of previously successful steps in reverse order.
Tracing requires instrumentation, such as mlflow.langchain.autolog() for the LangGraph integration, plus application context linking calls to state versions and breaker transitions. Likewise, the compensation path must be implemented as workflow logic; LangGraph checkpoint recovery does not automatically undo external side effects. Together, these pieces connect execution, retained state, diagnosis, and recovery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production value comes from the system
Scaling past one agent introduces coordination, consistency, and partial-failure problems. How the architecture handles them is a choice. Orchestration or choreography establishes coordination; immutable state and contracts establish handoffs; circuit breakers and compensation establish behavior when execution fails. These mechanisms need to be designed into the workflow, not added only after an impressive demo encounters production traffic.
The work is ordinary infrastructure engineering, and that is why it matters. A circuit breaker may earn little applause, but preventing a dependency failure from becoming a late-night incident is visible over time. The production achievement is a system that continues to behave predictably when its agents do not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Configure checkpoints, inspect execution history, and recover stateful LangGraph workflows.
Enable MLflow autologging to capture graph execution in a unified trace.
Examples of Delta Lake mutation operations and their table semantics.
Further reading
- Compensating transactionsDocumentation
Design recovery actions for partially completed workflows, including concurrent changes and irreversible operations.
Reference for dataclass fields and the attribute protections provided by frozen instances.
Updates since the talk
Current instructions for configuring a managed supervisor that coordinates specialized agents and governed tools.
Legacy endpoint configuration for rate limits, logging, traffic splitting, and external-model fallbacks.
Read the complete timestamped transcript
- 0:00
Hi, everyone. I'm Sandy. I have spent eighteen years building data systems, a major part of it focusing on building and scaling distributed data systems in the cloud. I've done it for multi-tenant systems for software and SaaS companies, and then for scaling data and AI platforms in regulated industries like financial services and healthcare.
- 0:19
I've learnt a great deal about production-grade distributed systems while I have been working at AWS and now in Databricks. For the last two years, I've been deploying multi-agent AI systems in production, and I have watched brilliant engineers make the same mistakes over and over.
- 0:36
They think adding more agents is just like adding more features. It's not. It's building a distributed system. And today, I'm going to show you the patterns that actually work when you make that transition.
- 0:48
These are lessons that I have learnt working in the trenches, and today I'm here to share it with you. Here's what we are covering today. First, the problem. I'll share you a very basic production war story about race conditions and why complexity ex-explodes when you go from one agent to five agents.
- 1:07
Um, I'll, I'll talk about the patterns, choreography and orchestration patterns for coordination of agents. I'll talk about state management, uh, talk about failure recovery and how we can, um, design for failure in production systems.
- 1:21
And then I'll, I'll share how a production-grade architecture will look like, uh, uh, in as simple way possible, and I'll also show you an example on how we build this on Databricks.
- 1:31
So let's dive into it. You see, one agent works beautifully. You have got your LLM, some prompts, maybe a retrieval-augmented generation pipeline, maybe some tool calls. It demos great.
- 1:45
Leadership loves it. You feel happy, and your team is happy. And then product comes back with a request that changes everything. They want five more agents, and here's what happens.
- 1:57
You think, "Okay, I know how to build agents, and I will add five more." Except now you have coordination problems. Agent A produces data that Agent B needs. Agent C is waiting on both Agent A and Agent B.
- 2:11
Agent D just updated the shared state that Agent B was reading, and Agent E just crashed and took down, down this entire workflow. This is no longer an AI problem.
- 2:23
This is a distributed system problem, and most of you didn't sign up to be distributed systems engineer. Let me tell you about a production deployment where this went very wrong.
- 2:34
We built a credit decisioning system for a financial services company. The first agent credit score calculation worked perfectly. It worked great in demos, two weeks in production, zero issues.
- 2:45
Then we added four more agents, income verification, risk assessment, fraud detection, and final approval. Uh, we deployed all five. In three days' time, we started seeing weird approvals. Uh, twenty percent of the decisions had incorrect risk ratings.
- 3:01
Customers who should have been flagged were getting approved. The business team was panicking. It took us two days to find out what was happening. Credit score agent calculated a score of seven fifty and wrote to the database.
- 3:14
The risk assessment agent, on the other hand, read from the database five hundred milliseconds later and got a score of six eighty for the same customer. Why did it happen?
- 3:25
Because we had a caching layer for customer records. The write to PostgreSQL succeeded, but the cache was not invalidated. The risk agent read from the cache, and it got stale data.
- 3:40
Used-- It used the wrong score and made the wrong decision. This is a classic distributed systems problem. We had caching layer between the agents and the database. Cache invalidation failed, and the agent was reading stale val-values.
- 3:56
The race condition wasn't in the database, it was in the architecture. Multiple agents shared cache, no coordination on cache invalidation. This took us quite a while to find the pattern.
- 4:09
It created delays in delivery and led to wrong decisions. And here's the lesson we learnt. The problem was, of course, not with the model. The problem wasn't with the prompts.
- 4:20
The problem was we built a distributed system without distributed system thinking, and that's what kills multi-agent projects. Not bad AI, but bad architecture. Now, I will show you the architecture that works.
- 4:34
We will also look into a production-grade architecture. But first, let's understand why this complexity explodes so quickly. Now, when you move from, uh, a one-agent system to a multi-agent, let's say five-agent systems, it doesn't get just five times harder.
- 4:51
It gets twenty-five times more complex. Coordination comp-complexity grows exponentially. One agent has got zero coordination problems. Two agents have got at least one connection. Five agents have got at least ten potential connections and coordination.
- 5:06
Each connection is a failure point, a race cond-condition, a state synchronization problem. You are not just building five agents, you are building a coordination problem across multiple relationships and across...
- 5:20
and, and possibility to have multiple failure modes. And that's why the complexity increases very, very quickly. Now, I'm going to show you two critical patterns. First pattern is about how to coordinate multiple agents.
- 5:35
Then we will talk about how you can manage state, and then we'll talk about how we can recover and design for failure. Now, these patterns come from multiple years of distributed systems work, and I can directly apply them on multi-agent AI system.
- 5:48
Once you get the basics, it's really hard to miss these patterns when you build multi-agent AI architecture. The first decision you need to make is about choreography or, uh, orchestration.
- 5:59
These are the two fundamental patterns for distributed coordination. Choreography means agents coordinate through events. They are decentralized, they are autonomous. Orchestration means a central coordinator manages the workflow. This is centralized and controlled.
- 6:15
Most teams pick one instinctively and regret it. Let me show you when to use each. Let's start with choreography. Choreography is event-driven. Um, the research agent finishes, uh, research and publishes a research completed event to a message bus.
- 6:33
Event B subscribed to that message bus and listens for the e-event type it is interested in. The analysis agent subscribes to that event type, picks it up, does analysis, and publishes analysis ready.
- 6:46
Then the report agent picks that analysis ready event, generates the report. There is no central coordinator here. Each agent is autonomous, listening for events it cares about, publishing when it is done.
- 7:00
This is the beauty of choreography. Agents are loosely coupled. It's easy to add, add new agents and make them subscribe to the events that they're interested in. This drives high autonomy and scales really well.
- 7:13
However, the nightmare of choreography is debugging. When something fails, you're playing detective with no real clue. Which agent failed to publish? Did the event get consumed? Did the event get consumed twice?
- 7:26
You need bulletproof observability to make choreography work. Even with the event propagation, you need strong, uh, guarantees across delivery of these events. Without this, debugging is really hard. So when should you use choreography?
- 7:42
You use choreography when your workflow is naturally event-driven, when agents need to operate independently, when you are adding agents frequently and don't want to update a central coordinator. But it is important to understand it is possible only if you have strong observability.
- 8:02
If you can't trace events through your system, choreography will destroy you. I've seen teams choose choreography because it feels more agentic, more autonomous. Then they fi-- spend months firefighting because they can't debug distributed event flows.
- 8:16
Don't make that mistake. Now let's look at the alternative, orchestration. Orchestration is centralized. You have a workflow orchestrator that calls each agent directly. Agent A runs first. The orchestrator calls agent A, waits for the result, gets the result back, then the orchestrator calls agent B and C in parallel if they are agents that need to run in
- 8:39
parallel. The orchestrator manages the parallelism, not the agents. B and C return their results to the orchestrator. Then the orchestrator calls agent D with the combined results from B and C.
- 8:50
Every call goes through the orchestrator. Agents never call each other. The orchestrator is the single source of truth. It knows the entire execution graph. It manages state. It handles retries.
- 9:02
It logs every step. Agents are dumb. They just take the input, they do the work, they return the output. The orchestrator does all the smart coordination. In Databricks, one way to implement this pattern would be with LangGraph wired into AI Agent Framework as the orchestrator.
- 9:21
But any workflow that gives you DAGs, direct acyclic graphs, and proper retry mechanisms would fit in this kind of orchestrator patterns. You use orchestration when you have complex dependencies that need central management, when you need to roll back, compensate for failures, when you want one dashboard showing the entire system state, when your workflow is
- 9:46
relatively stable. In financial services, for example, we use orchestration almost exclusively. Why? Because it provides easy debugging and the ability to roll back, and that matters more than autonomy in these kind of industries.
- 10:01
When something goes wrong with a credit decision, for example, we need to know exactly which agent made that call, in what order, and with what data. Orchestration gives us that.
- 10:12
Choreography doesn't. So how do you choose? Here's your decision framework. Two axes.
- 10:18
Workflow complexity, simple to complex. Autonomy requirements, low to high. Simple workflow, high autonomy, you go with choreography. You need complex workflow with low autonomy tolerance, you go with orchestration.
- 10:32
The interesting quadrant is the top right, where you need complex workflow, but agents need autonomy. This is where you use hybrid patterns, choreography with Saga patterns for compensation. I'll talk about this pattern later in this, uh, session as well.
- 10:49
Uh, tools like Agent Bricks on Databricks are starting to package these orchestration patterns for common multi-agent use cases, so you don't need to rebuild them every time. It makes building these patterns really easy in production environments.
- 11:06
Now, I use the decision metrics, uh, every time to make decisions with customers based on their use cases. Uh, it's worth you take a screenshot. I'm sure you will reference it.
- 11:16
Let me show you what a production orchestration actually looks like at the tail end of this session. All right. Now we have chosen a co-coordination, uh, pattern. Now let's talk about the thing that actually breaks when you scale, state.
- 11:28
How do agents share data without race conditions, without stale reads, without mystery bugs? Here's what most people do first, and it's wrong. Shared mutable state. Multiple agents writing at the same database records at the same time.
- 11:44
Agent A reads credit score, calculates the value, writes it back. Agent B does the same thing at the same time. Both read six eighty. Agent A writes seven fifty.
- 11:56
Agent B writes seven twenty. Last write wins. Agent A's update disappears. Lost update. Uh, I understand, yes, modern databases have protections in place, row logs, isolation levels, et cetera, but you have to use them correctly.
- 12:12
Explicit transactions, um, you have to build, uh, serializable isolation. Uh, you have to make sure that you select for update Uh, and, and many teams don't. Uh, they use default isolation.
- 12:27
They don't use explicit locks, and they ship a race condition to production. We did it. We did that mistake, and that resulted in delayed value to the business. We just assumed that the database would handle these conditions, but they don't.
- 12:40
When it gets really complex, you have to handle them explicitly in the code. Now, here's what works. Uh, immutable state snapshots with versioning. Agent A produces a state version, let's say version one.
- 12:52
It's sealed, it's immutable. Nobody can modify it. State is stored in the orchestrator database as an append-only log. These are insert operations, not, not any update. Agent A hands state version one to Agent B.
- 13:07
Agent B validates the schema, checks that the data contract matches with its expectations. It processes it, produces state version two, also immutable. Agent B inserts version two as the new row, it doesn't update version one, and then hands it to Agent C.
- 13:23
Same thing, schema validation, version tracking, immutability guarantee at each handoff. Agent C fails. Now, if Agent C fails, you roll back to version two. If you need to debug, you replay state evolution, uh, from version one through version N.
- 13:41
You can see exactly what each agent received and produced. This eliminates race conditions. No concurrent modification to the same record. Each agent appends a new version instead of updating the shared state.
- 13:56
Now, of course, if you want to s- uh, save these state snapshots, they can be logged, uh, in any sort of append-only storage for audit replay, but they are never shared for read or write.
- 14:07
Now, here's how it looks like in code. AgentState class, the frozen means immutable in Python. It has a version number, the data pa-payload, and who created it. The handoff function does three things.
- 14:19
First, it validates the schema. Uh, this is the contract enforcement. We are checking that Agent A's output matches Agent B's input contract. This is critical, and we will come back to this.
- 14:32
Second, increment version. Create a new immutable state object with version n plus one. Third, execute the next agent with that immutable state. The agent can't modify the input state, it can only produce a new state.
- 14:48
This prevents an entire class of bugs. It prevents race conditions on shared state. No stale reads. It provides a clear lineage. Every state has a version, and you know who has created it.
- 15:01
When something goes wrong, you can trace back through state evolution. Version seven produced bad output? Look into version six that went into the agent. Look at version five before that.
- 15:11
You can binary search through your state history to find where things w-went wrong, and this becomes really, really powerful. Now, state management is half the battle. Data contracts are the other half.
- 15:23
Agent A can just throw, um, arbitrary data at Agent B and hope it works. This doesn't work that way. They need a contract in place. In this example, research agent promi-promises to output findings, confidence score, sources, timestamp, et cetera.
- 15:42
Analysis agent declares it requires research agent output with type and first,
- 15:48
uh, and it validates. If confidence is below zero point seven, it will reject the handoff. This is the contract. If the research hand-- uh, if the research agent tries to hand off low-quality data, the contract catches it at the boundary.
- 16:05
You find out immediately, not three agents down the stream when it produces a report in garbage. When we work with our customers, um, using Databricks, uh, one way of doing it is, uh, registering these input/output schemas in Unity Catalog, uh, so every agent's contract is versioned and governed in one place.
- 16:24
All right, we talked about coordination patterns, we talked about state management. Now talk about... Now, now let's talk about another thing that you need to keep in mind, and that's failure and recovery.
- 16:34
And, and the reason this is important is because agents will fail. That's inevitable. The LLM will time out, the API will rate limit you, the agent will crash mid-workflow.
- 16:44
What happens then? What happens then is what you need to plan for and design in the system. Let's talk about a few patterns. Let's talk about the first pa-pattern, which is a circuit breaker pattern, and this comes straight from distributed system.
- 16:56
When Agent A calls Agent B, it wraps that call in a circuit breaker. If Agent B fails repeatedly, say five times in a row, the circuit breaker opens. Now, instead of waiting for a timeout every single time, you basically fail fast.
- 17:12
Circuit open, Agent B is down, you just try again later. You are not bombarding Agent B with requests. You're protecting your system. After a ta-timeout period, let's say sixty seconds, k- s- the circuit goes half open.
- 17:26
Then you test Agent B again with one request. If it succeeds, the star- circuit closes and normal operation resumes. If it fails, the circuit opens again, and it resets the timer.
- 17:37
This prevents you from cascading failures into the system.
- 17:42
One agent going down doesn't bring your entire workflow down. You gracefully degrade. Maybe you skip that agent and continue with a reduced functionality. Maybe you use cached results. Maybe you alert a human, but you don't crash the entire workflow.
- 18:01
Circuit breakers are the single most important failure recovery pattern for multi-agent systems. Every agent call should be wrapped with a circuit breaker. We enforce these circuit breaker policies at the serving layer on Databricks through Model Serving or through AI Gateway.
- 18:15
Here's how it looks like in code. You track the failure count, and you track the state. When you call an agent, you check the state first. If it is open, you fail fast.
- 18:24
You don't even try. If it is closed, you make the call. If the call succeeds, you reset the failure count and stay closed. If it fails, you increment the failure count.
- 18:34
If you hit the threshold, you open the circuit. After the timeout period, you transition to half open. You test one request. If it succeeds, you close the circuit. If it fails, you open it again.
- 18:46
This is a simple pattern, but it has got a massive impact. And in Databricks, you can log every open-closed transition in MLflow, so you can see when an agent started flaking out.
- 18:58
Now let's talk about ano-another pattern. We call it the compensation pattern, also called Saga pattern. Every agent has two methods: execute and compensate. Execute does the work, compensate rolls it back, undoes it.
- 19:13
The orchestrator tracks which agents have executed. If the execution agent fails, the orchestrator work-- walks backward through the executed agents, and it calls compensate for each one.
- 19:28
Analysis agent compensates. It deletes the draft recommendation from the system that it has written originally, and then the research agent compensates by clearing the cached research data that it gathered previously.
- 19:40
So you're back to the initial state. No partial transactions, no stuck workflows. This is a simple rollback pattern that you can implement in multi-agent system. Compensation gives you transactional semantics a-across distributed agents.
- 19:55
It is not sexy, but it's how production systems handle partial failures. Every orchestrated workflow needs this kind of compensation pattern, and you need to plan for it depending on what you're doing with your workflows.
- 20:07
Here's how compensation looks in code. Every agent, as I mentioned earlier, has got two methods: the execution method and the compensate method. The execution does the work, the compensate undoes it.
- 20:20
Uh, that is the contract. Every operation must be reversible. The orchestration tracks which, uh-- The orchestrator tracks which agents have run successfully, and then it keeps a list. Agent A ex-executes, gets added.
- 20:34
Agent B executes, gets added. Agent C fails. Now we walk backward through the list in reverse order. Agent B compensates first, is it undoes the work that it has done.
- 20:45
Agent A compensates next. It undoes the work that Agent A has done, and it goes back to the initial state. This is Saga pattern from distributed databases. Financial services requires this.
- 20:56
Now that we have covered these different patterns, I want to show you what a production architecture would look like when you bring these things together. You have got the orchestrator at the left-hand side.
- 21:05
Um, it's the brain of the workflow. It contains the workflow engine, it contains the state store, uh, holding versions to zero to N, and it has, uh, it, it, it can look into the observability layer.
- 21:19
It handles the observability data. Every call goes to the orchestrator. Orchestrator calls Agent A. Agent Bay-- Agent A returns state version one to the orchestrator. Orchestrator then calls Agent B and C in parallel, if they need to run in parallel.
- 21:35
Both receives ver-- state version one from the orchestrator. They return results. Orchestrator stores as version three-- two and three. Finally, orchestrator calls D with these combined results. Agents never call each other.
- 21:47
All coordination happens through the orchestrator, and this is what gives us control, observability, capability to roll back. This runs twenty-four cross seven across billions of transactions because the orchestrator is the single source of truth.
- 22:03
All right, here's a production architecture that you could implement with the Databricks Data Intelligence platform. The orchestration layer, you can have LangGraph wired into Mosaic AI Agent Framework. It handles multi-agent orchestration.
- 22:18
It manages the workflow graph and knows which agents to call in what order. Each agent is implemented as a Unity Catalog function. It could be written in SQL or Python, or it could be a model registered in a Unity Catalog.
- 22:32
Um, they are-- When you register these assets in Unity Catalog, they are se-- discoverable centrally within the organization. Uh, they can be governed in one place, and they can be versioned, which is really critical, uh, in terms of operating these, uh, workflows in production.
- 22:50
We expose these agents through a Databricks Model Serving or Function Serving, and that's where we enforce these circuit breaker-style policies, like retries or timeouts or rate limits, uh, at the serving layer, typically via AI Gateway configuration.
- 23:05
Now, when we talk about the data layer, Delta Lake stores everything. It not only stores the state versions from the agent, it also stores customer data and, you know, all, all, all the data that you need for your workflows to work.
- 23:21
Um, talking about the sna-state snapshots, Delta Table, uh, is immutable and versioned. For us, those state versions are just rows in a Delta table. Uh, we never update them in place.
- 23:36
Each agent run is tied to a state version via MLflow traces, so we can step through the evolution when something breaks. Now, uh, I just wanted to touch upon, uh, Unity Catalog.
- 23:47
It, it, it governs everything: access control, lineage, audit trail for both data and agents. MLflow gives us per-agent tracing, evaluation capabilities with out-of-the-box LLM as judges and,
- 24:03
and metrics on every call. And as I mentioned earlier, um, tools like Agent Bricks is the higher-level way of Databricks packaging these orchestration patterns for common multi-agent use cases, so you don't need to rebuild them every time.
- 24:19
So just to wrap up this workflow, uh, you see Ran-- uh, the LangGraph orchestrator calls Agent A a Unity Catalog function or model. It gets the result, writes version one state to Delta.
- 24:32
It then calls Agent B with state version one, writes version two, and so on. MLflow traces every call, latency, inputs, outputs, token usage. A circuit breaker at the serving layer guards each call.
- 24:46
If Agent C fails, LangGraph triggers compensation logic and walks backward, calling the compensate functions for previous successful steps. These kind of patterns run in production day in and day out.
- 24:58
So thank you for hearing me out. Uh, you can reach out to me over LinkedIn. You can scan this QR that will take you directly to my LinkedIn profile.
- 25:07
Uh, I, I would like, like to three-- leave you with three final thoughts. First of all, agent chaos is inevitable. When you scale past one agent, you will, you will hit coordination problems, race conditions, cascading failures.
- 25:22
That's guaranteed. The complexity curve doesn't lie. Your agent choreography is a choice. You can build systems with proper patterns, orchestration, choreography, immutable state, circuit breakers, compensation patterns, data contracts.
- 25:39
Make sure you understand these patterns and bring them to your production architecture. Doing so will help you build systems, not demos. Demos are easy. You use an LLM to show something cool, everyone can do it.
- 25:52
These things don't work in production. In production, you have to build systems, and systems are hard. Systems are what create value for businesses. Everything I showed you today, choreography versus orchestration, immutable state, circuit breakers, these are all unsexy infrastructure work.
- 26:10
You won't get applause for implementing a circuit breaker, but you make your systems more reliable. They don't fail at two AM in the night. That is what people notice over time.
- 26:20
Be a systems engineer. The patterns here, they work. Apply these patterns in your production architecture. Thank you very much for watching. Bye.