# AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok

Salman Munaf, TikTok

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/hD9-V56FNRI-ai-agents-are-just-distributed-systems-now) · [Video](https://www.youtube.com/watch?v=hD9-V56FNRI)

## AI Agents Are Just Distributed Systems Now

Salman Munaf explains how tool-using agents inherit remote-call failures, stale state and partial transactions—and how deterministic controls can limit the consequences of probabilistic decisions.

### When a wrong answer can change production

[0:12](https://www.youtube.com/watch?v=hD9-V56FNRI&t=12s)

A wrong model answer becomes a different engineering problem when the system can delete a database or make a refund decision. Salman Munaf, listed for this session as a Lead Site Reliability Engineer at TikTok, opens with two incident examples: a Replit agent deleting a production database and an Air Canada chatbot making what he describes as an incorrect refund. These are motivations for his proposed safeguards, rather than detailed incident reconstructions.

The examples expose two separate responsibilities:

- **Limit destructive actions:** Backups provide a recovery path, while scoped permissions can prevent an agent from deleting a production database in the first place.
- **Retrieve authoritative policy:** A chatbot making policy-dependent decisions needs a source of truth so that stale or incorrect information does not drive its answer.

The transition from chatbot to agent expands the architecture. A text-only interaction ends with an output; an agent loop can call tools, reach external services and change their state. The design inventory therefore has to include the systems the agent talks to, the state it reads or changes, the credentials it holds and the actions those credentials permit. Model quality covers only part of that system.

### A probabilistic coordinator needs recorded steps

[3:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=212s)

At [3:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=212s), the useful architectural description is a **probabilistic coordinator**. Distributed services already coordinated multistep workflows, but the traditional comparison here is a mapped decision tree. An agent can vary both the kind and the number of actions it chooses. That flexibility requires deterministic controls around the actions whose consequences would be unacceptable.

The loop plans, acts, observes, may persist information and then decides what to do next. Planning can retrieve external data. Acting can call an API or write to a database. Observation may return only partial results, which then become inputs to the next decision. Persisting an incorrect observation can carry the mistake forward, and a decision to retry can multiply the work sent downstream.

Recording each step’s actions and retrieved context gives recovery a place to start: the workflow can identify where it failed and which earlier operations need attention. But recording an operation does not define its repair. Each step also needs an explicit failure response or compensating action. An incorrect email to a customer makes the distinction clear: knowing that the message was sent is necessary, yet the system still needs a decision about how to correct its consequences.

### A refund timeout leaves the outcome unknown

[6:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=413s)

A tool call wraps an external API, database or queue. The wrapper inherits network delays, timeouts and duplicate requests. Crucially, the service may complete an operation even though the caller receives an error. A database write can exist in the authoritative store while the client’s observation suggests something went wrong.

Follow the refund example through that discrepancy. The agent invokes a customer-refund tool, then the request times out. The visible change is in the agent’s knowledge: it no longer has a confirmed outcome. The customer’s money may already have been refunded. Treating the timeout as proof of failure and issuing a second refund can therefore turn one intended operation into two financial effects.

Three tool capabilities address different parts of this problem:

- **Request identifiers:** Identify the earlier operation so that the workflow can refer to it again.
- **Idempotency keys:** Let the receiving tool recognize repeated requests and prevent repeated side effects.
- **Status lookup:** Check what happened to the earlier request against the system that owns its outcome.

How can a completed refund and an uncertain caller coexist? The diagram separates the operation’s effect from the response the agent receives. It also shows why recovery needs both an outcome lookup and duplicate protection: inspecting the earlier request resolves uncertainty, while recognizing a repeated operation prevents a retry from refunding the customer again.

One refund operation, two different observations
The service can complete the refund while the agent receives a timeout. A status lookup checks the earlier outcome; idempotency prevents a repeated request from creating another refund.
- Agent requests a refund — The tool request identifies the intended operation.
- Refund tool and external service — A remote operation may succeed even when the caller receives an error.
- Refund may already exist — A timeout does not establish whether the financial effect happened.
- Agent sees a timeout — The outcome is unknown to the caller.
- Look up the earlier request — Check its status using the operation’s identity.
- Recognize a repeated operation — The receiving tool prevents another side effect.
- Agent requests a refund → Refund tool and external service (Remote tool call)
- Refund tool and external service → Refund may already exist (Possible successful effect)
- Refund tool and external service → Agent sees a timeout (Caller receives no confirmed outcome)
- Agent sees a timeout → Look up the earlier request (Check what happened)
- Agent sees a timeout → Recognize a repeated operation (If the operation is retried)

### Duplicate protection does not stop a retry storm

[9:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=545s)

Retrying is the failure response Munaf expects builders to plan around. Idempotency belongs in the receiving API or tool because the agent’s choice to retry does not establish that another side effect is safe. The tool must recognize the repeated request and avoid doing the operation again.

Even retries that produce no additional side effects still send traffic downstream. A large fan-out or aggressive retry loop can burden an external API and cause cascading failures. Maximum turns limit how long the loop continues; maximum parallel calls limit simultaneous work; spending ceilings limit the resource commitment. Exponential backoff increases the spacing between attempts, reducing pressure on dependencies while the workflow tries to recover.

These controls solve different problems. Idempotency prevents repeated effects, backoff and execution limits restrain repeated work, and compensation addresses the consequences of work that has already happened. None substitutes for the others.

### Context that influences an action is state

[10:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=625s)

At [10:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=625s), the same systems thinking reaches memory. Context becomes state when it can influence an action. It can go stale, conflict with authoritative data and corrupt future decisions. A remembered policy is operationally consequential if it determines what the agent does next.

The memory distinction here concerns where information persists:

- **Short-term memory:** The chat thread belongs to a single execution thread.
- **Long-term memory:** Project files, system prompts, databases and cache layers can supply information beyond that thread.

Neither location settles which information should win when sources disagree.

Treating memory as a cache makes that decision explicit. Attach provenance so that remembered information has a known origin, choose the authoritative source when values conflict, and invalidate the corresponding memory when that source changes. The causal sequence matters: updating a database does not by itself remove the old value from the agent’s context. Invalidation prevents that retained value from continuing to drive actions.

### A failed CRM update can leave two successful steps behind

[12:08](https://www.youtube.com/watch?v=hD9-V56FNRI&t=728s)

A multistep workflow can update an internal ticket, send an email to a customer and then fail to update the CRM. The failure leaves the systems at different stages of completion. Recovery must consider the earlier ticket change and email alongside the failed CRM operation; simply treating the whole workflow as unsuccessful ignores effects that already exist.

What remains after the last step fails? The diagram preserves the successful earlier steps instead of collapsing the workflow into a single failure marker. That partial outcome is the input to compensation: the system needs a defined response for the consequences of each completed operation.

The email example also sets a limit on what recovery means. An incorrect message can be followed by an apology or a corrective email, but the original communication has already occurred. Compensation can repair its consequences through another action without restoring the exact prior state. Its meaning depends on the operation, so the agent needs that response defined rather than being left to improvise after failure.

Partial completion across three systems
The failed CRM update leaves a changed ticket and a sent email behind. If the email was incorrect, its defined compensation is a corrective or apology message.
- Internal ticket updated — The first operation succeeds.
- Customer email sent — The second operation also succeeds.
- CRM update fails — The workflow stops with earlier effects still present.
- Apply defined compensation — Recovery accounts for operations that already succeeded.
- Correct an incorrect email — Send an apology or corrective message if that mistake occurred.
- Internal ticket updated → Customer email sent (Next step)
- Customer email sent → CRM update fails (Next step)
- CRM update fails → Apply defined compensation (Recover from partial completion)
- Apply defined compensation → Correct an incorrect email (If the sent message was incorrect)

### Stop unsupported work and scope the work that remains

[13:22](https://www.youtube.com/watch?v=hD9-V56FNRI&t=802s)

Circuit breakers put a stopping condition around calls to unhealthy or saturated dependencies. An agent may keep trying to solve a problem, but the surrounding system should prevent those attempts from burdening a dependency that cannot support them. This helps contain cascading failures. Rate limits and maximum turns, parallelism and spending separately bound how much effort the agent can commit.

Those ceilings trade open-ended persistence for a predictable resource commitment. The agent can continue problem-solving only within the limits the system sets. A desire to complete the task cannot be allowed to expand traffic or cost indefinitely.

Permissions need the same specificity. Broad database read-and-write access may make it easy to equip an agent for a task, but it also makes unsafe decisions executable. Separate read and write permissions, scope credentials and allowlist the tools the agent can call. The permitted action set determines how much damage an incorrect decision can cause.

Human approval is another scoped permission. It should bind to the action and its parameters, the actor, timestamp and expiration. Approval for a **$30 refund** cannot become authorization for a subsequent **$300 refund**. Changing the amount changes the operation that was approved; a blanket approval loses precisely the information needed to enforce that distinction.

### Reconstruct the failure, then ask what the system permits

[16:31](https://www.youtube.com/watch?v=hD9-V56FNRI&t=991s)

Observability has to reconstruct a workflow, including what information the agent reacted to. An isolated error log cannot explain whether a bad decision came from stale context, a tool response or an earlier write. Traces need to connect the model invoked and its prompt with retrieved context, tool requests, responses, errors, writes and approvals.

For the refund example, that connection distinguishes the request the agent attempted from the outcome it observed and the state the external system actually changed. For a policy-dependent action, it preserves the information used to decide. The purpose is to understand what happened and locate a recovery path across the model and the services around it.

The [closing discussion at 17:31](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1051s) retains a role for better models: greater capability improves the likelihood of correct operations. It cannot eliminate network failures, stale data or adversarial input. The architecture still needs to bound actions, observe their effects and recover when something goes wrong.

Tool contracts make those expectations concrete. Define the operations a tool permits, its request and response types and its schemas. Build idempotency into tools that receive repeated requests. Decide which source wins when memory conflicts, constrain retries with policies and rate limits, and provide permissions, traces and recovery paths. These are parts of the executable system around the coordinator.

The final design question is what the system lets the agent do when it is wrong. That question makes mistakes an expected operating condition. It asks whether an incorrect decision can become an unsafe operation, whether its effects can be understood and whether the workflow has a way back.

## Key takeaways

- An agent coordinates external systems probabilistically. Deterministic controls must constrain the actions its decisions can cause.
- A timeout leaves the operation’s outcome unknown. Request identifiers and status lookups help resolve it; idempotency prevents repeated requests from creating repeated effects.
- Duplicate protection, backoff, circuit breakers and execution budgets address different failure modes: repeated effects, excessive load, unhealthy dependencies and runaway cost.
- Memory that influences action needs provenance, an authoritative source and invalidation. Recovery also needs defined compensation for earlier successful steps.
- Scope credentials and human approvals to the intended operation. Approval for a $30 refund does not authorize $300.
- Better models reduce mistakes, while tool contracts, traces and recovery paths determine what happens when a mistake or infrastructure failure still occurs.

## Related talks

- [12-Factor Agents: Patterns of reliable LLM applications](https://ai.engineer/talks/8kMaTybvDUw-12-factor-agents) — Develops practical patterns for owning deterministic control flow, structured tool calls, context and human interaction.
- [Building Durable, Production-Ready Agents with OpenAI SDK and Temporal](https://ai.engineer/talks/k8cnVCMYmNc-building-durable-production-ready-agents-openai) — Shows an implementation approach to persisted calls, workflow state and worker recovery, extending the talk’s requirement to record steps.
- [Your Agent Didn’t Fail. Your Harness Did.](https://ai.engineer/talks/BInpv7lGp1o-your-agent-didnt-fail-your-harness-did) — Examines related failures involving missing state, scoped approvals and tool outcomes, with concrete harness controls.

## Slides

### AI Agents Are Also Distributed Systems

Salman Munaf argues that agents interacting with external systems require distributed-systems thinking.

[0:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=19s) [Title slide with a colorful network illustration and presenter Salman Munaf.](https://assets.ai.engineer/sites/aiecode2025/sha256/b50a9e08269c2b1e42cbf1e0a11241b10ab130c5fe85622c2c11437bd5e7fa1b.webp)

AI Agents Are Also Distributed Systems

Salman Munaf

### When AI Agents Fail: Lessons from Distributed Systems

Munaf connects a reported Replit database deletion, a multi-agent privilege escalation, and Air Canada’s refund advice to controls including scoped permissions, enforced oversight, and current policy retrieval.

[1:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=66s) [Table of three AI incidents, what happened, and proposed distributed-systems lessons.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d3260720d97f8d8ca8d73393aa59d666396c3af46c0148faff6350b8059ad19.webp)

When AI Agents Fail: Lessons from Distributed Systems

Public examples of true LLM agents causing catastrophic production failures are still emerging. These incidents highlight critical lessons when AI or autonomous software operates without robust distributed-systems controls.

Incident: Replit AI agent deleted a production database, 2025
What Happened: An AI coding agent reportedly ignored a code freeze and deleted a live production database, impacting over 1,200 executives/companies.
Distributed Systems Lesson: Emphasizes need for dev/prod separation, durable code freezes, scoped authority, robust backups, and explicit approval workflows.

Incident: Multi-agent system escalated privileges, 2026
What Happened: A deployed multi-agent system installed 107 unauthorized components, overwrote registry, ignored prior oversight rejection, and attempted high-privilege commands.
Distributed Systems Lesson: Oversight rejections must be machine-enforced; policy gates are crucial for shell access and privilege escalation.

Incident: Air Canada chatbot invented refund policy, 2024
What Happened: Air Canada's chatbot provided incorrect bereavement-fare refund advice, leading to the airline being legally liable for compensation.
Distributed Systems Lesson: Requires authoritative source-of-truth retrieval, versioned policy checks, and proper escalation for ambiguous or critical information.

### From Chatbot to Production System

A chatbot returns text; an agent can call tools and change state, creating side effects in the outside world. Map its external dependencies, credentials, and actions before building the agent loop.

[2:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=140s) [Comparison of chatbot responses and agent actions, highlighting external side effects.](https://assets.ai.engineer/sites/aiecode2025/sha256/9609b27a8e13ac35e19ecaa29b8b4467881fbb21a0be8e22c44e9f8576b76d5a.webp)

THE ARCHITECTURAL SHIFT
From Chatbot to Production System

Chatbot
Prompt in, text out. No side effects.

Agent
Prompt in, loop runs, tools called, state changed. The architectural boundary shifts when the model can affect the outside world.

The difference is side effects.

Design takeaway: Before building the agent loop, map every external dependency, state store, tool, credential, and irreversible action.

Diagram:
LLM CHATBOT
PROMPT: Tell me a joke.
PROCESSING
RESPONSE: Why did the scarecrow win an award? Because he was outstanding in his field!

AI AGENT
Book a weekend trip to London.
PLAN
ADVANCED AGENT
TOOL SELECTION
CHECK
RE-EVALUATE
EXECUTE
EXECUTION
Search Flights API
Find Hotel API
Booking API
[illegible]

### The Probabilistic Coordinator

The LLM planner sits inside deterministic controls, which confine its variable actions, and a production system containing tools, memory, traces, and APIs.

[3:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=210s) [Nested circles show an LLM planner within deterministic controls and a production system.](https://assets.ai.engineer/sites/aiecode2025/sha256/056b335859947013fd94ea9eef16aee25fb9ba29fd9a936be0cd9adab2ebba41.webp)

The Probabilistic Coordinator

MENTAL MODEL

An agent is a probabilistic coordinator inside a distributed system. The LLM plans — but the runtime, tools, memory, policies, and traces determine whether the system is safe to operate.

Production System
Tools, memory, traces, and APIs

Deterministic Controls
State machine, retries, permissions, approvals

LLM Planner
Probabilistic, nondeterministic coordinator

### Tool Calls Are Remote Calls

A refund request can time out after the server succeeds, leaving the agent unsure whether to retry. Request IDs, idempotency keys, and status lookup help prevent duplicate side effects.

[6:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=412s) [Slide describing partial failures, unknown outcomes, and safeguards for side-effecting tools, beside a server illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/270c8b3c308b595b8f21f0a1e759d356235a194afc4aa8079aabde9e6b2e9489.webp)

Tool Calls Are Remote Calls
REMOTE CALLS

Tool calls are not function calls. In production, they are calls to APIs, databases, payment systems, or internal services. Remote calls have failure modes a local function does not.

Partial Failure
Network delay, timeout, duplicate request, server-side success with client-side failure.

Unknown Outcomes
Agent calls refund_customer. The request times out. Did the refund happen?
Timeout does not mean failure. It means unknown.

Design Takeaway
Every side-effecting tool needs request IDs, idempotency keys, structured errors, status lookup, and clear timeout semantics.

### Safe Retries Require Idempotency

Idempotent writes prevent duplicate side effects when agents retry. Retry budgets and exponential backoff limit load; compensation reverses unsafe effects when retry is not possible.

[9:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=543s) [Slide explaining idempotency, retry budgets, exponential backoff, and compensation.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfb5349dc6600412b65999b587fd5795cd621738479bef3adf66273d985d708c.webp)

RELIABILITY PATTERNS

Safe Retries Require Idempotency

1 Idempotency
(same request produces same result, no duplicate side effects)

2 Retry Budgets
(max turns, max spend, max parallel calls)

3 Exponential Backoff
(increasing delay between retries to reduce load)

4 Compensation
(undo or reverse unsafe side effects when retry is not possible)

Agents retry constantly

Tools fail, models replan, users resubmit, workers crash. Retrying unsafe actions can duplicate side effects.

• Classify retryable vs. non-retryable failures
• Use retry budgets and exponential backoff
• Design idempotent write tools
• Define compensation for unsafe operations

A retry policy is part of the agent design, not an implementation detail.

### Memory Is Production State

Agent memory can influence future actions after its data becomes stale. Choose a source of truth and, unless designed otherwise, treat memory as a cache with provenance and invalidation rules.

[10:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=623s) [Slide comparing short-term and long-term memory and identifying a source of truth.](https://assets.ai.engineer/sites/aiecode2025/sha256/354d29a5c64a858e5ff4fc4c262f0413453d64056cc333d24c6c2b89dc9a9b64.webp)

STATE MANAGEMENT

Memory Is Production State

If memory influences future action, it is no longer just context — it becomes state that can go stale, conflict with authoritative data, or corrupt future decisions.

Short-Term Memory
Current thread, checkpoint, run state. Tied to a single execution.

Long-Term Memory
User facts, preferences, summaries. Can be stale, poisoned, or inconsistent.

Source of Truth
Decide what is authoritative. Treat memory as a cache unless designed otherwise. Attach provenance and invalidation rules.

If agent memory disagrees with the database, which one wins?

### Partial Failure Requires Compensation

When an agent fails midway across services, it must identify the partial state and compensate for completed steps. An incorrect email, for example, may require a correction or apology.

[12:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=725s) [Slide describing partial failures across services and a workflow to execute, detect failure, and compensate.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a483a37fa00ee82677f351c6849bd9b75590d6ad8115e5c2ff5346e8a95230a.webp)

FAILURE HANDLING

Partial Failure Requires Compensation

The Problem

Agent completes step 1, fails step 2, never reaches step 3. It updates a ticket but fails to email the user. It sends the email but fails to update the CRM.

Agent workflows operate across services that do not share one transaction. You cannot roll everything back atomically.

Rollback is usually a fantasy. Compensation is engineering.

1 Execute
run steps 1 through N, persist each status
1 → 2 → N

2 Detect Failure
identify which step failed, what partial state exists

3 Compensate
run undo actions for completed steps, escalate if compensation fails

### Control the Loop

Circuit breakers stop calls to unhealthy dependencies, while limits on turns, parallelism, and spending keep agent retries and fan-out within budget.

[13:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=796s) [Slide describing circuit breakers and limits on turns, parallelism, and spending.](https://assets.ai.engineer/sites/aiecode2025/sha256/c07ebf7ecba36b4c5421a14322ef164db3e11aa6148e27b10a88ac579f64c93c.webp)

Control the Loop

FLOW CONTROL

Circuit Breakers
Stop the loop when a dependency is unhealthy. Prevent cascading failures from saturating tools.

Rate Limits & Budgets
Max turns, max parallelism, max spend. Agents can fan out and burn through model budget without explicit stopping behavior.

### Scope the Authority

Give agents scoped credentials and tool access, then bind approvals to specific actions and parameters. A $30 refund approval should not cover a $300 refund.

[14:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=890s) [Slide outlining least privilege and durable approval for agent actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/9de7de5749fea2b29b8bb06dc98e959a5b66e0f5a4c4856fec7f4756bb5c350d.webp)

AUTHORITY
Scope the Authority

Least Privilege
Scoped credentials, read/draft/execute separation, tool allowlists. A harmless model becomes risky when it can spend money or delete data.

Durable Approval
Bind approval to action, parameters, actor, timestamp, and expiration. Re-approve if material inputs change. Approval is a workflow event, not a UI modal.

### Trace The Causal Path

A causal trace connects model calls, context reads, approvals, tool calls, and memory writes so teams can reconstruct what information an agent acted on and why it failed.

[16:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=989s) [Slide showing a path from model call to memory write and listing events to trace.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffc1da020e288ed97dfb49ce733c68692b8cb08562793505607186f44917fe1b.webp)

OBSERVABILITY & DEBUGGING

Trace The Causal Path

Model Call → Context Read → Approval → Tool Call → Memory Write

Why Traces Matter
Teams need to reconstruct what happened. Logs alone are not enough; agent runs need causal traces.

What to Trace
• Model calls (prompt, tokens, latency)
• Tool calls (request, response, errors)
• Retrieved context (what the agent saw)
• Memory writes (state changes)
• Guardrails and approvals (policy decisions)
• Handoffs and retries (workflow events)
• Errors and final actions (outcomes)

### The Architecture Is the Safety Case

Smarter models reduce mistakes, but failures remain. Tool contracts, idempotency, permissions, traces, and recovery paths help bound what an agent can do and recover when it is wrong.

[17:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1049s) [Closing thesis comparing model capability with system architecture, emphasizing contracts, retries, permissions, traces, and recovery.](https://assets.ai.engineer/sites/aiecode2025/sha256/70e22154dfa6147d0a3f731a307fee55fc632dec497ec89356d85f4bc75b7396.webp)

CLOSING THESIS

The Architecture Is the Safety Case

Model Capability
Can it decide?
• Better prompts improve local behavior
• Smarter models reduce mistakes
• Cannot eliminate network failure, stale data, or adversarial input

System Architecture
Can we bound, observe, and recover?
• Tool contracts and idempotency
• Source-of-truth decisions and retry policy
• Permissions, traces, and recovery paths

Do not only ask only how smart the agent is. Also ask what the system lets it do when it is wrong.

## Full transcript

[0:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=hD9-V56FNRI&t=12s) Hello everyone. Good good afternoon. Uh

[0:15](https://www.youtube.com/watch?v=hD9-V56FNRI&t=15s) today uh I will be talking about AI

[0:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=17s) agents are also distributed systems.

[0:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=20s) So as uh the models have started to

[0:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=24s) become more complex. Initially the uh

[0:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=27s) LLM models were just text in text out

[0:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=30s) without performing any actions and uh

[0:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=33s) the

[0:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=35s) effect that they can produce was just a

[0:38](https://www.youtube.com/watch?v=hD9-V56FNRI&t=38s) wrong model output. However, with

[0:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=42s) now the capability of agent the agent

[0:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=45s) the rise in agent capabilities where the

[0:47](https://www.youtube.com/watch?v=hD9-V56FNRI&t=47s) systems can now talk to external systems

[0:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=50s) uh it has turned into a distributed

[0:54](https://www.youtube.com/watch?v=hD9-V56FNRI&t=54s) systems and it is important to

[0:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=56s) incorporate distributed systems thinking

[0:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=59s) and concepts when building AI agents. So

[1:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=62s) I will be going over uh that uh in this

[1:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=65s) talk.

[1:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=67s) So you guys might have uh heard about

[1:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=71s) incidents being caused by a AI agents.

[1:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=74s) Uh for instance, the replicate AI agent

[1:18](https://www.youtube.com/watch?v=hD9-V56FNRI&t=78s) deleting a production incident

[1:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=80s) production database or Air Canada

[1:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=83s) chatbot basically making an uh an

[1:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=86s) incorrect refund. And both of these uh

[1:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=90s) incidents or a lot of these incidents

[1:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=92s) could have been prevented uh by good

[1:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=96s) systems thinking when building these uh

[1:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=100s) AI agents. So for instance for the

[1:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=102s) replet a uh uh uh incident we could have

[1:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=108s) good uh

[1:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=110s) uh we could have robust backups. We

[1:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=113s) could have scoped authority. we should

[1:54](https://www.youtube.com/watch?v=hD9-V56FNRI&t=114s) we shouldn't ideally have uh allow AI

[1:58](https://www.youtube.com/watch?v=hD9-V56FNRI&t=118s) agents to delete production databases.

[2:00](https://www.youtube.com/watch?v=hD9-V56FNRI&t=120s) Uh moreover for Air Canada chatbot

[2:04](https://www.youtube.com/watch?v=hD9-V56FNRI&t=124s) it would have been uh a good idea to

[2:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=126s) have uh authoritative source of truth

[2:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=129s) retrieval so that it's not making uh uh

[2:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=133s) decisions based on stale or incorrect

[2:15](https://www.youtube.com/watch?v=hD9-V56FNRI&t=135s) policies.

[2:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=139s) So let's uh go over the transition from

[2:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=141s) chatbot to production system. Uh so

[2:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=146s) uh initially when we were in the uh in

[2:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=150s) the uh age where LLMs were just chat

[2:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=153s) bots uh we had prompt in and we were

[2:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=157s) outputting text there were no side

[2:39](https://www.youtube.com/watch?v=hD9-V56FNRI&t=159s) effects the agent was not interacting

[2:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=161s) with any other system.

[2:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=164s) However, uh due to agentic uh in the

[2:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=168s) agentic era in the agentic revolution,

[2:51](https://www.youtube.com/watch?v=hD9-V56FNRI&t=171s) now those agents uh by ingesting prompt

[2:54](https://www.youtube.com/watch?v=hD9-V56FNRI&t=174s) can uh run an agent loop, call external

[2:58](https://www.youtube.com/watch?v=hD9-V56FNRI&t=178s) services, call tools and also perform

[3:00](https://www.youtube.com/watch?v=hD9-V56FNRI&t=180s) state changes. The architectural

[3:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=182s) boundary now has moved uh way beyond an

[3:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=186s) LLM model. And the difference is that it

[3:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=190s) can now cause side effects in the

[3:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=191s) outside world. So when basic when

[3:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=194s) building AI agents, it is important to

[3:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=197s) recognize the external systems that it

[3:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=199s) is talking to, [clears throat] the

[3:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=201s) states that uh it is interacting with

[3:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=205s) and what credentials does it have and

[3:28](https://www.youtube.com/watch?v=hD9-V56FNRI&t=208s) the actions that it can perform.

[3:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=212s) uh I ideally like to think about it as

[3:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=216s) uh uh AI agents as basically having a

[3:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=220s) probabilistic coordinator.

[3:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=222s) In distributed systems as well, we used

[3:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=224s) to have services which were coordinating

[3:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=226s) uh multi-step workflows. However, they

[3:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=229s) were deterministic in nature. But in the

[3:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=232s) case of AI agent, the AI acts as a

[3:55](https://www.youtube.com/watch?v=hD9-V56FNRI&t=235s) probabilistic coordinator. The amount of

[3:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=237s) action, the kind of actions that it can

[3:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=239s) take can vary quite a lot. It is not

[4:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=242s) just a decision tree that uh we

[4:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=245s) typically in traditional systems would

[4:08](https://www.youtube.com/watch?v=hD9-V56FNRI&t=248s) have mapped out

[4:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=250s) and those uh actions can have severe

[4:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=253s) consequences uh ba uh if they are not

[4:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=257s) confined by our determinist by having

[4:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=261s) deterministic controls in place. So it

[4:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=264s) is important to ensure uh that we have

[4:28](https://www.youtube.com/watch?v=hD9-V56FNRI&t=268s) deterministic controls in place to

[4:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=270s) ensure that agent or the AI agent is not

[4:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=273s) performing any uh

[4:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=276s) any actions that might be uh uh

[4:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=280s) problematic.

[4:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=282s) So uh

[4:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=285s) let's uh discuss the how a typical agent

[4:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=289s) loop might look like. So at first it

[4:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=293s) might uh do some planning. Then based on

[4:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=297s) that plan it will it will perform an

[4:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=299s) action and it will then observe the

[5:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=302s) results of those actions and uh it might

[5:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=305s) persist that into some d some data store

[5:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=309s) and then decide what to do next.

[5:12](https://www.youtube.com/watch?v=hD9-V56FNRI&t=312s) Each step in this loop is basically

[5:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=314s) crossing a a boundary. During planning,

[5:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=317s) it can interact with data sources to

[5:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=320s) retrieve some data. Uh during action, it

[5:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=323s) can call external APIs, tools, uh

[5:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=327s) databases and perform any actions.

[5:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=330s) During observation phase, it can perform

[5:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=333s) it can get partial results and basically

[5:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=337s) plan or make subsequent actions based on

[5:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=341s) those partial results. It can persist

[5:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=344s) incorrect data or uh and uh when

[5:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=348s) deciding it might also

[5:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=352s) decide to uh perform an incorrect action

[5:55](https://www.youtube.com/watch?v=hD9-V56FNRI&t=355s) or uh worse it can also do a retry

[5:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=359s) storm.

[6:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=361s) So it is very important when building an

[6:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=363s) agent loop to persist every step of the

[6:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=367s) process. Whatever actions the agent is

[6:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=369s) doing, whatever context it is

[6:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=371s) retrieving, it is important to uh

[6:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=373s) persist that so that if anything fails,

[6:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=376s) the agent is able to recognize where it

[6:18](https://www.youtube.com/watch?v=hD9-V56FNRI&t=378s) failed and it can perform uh a

[6:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=381s) reversible action. Uh it can perform

[6:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=384s) undo operations. Similarly, there should

[6:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=386s) be explicit transactions uh identified

[6:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=390s) for each step. So for instance, if an

[6:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=392s) agent is making a call, if it fails,

[6:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=395s) what it should do? What should be the

[6:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=397s) transaction to compensate for a uh for a

[6:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=400s) irreversible or unsafe operation? For

[6:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=403s) instance, if an agent makes sends an

[6:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=405s) email to a a wrong email to a customer,

[6:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=408s) what should it do to compensate for

[6:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=410s) that?

[6:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=413s) So, uh tool calls are just wrappers

[6:58](https://www.youtube.com/watch?v=hD9-V56FNRI&t=418s) around uh external external APIs,

[7:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=422s) databases, cues, uh and so on.

[7:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=426s) And uh with uh when calling the when

[7:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=430s) making these remote calls, there are

[7:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=433s) some failures that you incorporate uh

[7:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=436s) such as network delays, timeouts, uh you

[7:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=439s) can make duplicate requests or worse the

[7:22](https://www.youtube.com/watch?v=hD9-V56FNRI&t=442s) server side request uh might succeed.

[7:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=446s) However, the client however the client

[7:28](https://www.youtube.com/watch?v=hD9-V56FNRI&t=448s) might be reported an error.

[7:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=450s) We have we have seen uh instances where

[7:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=455s) uh a data by base might have written the

[7:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=457s) data. However, due to some other errors,

[7:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=460s) the server might have reported uh uh to

[7:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=464s) us the error and

[7:47](https://www.youtube.com/watch?v=hD9-V56FNRI&t=467s) uh with humans in the loop we can make

[7:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=470s) we can basically perform correct

[7:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=473s) corrective actions based on uh by seeing

[7:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=476s) uh the database and actual source of

[7:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=477s) truth. But in agent's case, we need to

[8:00](https://www.youtube.com/watch?v=hD9-V56FNRI&t=480s) ensure that we have uh we have proper

[8:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=483s) guardrails in place. So for instance,

[8:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=486s) an agent calls refund customer uh tool

[8:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=489s) call which basically performs a refund

[8:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=491s) to the customer. The request times out

[8:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=494s) uh that did the refund happen or not?

[8:18](https://www.youtube.com/watch?v=hD9-V56FNRI&t=498s) What will the agent uh infer from that?

[8:22](https://www.youtube.com/watch?v=hD9-V56FNRI&t=502s) Would it retry uh refunding to the

[8:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=504s) customer? you basically the the timeout

[8:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=507s) does not actually mean that there a

[8:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=509s) failure had occurred. It means unknown.

[8:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=512s) And it is important to have uh when

[8:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=516s) designing these tools, it is important

[8:38](https://www.youtube.com/watch?v=hD9-V56FNRI&t=518s) to have request ids, item potency keys

[8:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=522s) so that when making duplicate requests,

[8:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=524s) they are not causing duplicate side

[8:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=526s) effects. uh and the system can always do

[8:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=529s) a status lookup like what the previous

[8:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=532s) request was and what was the status of

[8:55](https://www.youtube.com/watch?v=hD9-V56FNRI&t=535s) that so that it is not making side

[8:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=537s) effect it is not making side effects

[9:00](https://www.youtube.com/watch?v=hD9-V56FNRI&t=540s) with duplicate a with duplicate

[9:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=542s) requests.

[9:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=545s) So [clears throat] AI agents when they

[9:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=549s) whenever they uh uh whenever they uh

[9:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=554s) they face failures they retry the their

[9:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=557s) first uh action is to perform retries.

[9:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=560s) So it is really important to have item

[9:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=563s) potency baked in. uh it if a same

[9:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=567s) request is coming in to an external API

[9:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=570s) or the tool it should recognize that

[9:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=572s) this is a duplicate request and ensure

[9:34](https://www.youtube.com/watch?v=hD9-V56FNRI&t=574s) that no side effects are being take are

[9:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=576s) taking place. Moreover, uh we should

[9:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=580s) also prevent uh AI agents to perform

[9:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=583s) retry storms to external APIs because

[9:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=585s) this can cause cascading failures. uh we

[9:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=589s) we should have max turns budget spend

[9:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=592s) and max parallel calls to prevent uh to

[9:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=596s) uh to ensure that the fan out is not

[9:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=599s) that large. Moreover, we should have

[10:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=602s) exponential back back off in place to

[10:04](https://www.youtube.com/watch?v=hD9-V56FNRI&t=604s) ensure that uh the downstream

[10:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=607s) dependencies are not being uh burdened

[10:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=611s) and we should also have compensation uh

[10:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=614s) operations in place for uh operations

[10:18](https://www.youtube.com/watch?v=hD9-V56FNRI&t=618s) that uh that that can have side effects.

[10:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=625s) Uh

[10:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=627s) so uh

[10:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=629s) a lot of uh

[10:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=632s) teams when building AI agents think of

[10:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=635s) AI agent context as just a AI cont the

[10:39](https://www.youtube.com/watch?v=hD9-V56FNRI&t=639s) context that uh the AI agent has as uh

[10:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=643s) as just a context. However, when that

[10:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=646s) context can influence an action, it's a

[10:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=649s) state and that state can become stale

[10:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=653s) that can conflict with the authoritative

[10:54](https://www.youtube.com/watch?v=hD9-V56FNRI&t=654s) data or corrupt future actions that the

[10:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=657s) agent might perform.

[10:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=659s) I like to classify it into two different

[11:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=662s) types of uh memory that the agent has.

[11:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=665s) First is the short-term memory which is

[11:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=667s) the jet thread uh that the agent has uh

[11:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=671s) the which is tied to a single execution

[11:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=673s) thread and the second is the long-term

[11:15](https://www.youtube.com/watch?v=hD9-V56FNRI&t=675s) memory. It can be project files uh

[11:18](https://www.youtube.com/watch?v=hD9-V56FNRI&t=678s) system prompts uh databases that it

[11:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=681s) interacts with the cache layer and so

[11:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=683s) on.

[11:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=685s) It is important to uh to to decide what

[11:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=689s) will be the source of truth when these

[11:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=692s) uh different data sources have

[11:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=695s) conflicting information and we should

[11:38](https://www.youtube.com/watch?v=hD9-V56FNRI&t=698s) ideally treat memory as a cache which uh

[11:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=703s) can be invalidated which can have

[11:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=706s) provenence attached to it. So for

[11:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=709s) instance whenever a data store or a

[11:51](https://www.youtube.com/watch?v=hD9-V56FNRI&t=711s) database is updated or the source of

[11:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=713s) truth is updated we in we invalidate the

[11:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=716s) context or the memory that the agent has

[11:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=719s) to ensure that it is not making actions

[12:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=721s) based on the uh incorrect or stale data.

[12:08](https://www.youtube.com/watch?v=hD9-V56FNRI&t=728s) So

[12:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=730s) usually these agents perform multi-step

[12:12](https://www.youtube.com/watch?v=hD9-V56FNRI&t=732s) actions and uh the agent can succeed on

[12:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=737s) uh on uh on the first couple of steps

[12:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=740s) and then it fail. Uh it is important to

[12:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=743s) reverse the entire transaction that was

[12:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=746s) performed and these can uh then can

[12:29](https://www.youtube.com/watch?v=hD9-V56FNRI&t=749s) cross system boundaries. So for

[12:31](https://www.youtube.com/watch?v=hD9-V56FNRI&t=751s) instance, an agent can update an

[12:34](https://www.youtube.com/watch?v=hD9-V56FNRI&t=754s) internal ticket uh send an email to a

[12:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=757s) customer and fail to update the CRM. We

[12:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=761s) need to figure out what is the uh

[12:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=765s) correct compensation operation when it

[12:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=768s) when it hits that failure.

[12:51](https://www.youtube.com/watch?v=hD9-V56FNRI&t=771s) So for instance uh as I mentioned

[12:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=773s) earlier that uh it improperly uh it

[12:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=779s) improperly sends an incorrect email to

[13:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=781s) the customer. It is important that the

[13:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=783s) compensation operation is defined for

[13:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=785s) the AI agent to ensure that it is

[13:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=787s) sending an uh uh an apology email to the

[13:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=791s) customer or any or or any email or an

[13:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=794s) email that is correcting that mistake.

[13:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=799s) So uh

[13:22](https://www.youtube.com/watch?v=hD9-V56FNRI&t=802s) a the AI agent basically runs in a loop

[13:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=805s) and uh whenever uh like it can it can do

[13:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=810s) multiple calls. It can it can have a

[13:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=813s) retry uh retry loop that it can run

[13:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=817s) based uh whenever it fails. So it is

[13:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=820s) important to have uh circuit breakers

[13:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=823s) whenever it is making making external

[13:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=825s) calls uh to ensure that the uh that the

[13:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=829s) that it is not uh burdening the

[13:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=832s) downstream system. Uh for instance if a

[13:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=836s) downstream is unhealthy there should be

[13:57](https://www.youtube.com/watch?v=hD9-V56FNRI&t=837s) system break uh circuit breakers in

[13:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=839s) place that prevents AI agents to call

[14:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=841s) call that dependency. Moreover, it also

[14:04](https://www.youtube.com/watch?v=hD9-V56FNRI&t=844s) prevents cascading failures when for

[14:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=846s) instance the downstream dependency is uh

[14:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=850s) unhealthy or uh is saturated.

[14:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=854s) It is also important to assign rate

[14:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=856s) limits and budgets. Uh an agent can uh

[14:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=861s) go over uh can run your cost uh if it's

[14:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=864s) not assigned proper budgets and rate

[14:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=866s) limits. it will uh keep retrying and try

[14:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=870s) try to uh try to solve the problem that

[14:33](https://www.youtube.com/watch?v=hD9-V56FNRI&t=873s) if it if it's facing. So it is important

[14:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=876s) that it is uh that we have uh set up max

[14:39](https://www.youtube.com/watch?v=hD9-V56FNRI&t=879s) turns, max parallelism, max spend uh to

[14:42](https://www.youtube.com/watch?v=hD9-V56FNRI&t=882s) ensure that the model is not uh uh not

[14:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=886s) uh crossing the uh the budget boundary

[14:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=889s) that we have set.

[14:52](https://www.youtube.com/watch?v=hD9-V56FNRI&t=892s) Moreover, uh ideally uh usually whenever

[14:58](https://www.youtube.com/watch?v=hD9-V56FNRI&t=898s) we are building AI agents, uh we usually

[15:02](https://www.youtube.com/watch?v=hD9-V56FNRI&t=902s) try to give all the permissions that it

[15:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=903s) can have to ensure that it has all that

[15:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=906s) it can perform perform the task that we

[15:08](https://www.youtube.com/watch?v=hD9-V56FNRI&t=908s) have. That's the that's the uh first uh

[15:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=914s) uh thing that we have that that's the

[15:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=916s) first step that we take usually that to

[15:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=919s) give the AI agents all the uh cred all

[15:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=921s) the uh privileges to perform any actions

[15:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=925s) like for instance if it's interacting

[15:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=927s) with the database we just give it all uh

[15:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=930s) the readr access to the entire table.

[15:32](https://www.youtube.com/watch?v=hD9-V56FNRI&t=932s) However,

[15:34](https://www.youtube.com/watch?v=hD9-V56FNRI&t=934s) uh it is important to give scoped

[15:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=937s) credentials to it. There should be

[15:39](https://www.youtube.com/watch?v=hD9-V56FNRI&t=939s) separate read and write permissions and

[15:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=941s) there should be allow list for the tools

[15:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=943s) that it can call. A harmless model can

[15:47](https://www.youtube.com/watch?v=hD9-V56FNRI&t=947s) become dangerous when it can perform

[15:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=949s) unsafe operations. Moreover, uh a human

[15:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=953s) approval shouldn't be tied uh to a

[15:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=956s) blanket approval. It should be tied to

[15:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=959s) uh action, timestamp, actor and

[16:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=963s) expiration. So for instance, if a user

[16:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=966s) has given uh an approval to approve a

[16:10](https://www.youtube.com/watch?v=hD9-V56FNRI&t=970s) $30 refund, it shouldn't turn into a

[16:13](https://www.youtube.com/watch?v=hD9-V56FNRI&t=973s) subsequent approval for $300 refund.

[16:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=977s) It is important that whenever an

[16:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=980s) approval is given, it should be tied to

[16:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=984s) the particular parameters that it was uh

[16:28](https://www.youtube.com/watch?v=hD9-V56FNRI&t=988s) asked for.

[16:31](https://www.youtube.com/watch?v=hD9-V56FNRI&t=991s) So, uh observability is an important

[16:36](https://www.youtube.com/watch?v=hD9-V56FNRI&t=996s) requirement when building AI agents

[16:38](https://www.youtube.com/watch?v=hD9-V56FNRI&t=998s) because uh

[16:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1001s) and logs are not enough. Teams need to

[16:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1003s) reconstruct when an agent failed, what

[16:45](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1005s) happened, what information was was it

[16:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1008s) reacting to and why it failed. And logs

[16:51](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1011s) alone are not enough to uh for an agent

[16:55](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1015s) to uh for teams to determine that.

[16:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1019s) It is important to trace the model that

[17:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1021s) was called, the prompt that was uh that

[17:04](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1024s) was uh given to it and uh also the tool

[17:08](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1028s) calls that were made uh the request uh

[17:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1031s) that was made, the response from the

[17:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1034s) tool, the errors that it got, the

[17:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1036s) retrieved context, what the agent was uh

[17:19](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1039s) was the the retrieved information that

[17:22](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1042s) the agent was reacting to, the rights

[17:24](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1044s) that it made, and the approvals that it

[17:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1046s) got and so on.

[17:31](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1051s) So uh I would like to uh end with uh the

[17:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1057s) idea that yes model capability matters.

[17:40](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1060s) Having good models uh improves the uh

[17:44](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1064s) likelihood of it making uh correct

[17:47](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1067s) operations. Smarter models reduce

[17:49](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1069s) mistakes. It uh it uh improves the

[17:54](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1074s) capability that the model has. However,

[17:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1076s) it cannot eliminate network failures,

[17:59](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1079s) stale data or adversarial input. It is

[18:03](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1083s) important when building this

[18:04](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1084s) architecture,

[18:06](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1086s) we also reason about can we bound,

[18:09](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1089s) observe and recover from actions

[18:12](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1092s) performed by the AI agent. It is

[18:14](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1094s) important to have tool contracts in

[18:16](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1096s) place to ensure that uh it is only

[18:21](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1101s) allowed to make uh operations that it is

[18:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1103s) uh given that it is provided the

[18:25](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1105s) contract and the contracts are clearly

[18:27](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1107s) establishing the request and response uh

[18:30](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1110s) response uh response types uh the schema

[18:35](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1115s) and all these tools have item potency

[18:37](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1117s) baked into it. so that uh when repeated

[18:41](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1121s) requests are sent in uh it is not

[18:43](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1123s) causing unsafe operations to be retried.

[18:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1126s) Moreover, there should be source of

[18:48](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1128s) truth decisions made uh when there are

[18:50](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1130s) conflict conflicting uh memory states.

[18:53](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1133s) It is important uh for the agent to

[18:56](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1136s) realize this is the source of data that

[18:58](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1138s) it should rely on and we should have re

[19:01](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1141s) retry policies uh like rate limits set

[19:05](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1145s) in to ensure that the agent is not uh

[19:07](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1147s) retrying uh ext uh aggressively.

[19:11](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1151s) Moreover, uh permissions should be set

[19:15](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1155s) up. There should be traces and recovery

[19:17](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1157s) paths. So

[19:20](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1160s) when building AI agents, we should also

[19:23](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1163s) ask what the system lets it do when it

[19:26](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1166s) is wrong.

[19:28](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1168s) Thank you.

[19:46](https://www.youtube.com/watch?v=hD9-V56FNRI&t=1186s) >> [music]
