AI Agents Are Just Distributed Systems Now — Salman Munaf, TikTok

[music]

Hello everyone. Good good afternoon. Uh

today uh I will be talking about AI

agents are also distributed systems.

So as uh the models have started to

become more complex. Initially the uh

LLM models were just text in text out

without performing any actions and uh

the

effect that they can produce was just a

wrong model output. However, with

now the capability of agent the agent

the rise in agent capabilities where the

systems can now talk to external systems

uh it has turned into a distributed

systems and it is important to

incorporate distributed systems thinking

and concepts when building AI agents. So

I will be going over uh that uh in this

talk.

So you guys might have uh heard about

incidents being caused by a AI agents.

Uh for instance, the replicate AI agent

deleting a production incident

production database or Air Canada

chatbot basically making an uh an

incorrect refund. And both of these uh

incidents or a lot of these incidents

could have been prevented uh by good

systems thinking when building these uh

AI agents. So for instance for the

replet a uh uh uh incident we could have

good uh

uh we could have robust backups. We

could have scoped authority. we should

we shouldn't ideally have uh allow AI

agents to delete production databases.

Uh moreover for Air Canada chatbot

it would have been uh a good idea to

have uh authoritative source of truth

retrieval so that it's not making uh uh

decisions based on stale or incorrect

policies.

So let's uh go over the transition from

chatbot to production system. Uh so

uh initially when we were in the uh in

the uh age where LLMs were just chat

bots uh we had prompt in and we were

outputting text there were no side

effects the agent was not interacting

with any other system.

However, uh due to agentic uh in the

agentic era in the agentic revolution,

now those agents uh by ingesting prompt

can uh run an agent loop, call external

services, call tools and also perform

state changes. The architectural

boundary now has moved uh way beyond an

LLM model. And the difference is that it

can now cause side effects in the

outside world. So when basic when

building AI agents, it is important to

recognize the external systems that it

is talking to, [clears throat] the

states that uh it is interacting with

and what credentials does it have and

the actions that it can perform.

uh I ideally like to think about it as

uh uh AI agents as basically having a

probabilistic coordinator.

In distributed systems as well, we used

to have services which were coordinating

uh multi-step workflows. However, they

were deterministic in nature. But in the

case of AI agent, the AI acts as a

probabilistic coordinator. The amount of

action, the kind of actions that it can

take can vary quite a lot. It is not

just a decision tree that uh we

typically in traditional systems would

have mapped out

and those uh actions can have severe

consequences uh ba uh if they are not

confined by our determinist by having

deterministic controls in place. So it

is important to ensure uh that we have

deterministic controls in place to

ensure that agent or the AI agent is not

performing any uh

any actions that might be uh uh

problematic.

So uh

let's uh discuss the how a typical agent

loop might look like. So at first it

might uh do some planning. Then based on

that plan it will it will perform an

action and it will then observe the

results of those actions and uh it might

persist that into some d some data store

and then decide what to do next.

Each step in this loop is basically

crossing a a boundary. During planning,

it can interact with data sources to

retrieve some data. Uh during action, it

can call external APIs, tools, uh

databases and perform any actions.

During observation phase, it can perform

it can get partial results and basically

plan or make subsequent actions based on

those partial results. It can persist

incorrect data or uh and uh when

deciding it might also

decide to uh perform an incorrect action

or uh worse it can also do a retry

storm.

So it is very important when building an

agent loop to persist every step of the

process. Whatever actions the agent is

doing, whatever context it is

retrieving, it is important to uh

persist that so that if anything fails,

the agent is able to recognize where it

failed and it can perform uh a

reversible action. Uh it can perform

undo operations. Similarly, there should

be explicit transactions uh identified

for each step. So for instance, if an

agent is making a call, if it fails,

what it should do? What should be the

transaction to compensate for a uh for a

irreversible or unsafe operation? For

instance, if an agent makes sends an

email to a a wrong email to a customer,

what should it do to compensate for

that?

So, uh tool calls are just wrappers

around uh external external APIs,

databases, cues, uh and so on.

And uh with uh when calling the when

making these remote calls, there are

some failures that you incorporate uh

such as network delays, timeouts, uh you

can make duplicate requests or worse the

server side request uh might succeed.

However, the client however the client

might be reported an error.

We have we have seen uh instances where

uh a data by base might have written the

data. However, due to some other errors,

the server might have reported uh uh to

us the error and

uh with humans in the loop we can make

we can basically perform correct

corrective actions based on uh by seeing

uh the database and actual source of

truth. But in agent's case, we need to

ensure that we have uh we have proper

guardrails in place. So for instance,

an agent calls refund customer uh tool

call which basically performs a refund

to the customer. The request times out

uh that did the refund happen or not?

What will the agent uh infer from that?

Would it retry uh refunding to the

customer? you basically the the timeout

does not actually mean that there a

failure had occurred. It means unknown.

And it is important to have uh when

designing these tools, it is important

to have request ids, item potency keys

so that when making duplicate requests,

they are not causing duplicate side

effects. uh and the system can always do

a status lookup like what the previous

request was and what was the status of

that so that it is not making side

effect it is not making side effects

with duplicate a with duplicate

requests.

So [clears throat] AI agents when they

whenever they uh uh whenever they uh

they face failures they retry the their

first uh action is to perform retries.

So it is really important to have item

potency baked in. uh it if a same

request is coming in to an external API

or the tool it should recognize that

this is a duplicate request and ensure

that no side effects are being take are

taking place. Moreover, uh we should

also prevent uh AI agents to perform

retry storms to external APIs because

this can cause cascading failures. uh we

we should have max turns budget spend

and max parallel calls to prevent uh to

uh to ensure that the fan out is not

that large. Moreover, we should have

exponential back back off in place to

ensure that uh the downstream

dependencies are not being uh burdened

and we should also have compensation uh

operations in place for uh operations

that uh that that can have side effects.

Uh

so uh

a lot of uh

teams when building AI agents think of

AI agent context as just a AI cont the

context that uh the AI agent has as uh

as just a context. However, when that

context can influence an action, it's a

state and that state can become stale

that can conflict with the authoritative

data or corrupt future actions that the

agent might perform.

I like to classify it into two different

types of uh memory that the agent has.

First is the short-term memory which is

the jet thread uh that the agent has uh

the which is tied to a single execution

thread and the second is the long-term

memory. It can be project files uh

system prompts uh databases that it

interacts with the cache layer and so

on.

It is important to uh to to decide what

will be the source of truth when these

uh different data sources have

conflicting information and we should

ideally treat memory as a cache which uh

can be invalidated which can have

provenence attached to it. So for

instance whenever a data store or a

database is updated or the source of

truth is updated we in we invalidate the

context or the memory that the agent has

to ensure that it is not making actions

based on the uh incorrect or stale data.

So

usually these agents perform multi-step

actions and uh the agent can succeed on

uh on uh on the first couple of steps

and then it fail. Uh it is important to

reverse the entire transaction that was

performed and these can uh then can

cross system boundaries. So for

instance, an agent can update an

internal ticket uh send an email to a

customer and fail to update the CRM. We

need to figure out what is the uh

correct compensation operation when it

when it hits that failure.

So for instance uh as I mentioned

earlier that uh it improperly uh it

improperly sends an incorrect email to

the customer. It is important that the

compensation operation is defined for

the AI agent to ensure that it is

sending an uh uh an apology email to the

customer or any or or any email or an

email that is correcting that mistake.

So uh

a the AI agent basically runs in a loop

and uh whenever uh like it can it can do

multiple calls. It can it can have a

retry uh retry loop that it can run

based uh whenever it fails. So it is

important to have uh circuit breakers

whenever it is making making external

calls uh to ensure that the uh that the

that it is not uh burdening the

downstream system. Uh for instance if a

downstream is unhealthy there should be

system break uh circuit breakers in

place that prevents AI agents to call

call that dependency. Moreover, it also

prevents cascading failures when for

instance the downstream dependency is uh

unhealthy or uh is saturated.

It is also important to assign rate

limits and budgets. Uh an agent can uh

go over uh can run your cost uh if it's

not assigned proper budgets and rate

limits. it will uh keep retrying and try

try to uh try to solve the problem that

if it if it's facing. So it is important

that it is uh that we have uh set up max

turns, max parallelism, max spend uh to

ensure that the model is not uh uh not

uh crossing the uh the budget boundary

that we have set.

Moreover, uh ideally uh usually whenever

we are building AI agents, uh we usually

try to give all the permissions that it

can have to ensure that it has all that

it can perform perform the task that we

have. That's the that's the uh first uh

uh thing that we have that that's the

first step that we take usually that to

give the AI agents all the uh cred all

the uh privileges to perform any actions

like for instance if it's interacting

with the database we just give it all uh

the readr access to the entire table.

However,

uh it is important to give scoped

credentials to it. There should be

separate read and write permissions and

there should be allow list for the tools

that it can call. A harmless model can

become dangerous when it can perform

unsafe operations. Moreover, uh a human

approval shouldn't be tied uh to a

blanket approval. It should be tied to

uh action, timestamp, actor and

expiration. So for instance, if a user

has given uh an approval to approve a

$30 refund, it shouldn't turn into a

subsequent approval for $300 refund.

It is important that whenever an

approval is given, it should be tied to

the particular parameters that it was uh

asked for.

So, uh observability is an important

requirement when building AI agents

because uh

and logs are not enough. Teams need to

reconstruct when an agent failed, what

happened, what information was was it

reacting to and why it failed. And logs

alone are not enough to uh for an agent

to uh for teams to determine that.

It is important to trace the model that

was called, the prompt that was uh that

was uh given to it and uh also the tool

calls that were made uh the request uh

that was made, the response from the

tool, the errors that it got, the

retrieved context, what the agent was uh

was the the retrieved information that

the agent was reacting to, the rights

that it made, and the approvals that it

got and so on.

So uh I would like to uh end with uh the

idea that yes model capability matters.

Having good models uh improves the uh

likelihood of it making uh correct

operations. Smarter models reduce

mistakes. It uh it uh improves the

capability that the model has. However,

it cannot eliminate network failures,

stale data or adversarial input. It is

important when building this

architecture,

we also reason about can we bound,

observe and recover from actions

performed by the AI agent. It is

important to have tool contracts in

place to ensure that uh it is only

allowed to make uh operations that it is

uh given that it is provided the

contract and the contracts are clearly

establishing the request and response uh

response uh response types uh the schema

and all these tools have item potency

baked into it. so that uh when repeated

requests are sent in uh it is not

causing unsafe operations to be retried.

Moreover, there should be source of

truth decisions made uh when there are

conflict conflicting uh memory states.

It is important uh for the agent to

realize this is the source of data that

it should rely on and we should have re

retry policies uh like rate limits set

in to ensure that the agent is not uh

retrying uh ext uh aggressively.

Moreover, uh permissions should be set

up. There should be traces and recovery

paths. So

when building AI agents, we should also

ask what the system lets it do when it

is wrong.

Thank you.

>> [music]