AI Engineer World's Fair 2025
3 ingredients for building reliable enterprise agents
Read the talk
Three ingredients for reliable enterprise agents
Enterprise agents become useful when they do valuable work predictably, make mistakes recoverable, and keep people involved at the decisions that matter.
From a talk by Harrison Chase
Before you start: Familiarity with LLM tool calls, retrieval-augmented generation (RAG), and Git branches and pull requests will help with the examples.
An agent for every task—and a human supervising
What would it take to run an enterprise full of task-specific agents, with people coordinating their work as managers and supervisors? The challenge applies both to developers building agents inside a company and to vendors selling agents into one. A compelling vision of delegated work still has to become something an organization will adopt.
Why do some agents succeed in that setting while others fail? Harrison Chase credits a conversation with Assaf, whom he introduces as Monday’s head of AI and the author of GPT Researcher, for many of the ideas behind his answer. The starting point is the economics of using an agent, before choosing its architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The production threshold
An agent becomes more attractive as the value when it succeeds increases. It also becomes more attractive as its probability of success increases. The third ingredient moves in the opposite direction: a high cost when it fails discourages adoption. These are separate levers. A valuable task can justify imperfect performance when mistakes are cheap; a costly mistake can make an otherwise capable agent difficult to approve.
Chase expresses the relationship as an expected-value threshold. Let p be the probability of success, V the value of a successful result, L the loss from failure, and C the cost of running the agent:
The same reasoning applies beyond enterprises. The practical question is how product design and engineering can move these terms enough to make deployment worthwhile.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Increase the value of the work
The most direct lever is task selection. Chase points to Harvey in legal work and to research, investment research, and summarization in finance. People already pay substantial amounts for lawyers and professional research. That existing willingness to pay makes the value of a successful result easier to identify.
But increasing value does not require changing industries. It can mean changing how much work the agent completes per interaction. Chase describes older RAG and question-answering interfaces as aiming to answer within five seconds. Deep research instead spends an extended period investigating a question. In coding, Cursor’s inline autocomplete and chat illustrate short interactions; background coding agents represent a larger unit of delegated work. Chase informally counts seven examples appearing in the preceding three weeks, with agents running for hours at a time.
The coding slide makes the contrast concrete with the labels “Cursor: 30 seconds” and “Claude Code: 2 hours.” These illustrate different work horizons, rather than a controlled speed comparison. The product opportunity is to let an agent do more substantial work autonomously in the background. That requires changing the UI and interaction pattern, not merely replacing the model behind a quick-response chat box.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put required behavior in control flow
Increasing the probability of success starts with the gap between a prototype and a production system. A run that works once can make a good demonstration without establishing reliability. Some applications tolerate exploratory behavior whose steps are difficult to predict. Enterprises often need tighter control: a particular step must follow another, regardless of what the model would otherwise choose.
In Chase’s illustrative example, prompting an agent to perform A after B might preserve that order about 90% of the time. Encoding the sequence in deterministic control flow makes the ordering a property of the program. It does not guarantee that either step produces a correct result; it removes one decision from the model’s discretion. More of the agent can become deterministic wherever predictability is a requirement.
Anthropic’s Building effective agents distinguishes workflows from agents. Chase’s preferred framing is workflows and agents: tool-calling loops can coexist with fixed sequences. If agent A always hands its result to agent B, the outer structure is a workflow even when each agent makes its own internal decisions. In TypeScript, that boundary can be as small as this:
typescript
type Agent<Input, Output> = (input: Input) => Promise<Output>;
async function runInOrder<Input, Intermediate, Output>(
input: Input,
agentA: Agent<Input, Intermediate>,
agentB: Agent<Intermediate, Output>,
): Promise<Output> {
const resultA = await agentA(input);
return agentB(resultA);
}
Here, agentB receives the completed result of agentA; if A throws, B is not invoked. The agents can remain flexible internally while their handoff stays explicit.
LangGraph is Chase’s example of a framework designed around this spectrum. The useful question is where an application needs deterministic structure and where it benefits from model-directed execution. Different applications need different positions on that spectrum.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make performance legible to reviewers
Actual reliability is only part of getting an agent approved. Stakeholders may also have substantial uncertainty about how it behaves. Improving the agent and reducing that uncertainty are related but distinct jobs. Observability and evaluations help with both: they support engineering work while making performance understandable to the people deciding whether deployment is acceptable.
Chase describes LangSmith as a tool originally built so developers could inspect their agents, which also became useful for communicating with external stakeholders. Step-level traces reveal where a run succeeds, where it fails, and which patterns recur. A reviewer can see several LLM calls and the steps connecting them instead of imagining an opaque, single-shot answer. Evaluations and benchmarks add evidence about how the system performs.
Chase recounts a customer who first used LangSmith to build an agent, then brought the same view into a production review meeting. Showing the panel what happened inside the system helped reduce perceived risk, and the meeting finished early. The anecdote illustrates a second audience for developer tooling: the people responsible for approving its use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce the cost of being wrong
Reviewers also worry about consequences: an agent could damage a brand or give something away for free. Chase argues that those fears can dominate the adoption decision. Two product mechanisms address different parts of the risk: make changes reversible, and place a human approval boundary before consequential actions.
Code offers a natural example of reversibility. Chase points to a Replit Agent screenshot while discussing generated diffs and pull requests, then explains how returning to a previous commit can undo code changes. In his historical account, Replit saved each file change as a new commit. Today, Replit’s checkpoint documentation describes checkpoints at milestones and other strategic moments, rather than every edit. The enduring mechanism is a recoverable earlier state; it does not imply that every external effect can be rolled back.
Approval and reversal are separate protections. Opening a pull request instead of merging directly into main lets a person review the proposed change before it reaches the main branch. Reversal provides a recovery path after a change; approval provides a chance to stop it beforehand. Together they change the consequences of an agent mistake—and how an enterprise evaluates those consequences.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Clarify intent, then produce something reviewable
Deep research shows how human involvement can increase value as well as limit risk. Before investigating, the system asks follow-up questions and calibrates what the user wants. That exchange improves the chance that the eventual report addresses the right problem. At the other end, the output is a report for the user to read and act on—not an automatically published blog post or an email sent to clients. Research is delegated; publication remains a separate decision.
Chase describes a similar pattern with Claude Code: clarifying questions help establish intent, and a separate branch plus a pull request preserves a review boundary. The workflow does not require a commit for every individual file change. It requires keeping proposed work separate from the main branch until someone decides to accept it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale useful work beyond the chat box
Once an agent has positive expected value, the next opportunity is to repeat that useful work at greater scale. Ambient agents run in the background and start in response to events, rather than requiring a person to initiate every run. Chase contrasts managing one or two chat boxes with potentially hundreds of agents working in the background. The interaction changes from one-to-one to one-to-many. His description of unlimited concurrency concerns that interaction model, not measured infrastructure capacity.
The latency expectation changes too. In chat, someone is waiting for a response. With event-triggered background work, there is less pressure to return an immediate answer. That creates room for longer sequences of reasoning and tool use: changing a line of code can become changing a whole file or creating a repository. More time is useful because it permits a larger body of work, rather than merely a slower answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Background execution still needs oversight
Ambient does not mean fully autonomous. Running in the background does not remove the need for an interface through which people can supervise the work. Otherwise, the same fear returns: the agent might act without anyone understanding or controlling the consequences.
Chase describes several intervention patterns:
- Approve or reject: Require explicit permission before selected tools can run.
- Edit a tool call: Correct the proposed action in the UI instead of discarding the entire run.
- Answer questions: Let the agent ask for clarification or missing information when it gets stuck.
- Revisit an earlier step: After a run, return to a prior point and resume with different guidance. Chase calls this time travel, or human on the loop; his example is returning to step ten of a hundred after identifying a mistake there.
These controls place human judgment at particular boundaries without requiring a person to direct every internal operation.
There is also an intermediate stage between chat and ambient agents. Under Chase’s stricter definition, deep research and Claude Code are not ambient because humans still initiate them. He likes Factory’s term, async coding agents, for work that starts with a human request and continues in the background. Initial interaction calibrates the task; execution then proceeds asynchronously, with opportunities to ask for help.
Chase therefore adds a conceptual middle column to the chat-versus-ambient distinction:
| Interaction | Starts with | Work pattern |
|---|---|---|
| Chat | Human message | Immediate exchange |
| Sync-to-async | Human request and calibration | Background execution |
| Ambient | External event | Background execution with selective oversight |
The key distinction is who or what starts the work. Asynchronous execution alone does not make an agent ambient.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An inbox for pending agent actions
Agent Inbox gives this supervision model a concrete interface. It surfaces actions that need human attention so the user can approve, reject, or leave feedback. Those are interaction descriptions: the inspected implementation names its responses accept, edit, response, and ignore. A pending action is something the agent wants to do, not evidence that it has already done it.
Email is a natural application. An incoming message is an event that can initiate work without a user opening a chat. Many incoming messages can create concurrent tasks, while outgoing emails or calendar invitations remain subject to approval according to the user’s comfort level. Event-driven intake and controlled external actions can therefore coexist.
Chase built and uses an email agent to test these patterns internally. He closes the presentation by offering its open-source implementation through an on-screen QR code. The example brings the architecture together: events start the work, the agent prepares actions, and an inbox provides a place for human decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What generalizes beyond coding agents
In the Q&A, an audience member asks whether coding agents attract funding because their work is measurable and reversible. Chase answers by separating two advantages of code, without establishing the question’s premise that only coding agents receive funding.
The first advantage is verifiability. Code can be run and checked for compilation, although compilation alone does not establish correctness. Math also offers results that can be checked. Essay quality is more ambiguous: deciding whether an essay is correct is a less sharply defined task. Chase connects this difference to training—verifiable outputs make it easier to bootstrap training data, which contributes to stronger coding models and, in turn, stronger coding agents.
The second advantage is a product pattern that travels more easily across domains: first drafts. Code naturally supports commits, drafts, and previews, but legal documents and essays also have familiar draft-and-review workflows. An agent can complete a substantial amount of work before presenting an artifact for human judgment.
Putting a human in the loop at every tiny step erodes the value of delegation. The design challenge is to let the agent do substantial work while keeping people involved at key points. A first draft supplies exactly that boundary: enough completed work to be useful, with a clear decision before acceptance or use. That pattern generalizes across legal work, writing, and code more readily than the verifiability that helps train coding models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's guide to workflows, agents, and combining simple architectural patterns.
Framework for stateful workflows and agents with persistence and human oversight.
An inbox interface for reviewing agent actions, editing arguments, and supplying feedback.
An open-source agent for conducting research using language models.
Further reading
- Introducing ambient agentsArticle
Harrison Chase explains event-triggered agents, selective human oversight, and an email-assistant example.
Email-assistant implementation with triage, drafting, calendar configuration, and Agent Inbox integration.
Updates since the talk
Current checkpoint creation and rollback behavior, including database restoration limits.
Read the complete timestamped transcript
- 0:00
[upbeat music] I wanna talk today a little bit about trying to build reliable agents in the enterprise.
- 0:19
This is something we work with a bunch of people for, both people building as developers inside of an enterprise looking to build agents for, for their company, but also people who are looking to build solutions and, and, and bring them and sell them into enterprises.
- 0:34
Um, it, it-- and so I wanted to talk a little bit about some of what we see kind of being the, the success tips and tricks for making this happen.
- 0:42
So, uh, the, the vision of the future that, that I and other people I think have a, have a similar view of for agents is that there'll be a lot of them.
- 0:50
They'll be running around the enterprise doing different things. They'll, they'll be, you know, an agent for every different task. We'll be coordinating with them. We'll be kind of like a manager, a supervisor.
- 0:58
And, and so, and so how do we get to that vision, and what, uh, what, what parts of this will kind of like arrive before, before the others?
- 1:10
Um, it-- and, and so I was thinking about this question, what makes some agents kind of like succeed in the enterprise and, and, and some fail? And I was chatting, uh, with, with my friend Assaf.
- 1:19
He's the head of AI at Monday. He also wrote GPT-Researcher. It's a great open source package. Uh, I was chatting with him a few weeks ago, um, and, uh, a lot of the ideas here are borrowed from that conversation.
- 1:30
He'll probably write a, a blog post about this, uh, with a slightly different framing, which I would encourage everyone to, to check out. So I just wanna give him a massive shout-out, and if you have the opportunity to, to chat with him, you should definitely take that opportunity.
- 1:43
Um, thinking about it from like first principles, like what makes agents successful in the enterprise? Uh, it'll make it successful, it will make it more likely to be adopted the greater the value of the agent if it's right.
- 1:57
These, these probably aren't gonna sound kind of like earth-shattering, but hopefully we'll get to some interesting points. If-- the, the more value it provides when it's right, the more likely it will be to be adopted.
- 2:06
The more likely it is to have success, the more likely it will be to be adopted. And then the cost if it's wrong. If it-- if there's big costs when it's wrong, then it will be less likely to be adopted.
- 2:17
So I think these are three kind of like ingredients which are pretty simple and pretty basic, but I think provide an interesting kind of like first principles approach for how to think about building agents and what types of agents kind of like find success.
- 2:29
And, and you know, I say in the enterprise here, but I also think this applies just generally within, within, uh, kind of like society. Um, if, if we wanna try to put this into a fun little equation, you know, we can multiply the, the probability that something succeeds times the value that you get when it succeeds, and
- 2:44
then, and then do the opposite for the cost when it's wrong. And of course, like this needs to be greater than, than the cost of running the agent for you to wanna put it into production.
- 2:52
And so, uh, yeah, fun, fun little kind of like stats/math formula. So how can we build agents that score higher on this? 'Cause this is-- this hasn't been anything kind of like earth-shattering so far.
- 3:04
Hopefully, we'll get to some fun insights when we talk about how to make that, make that equation kind of like go up.
- 3:11
So how can we increase the, the, the value of, or uh, of, of things, um, when they go right, and, and what types of agents have higher value? So pa- so part of this is, uh, choosing kind of like problems where there is, there is really high kind of like value.
- 3:27
So a lot of the agents that have been successful so far, Harvey in the legal space is one of them. Um, in the finance space, we see stuff around research and summarization.
- 3:37
These are high value work tasks. People pay a lot of money for, for lawyers, uh, and, and, and for, and for research and investment research. And so these are examples of what I would say kind of like high value tasks are.
- 3:51
There's other ways to kind of like improve the value of what you're working on besides just switching kind of like the vertical completely, and I think we're starting to see some of this, especially more recently.
- 4:00
So if we think about RAG or if we think about kind of like existing question or old, older school question answering solutions, they would often respond kind of like quickly, ideally within five seconds, and give you a quick answer.
- 4:11
And we're starting to see a trend towards things like deep research, which go and run for an extended period of time. We're seeing the same with code. We start with Cursor.
- 4:19
It has kind of like inline auto-complete, maybe some chat question and answering there. In the past like three weeks, there's been, what? Seven different examples of these ambient agents that run in the background for like hours at a time.
- 4:31
And I think this speaks to ways that people are trying to get their agents to provide more value. They're getting them to do more work. Um, pr- pretty, pretty basic, but I do think that like as we have-- if you think about this future of agents working and what that means, that doesn't mean a co-pilot.
- 4:46
That means something working more autonomously in the background, doing more amounts of work. So besides kind of like focusing on areas or verticals that provide value, I think you can also absolutely reshift the, the UI, UX, the interaction pattern of what you're building to be kind of like more long-term and do more kind of like substantial patterns
- 5:08
of work. Let's talk about now the probability of success. How do we make this go up? So the, the, the, the-- there's a few-- there's two different aspects I wanna talk about here.
- 5:20
One, I think is about the reliability of agents. If you've built, uh, agents before, it's easy to get something that works in a prototype. It runs once, great. You can make a video, put it on Twitter, but it's hard to make it work reliably, put it in, in, in production.
- 5:34
And I think the, the core thing that we've seen... And, and, and by the way, for some parts of, of, of-- so for some types of agents, that's totally fine.
- 5:43
Um, you can have agents that, that run for a while and, and kind of like you don't know what they do and it-- and that, and that's totally fine.
- 5:51
Especially in the enterprise, we see oftentimes that people want more predictability, more control over what steps actually happen inside the agents. Maybe they always wanna do step A after step B.
- 6:02
And so if you prompt an agent to do that- Great. It might do that like 90% of the time. You don't know what the LLM will do. If you put that in a deterministic kind of like workflow or code, then it will always do that.
- 6:13
And so especially in the enterprise, we see that there are workflow-like things where you need more controllability, more predictability than you get by just prompting. And so what we've seen is more-- the, the solution for this is basically make more and more of your agent deterministic.
- 6:30
There's this concept of kind of like workflows versus agents. Anthropic wrote a great blog post on this that I'd encourage you to check out. Um, I, I would argue that instead of workflows versus agents, it's, it's oftentimes workflows and agents.
- 6:42
Uh, we see that parts of an agentic system are sometimes looping, calling a tool, and sometimes they're just doing A after B after C. An example of this is when you think about multi-agent architectures.
- 6:53
If you think about an architecture that has agent A, and then after agent A finishes, you always call agent B. Is, is that a workflow? Is that an agent?
- 7:01
It's, it's this middle ground. And so as we think about building tools for this, this, this future, uh, one of the, one of the things that we've released is LangGraph.
- 7:10
LangGraph is an agent framework. It's very different from other agent frameworks where it really leans into this spectrum of workflows and agents and allows you to be wherever, wherever is best for your application on that, on that kind of like curve.
- 7:24
And where, where on that curve is best totally depends on kind of like the application that you're building.
- 7:30
There, there, there's another thing, um, that, that is different from just building and changing the agent, and I think there's, uh, there's oftentimes really high error bars that people have when they think about how likely an agent is to work.
- 7:43
I think this technology's new. Um, when, when, uh, trying to get something built or approved or put into production inside an enterprise, I think there's a lot of, um, uncertainty and, and, and, and fear around this.
- 7:56
Um, and, and, and I think that relates to this fundamental kind of like
- 8:01
uncertainty around how this agent is kind of like performing. And so besides just like making it better, a really important thing that we see to do inside the enterprise, whether you're, you're, you're bringing a third-party agent and selling it as a service, or whether you're building inside, uh, uh, the enterprise yourself, is to work to kind of
- 8:20
like reduce the, the, the way that people see the error bars of how this agent performs. Um, so what I mean by that specifically is that this is where observability and evals actually plays a slightly different role than, than we would maybe think or we would maybe intend.
- 8:35
So w- we, we have a, we have an observability and eval solution called LangSmith. We built it for developers so that they could see what's going on inside their agent.
- 8:43
It's also proved really, really valuable for communicating to ex-external shareholders what's going on inside the agent and, and how the agent performs and where it messes up and where it doesn't mess up and, and basically communicate these kind of like patterns.
- 8:58
Um, and so, and so again, like the observability part, you can just see every step that's happening inside the agent. This reduces the kind of like uncertainty that people have around what the agent and what it's actually doing.
- 9:07
They can see that it's making three, five LLM calls. It's not just one. They're actually being really thoughtful about the steps that are happening. And then you can benchmark it against different things.
- 9:16
Um, and so there, there's a, a, a great story of a user of ours who used LangSmith initially to build the agent, but then, but then brought it and showed it to the review panel as they were trying to get their agent approved to go into production.
- 9:29
And they, uh, they ended the meeting under time, which almost never happens, uh, if, if you've been to these review panels. A- and, and they showed them basically everythide-- everything inside LangSmith, and it helped reduce kind of like the, the perception or the risk that people had, um, of, of, of these agents.
- 9:48
And then, uh, the, the, the last thing I wanna about-- talk about is, is kind of like the cost of something if it's wrong. Um, there's similar to kind of like the probability of things being right, this, this plays an outsized kind of like role in-- especially in larger enterprises among review boards and, and, and managers, people's
- 10:07
like perceptions of these agents. People hear stories of agents going wild and causing kind of like brain, uh, brand damage or, or, you know, giving away things for free.
- 10:15
There's this-- I think there's an outsized perception of, of, of kind of like what could be-- what could happen if things go bad. And, and so I think there's a few kind of like UI/UX tricks that people are doing and that successful agents have to kind of like just make this an, a, a non-issue.
- 10:36
So, so one is just make it easy to reverse the changes that the agent makes. So if you think about code, and this is a screenshot of, of, of Replit Agent, it's, it's a, it's a, it's a diff that it generates a PR.
- 10:47
Code's really easy to revert. You, you go back to the previous commit. And so I think, I think that's part of the reason why we see code being one of the first, uh, kind of like real places that you can apply agents, besides the fact that the models are trained on it.
- 11:00
It's also that when you, when you use these agents, you create all these commits and, and, and, and re-- well, it depends how you do it. Replit does it in a very clever way, where every time they change a file, they save it as a new commit.
- 11:10
So you can always go back, you can always revert kind of like what the agent does. Um, and then, and then the second com-- part is, is having a human in the loop.
- 11:18
So rather than, uh, you know, merging, uh, code changes into, into main directly, open a PR. That's putting the human in the loop. And so then the, the, the effect of the agent is, it's not kind of like making changes.
- 11:30
There's the human who's kind of like approving what the agent does. And this seems, uh, uh, uh, maybe a little subtle, but I think it completely changes the cost calculations in people's minds about what the cost of the agent doing something bad is, 'cause now it's reversible, and you have a human who's gonna prevent it from even
- 11:47
going in in the first place if it's bad. Um, and, and, and so human in the loop is one of the big things that we see, uh, people selling these enterprises and building inside enterprises really leaning into
- 11:59
So to make this a little bit more concrete, what are some examples of this? I think deep research is a pretty good example of this. If we think about this, there is a period of time up front when you're messaging with deep research that you go back and forth, it asks you follow-up questions, and you kind of
- 12:14
like calibrate on what you want to research. That puts kind of like the human in the loop. It also-- it makes sure that it gets a better result. So it increases kind of like the value that you're gonna get from the report because it's more aligned with what you actually want.
- 12:26
And then dur-- deep research, it doesn't, you know, take this and publish it as a blog out in the internet, or it doesn't take it and email it to your clients.
- 12:33
It produces just a, a, you know, a report that you can read and decide what to do with. So it's not actually doing anything. It's up to you to take that and, and, and do things.
- 12:41
I think similarly, when you think about, uh, code, it's a-- it's, it's another great example of, uh... So Claude Code also has, uh, this ability where it asks questions, it clarifies things.
- 12:52
Um, this is to kind of like, uh, both keep the human in the loop, but also make sure that it yields better results. And then again, with code, maybe you're not making a commit every time you change things, but it's, it's on a separate branch.
- 13:02
You open a PR, you're not pushing kind of like directly to master. And so I think these are, these are examples of things in the general industry that kind of like follow some of, of, of these patterns.
- 13:14
So okay, so we've figured out a few levers that we can kind of like pull to, to kind of like try to make our agents more interesting to, to be deployed in, in the enterprise.
- 13:23
What next? What next is kind of like, how, how do we scale that? So if, if this kind of like has, has kind of like positive value, then what we really wanna do is just multiply this a bunch and scale it up a bunch.
- 13:34
And I think this speaks to the, the concept of kind of like ambient agents, um, which is, uh, when we think about agents working in, uh, you know, in this futuristic view, agents working in an enterprise doing things in the background.
- 13:48
They're doing things in the background. They're not being kicked off by humans, kind of like still in the loop. They're being triggered by, by, by, by different events. Um, and, and I think the reason that this is so powerful is that it scales up this positive expected value thing even more, um, than, than we can.
- 14:04
Like, I can only really have one-- maybe I can have two chat boxes open at the same time, but now there can be, there can be, you know, hundreds of these running in the background.
- 14:12
And so when we think about the difference between chat agents, which I would argue we've, we've mostly seen, and ambient agents, one, one big difference is ambient agents are triggered by events.
- 14:21
That lets us scale ourselves. Instead of a one-to-one, it's now a one-to-many conversation that we can be happening. Um, and, and, and so the concurrencies of these agents that can be running goes from one to, to unlimited.
- 14:32
Um, the latency requirements also change. So when chat, you have this kind of like UX expectation that it responds really, really quickly. And that's not the case with, with ambient agents 'cause they're triggered without you even knowing.
- 14:44
So like how, uh, how do you, how do you know? How do you even care how long it's running? And so you can-- Th- what does this let you do?
- 14:49
Why does this matter? This lets you do more complex operations. So you can do more things. So you can start to bui- build up kind of like a bigger body of work.
- 14:57
You can go from kind of like changing one line of code to changing a whole file or making a new kind of like repo or, or any of that.
- 15:02
And so instead of this agent just responding directly or calling like a single tool call, which usually happens in these chat applications because of the latency requirements, it can now do these more complex things, and so the value can start kind of like increasing in terms of what you're doing.
- 15:16
And then the, the, the other thing that I wanna emphasize is that there, there's still kind of like a UX for interacting with these agents. Um, so ambient does not mean fully autonomous, and this is really, really important because autonomous-- when people hear autonomous, they think the, the cost of this thing doing something bad is, is, is,
- 15:33
is really high because I'm not gonna be able to, to oversee it. I don't know what's going on. How do I-- It could go out there and like run wild.
- 15:40
And so ambient does not mean fully autonomous. And so there are a lot of different kind of like human in the loop interaction patterns that you can bring into these kind of like background, these ambient agents.
- 15:50
Um, there can be like an approve, reject pattern where for certain tools you wanna explicitly say, "Yes, it's okay to call this tool." You might wanna edit the tool that it's calling.
- 15:59
So if it messes up a tool call, you can actually just correct it in the UI. You might wanna give it the ability to kind of like ask questions so that you can answer them.
- 16:06
You can provide more info if it gets stuck kind of like halfway through. And then time travel is something that we call, uh, human on the loop as well.
- 16:13
So this is after the agent's run. If it messed up on step like ten out of a hundred, you can reverse back to step ten and say, "Hey, no, like resume from here, but do this other thing like slightly differently."
- 16:24
And so it's-- so human in the loop we think is, is super, super important. The, the other thing that I wanna call out just briefly is I, I think there's this, there's this, uh, intermediary state where, where we're starting to be right now.
- 16:35
I-- like I wouldn't call deep research or, uh, or Claude Code or any of these coding agents ambient agents because they're still triggered by a human. And-- but I think these are good examples of kind of like sync to async agents.
- 16:47
Um, and so, so Factory, uh, is a coding agent. They, they use the term kind of like async coding agents, and I, I really like that. Um, but, but I think this, this kind of like sync to async agents is a natural progression if you think about it.
- 16:59
Like right now to start, or you know, a, a year ago, everything was a sync agent. We were chatting with it. It was very much in the moment. The future is probably these autonomous agents working in the background and still pinging us when they need help.
- 17:09
But there's this intermediate state where, where the human kicks it off, uses that kind of like human in the loop at the start to calibrate on what you want it to do.
- 17:16
And so I, I think that, that ch- table I showed of like chat and, and ambient is actually probably missing a column in the middle that's like these sync to async agents.
- 17:25
Um, anyways, an, uh, an example of some of the UXs that we think can be interesting for these ambient agents are, are basically what we call agent inbox, which is where you surface all the actions that the agent wants to take that need your approval, and then you can go in and, and approve, reject, leave feedback, things
- 17:38
like that. Just kind of tie this together and make, uh, it, it really concrete what I mean by ambient agents. Uh, email I think is a really natural place for ambient agents.
- 17:47
These, these agents can listen to incoming emails. Those are events. They can run on however a- many emails come in, so that's, you know, uh, in theory, unlimited. Um- But you still probably want an, an agent, uh, or you still probably want the human, the user to approve any emails that go out or any calendar events that
- 18:03
get sent, um, depending on your level of comfort. And so, uh, this is a concrete thing. Um, uh, I actually built one, uh, that I have myself. Uh, we've used it to kind of like test out a lot of these things.
- 18:14
Um, if people wanna try it out, the, there is a QR code that you can scan and get the GitHub repo. It's all open source. Um, and, and I think this is, uh, uh, it's not the only example of ambient agents, um, but it's one that w- that I've built myself and so we, we talk about, a
- 18:27
lot about internally. Um, that's all I have. Uh, I'm not sure if there's time for questions or not. One or two questions if, if people have them. Um, yeah.
- 18:46
So my question is, uh, although everybody's talking about agents, uh, but only code generating agents are the one who are getting funding. Is it because, uh, you can measure what you have done and you can reverse what you have done, but for all other agents you can do lot of stuff, but you cannot measure what you have
- 19:03
done, you cannot reverse what you have done?
- 19:06
Yeah. I, I, I think those are ... Yeah, I think there's a variety of reasons. I think those two measure and, and ... Well, okay, so the measure thing I think probably more so.
- 19:15
Like, you can ... Uh, a lot of the large model labs train on a lot of coding data because you can test whether it's correct or not. You can run it, see if it compiles.
- 19:22
Same with math data. Math is ve- it's verifiable, right? So math and code are two examples of verifiable domains. Essay writing's less verifiable. What does it mean for an essay to be correct?
- 19:31
That's far more ambiguous. And so because of these verifiable things, uh, you're able to bootstrap a lot of training data, and so there's a lot of training data in the models already about code, and so the models are better at that.
- 19:41
That makes the agents that use those models better at that. Then the second part, uh, I, I do think code lends itself naturally to kind of like this commit and this draft and this preview thing.
- 19:51
I think that's more generalizable. So like legal is a great example. Legal you can, you can have first drafts of things. That's very common. Same with essay writing. I think the concept of like a first draft is actually a really good UX to aim for.
- 20:03
It lets you do far more. It also puts the human in the loop, and so you kind of get, it, you get this dual kind of like ... Like, if you put the human in the loop at every step, like that doesn't provide any value.
- 20:12
Like, each step is so small. So the key is like h- finding these UX patterns where the agent does a ton of work, but the human's still in the loop at key points.
- 20:20
It's first drafts I think are a great kind of like mental model for that. And so anything where there's like first drafts, legal, writing, code, I think, I think that's a little bit more generalizable.
- 20:31
The i- i- the verifiable stuff, that's a little bit tougher. Um, uh, but yeah.
- 20:39
Um, yeah. Oh, no, we're good? I'll talk to you afterwards. Cool. [laughs] Yeah, more than happy to chat after. Thank you all. [clapping] [outro music]