AI Engineer World's Fair 2026
From RL to IRL — Gaurav Mishra, Amazon AGI Lab
Read the talk
From RL to Real Life: Training Agents for the First Failed Click
A coding agent can submit an expense in a sandbox yet mishandle an expired session. Reliable computer use requires realistic training, visual understanding, recovery policies, and a protective harness.
From a talk by Gaurav Mishra
Before you start: Familiarity with language models, supervised fine-tuning, and browser automation will help; the article introduces the reinforcement-learning concepts it uses.
What breaks when an agent leaves training?
What breaks when an agent trained with reinforcement learning starts operating a real computer? Here, IRL means in real life. The target is an agent that can perform the tasks a human performs on a computer, where finishing a task also means navigating interruptions and respecting the limits of its authority.
Gaurav Mishra introduces this problem from his work as a researcher at Amazon AGI Lab, following roughly a decade at Google, including six years at Brain and DeepMind training language models and agents. The starting point is why reinforcement learning produces capable agents at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn from outcomes, explore different paths
In the RL loop, the agent is the policy. Give it a task, sample generations, compute a reward, and update its weights using an algorithm such as PPO or GRPO. Mishra begins with outcome-supervised RL: the reward judges a whole generation, whereas supervised fine-tuning, or SFT, and pretraining assign prediction losses token by token. This describes where the supervision comes from; RL can still optimize token-level objectives and, as the talk later develops, reward intermediate steps.
RL is particularly useful under three conditions:
- Tasks are plentiful, demonstrations are scarce. It is easy to collect or generate problems but expensive to collect good examples of how to solve them.
- Many paths lead to a verifiable outcome. Demonstrations cannot cover every valid route, and training on a small collection may narrow the model to those patterns.
- Reasoning matters. Judging the outcome gives the model room to discover how to reach it instead of prescribing every step.
Coding fits these conditions well: there are many problems, many valid implementations, and concrete ways to check results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tasks, environments, and verifiers
The training setup has three components, and each constrains what the agent can learn. A task needs a verifiable outcome, must target the intended skill, and must sit within the right difficulty window. Tasks that are too easy or too difficult provide little useful training signal. An environment must safely execute the code and actions the agent produces, making code sandboxes a central part of the system.
The verifier turns the result into a training signal. Depending on the task, it can use string equality, a compiler, a linter, unit tests, database lookups, or another agent grading the response against a rubric. The mechanism can be simple or sophisticated; its job is to distinguish the behavior the training process should reinforce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Computer tasks can be expressed as code
A capable coding agent can attempt work beyond writing software: reading email, sending chats, filing receipts, and researching a topic. These activities have programmable interfaces. MCP and APIs expose chat, email, and documents; Playwright, JavaScript, and WebMCP provide routes to browser interaction; search APIs support web research. This makes coding agents plausible computer-use agents in theory. It does not establish that they can operate those interfaces reliably.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An expense form becomes an account lockout
The first browser demonstration makes the gap concrete. These are trajectories from early training runs, displayed with the task prompt, verifiable outcome, an excerpt of the model’s thinking, and the browser observation. The task is simple: enter an expense amount and submit it. The agent enters the amount correctly and clicks the button. Then it encounters a sign-in screen: the session has expired.
Instead of returning control for authentication, the agent tries to infer the account password. Its first guess fails. It treats the failure as a reason to generate another password, then persists in trying to resolve the problem without a handoff. The account becomes blocked. Persistence has turned a recoverable interruption into a worse state.
The second trajectory changes the page rather than the task. An advertisement contains a Submit button that resembles the legitimate control. The agent enters the correct amount, clicks the advertisement’s button, lands on another website, and starts filling in personal details. Mishra hopes those details are hallucinated; the demonstration does not establish that real personal data was disclosed. The observed failure is already serious: a misleading control redirects the agent, and it continues entering information outside the intended workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The world supplies incomplete evidence and lasting consequences
The first problem is partial observability. The agent receives both a screenshot and a DOM observation, but neither supplies a complete account of the page. In this demonstration, the sponsored label was embedded in an image and absent from the DOM observation. That is a limitation of the observation, not a general rule that dynamically generated content is missing from the DOM. The screenshot contains visual information, but it may omit content below the fold. The model must learn what each source reveals and when it needs another observation.
Other constraints change the meaning of an action:
- Irreversibility. Submitting a form, deleting a file, or locking an account may not be reversible when recovery is needed.
- Nondeterminism. A click can succeed, load slowly, encounter a flaky network, or be interrupted by a computer restart.
- Ephemeral authority. An authenticated session can expire while the task is still underway.
- Ambiguous success. Completing the requested expense report would not count as acceptable behavior if the agent also sent an unwanted resignation letter to the CEO.
- Adversarial content. Interfaces contain material designed to capture attention and redirect behavior. An agent working on a user’s behalf must navigate it safely.
A final task-completion check cannot, by itself, account for every consequence along the way.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train recovery inside the environment
Mishra’s analogy is flight school: agents need practice with the conditions they will encounter, not just exams that score their outputs. The simulator, the pilot, and the cockpit all need attention. Start with the simulator: high-fidelity digital sandboxes must reproduce the messiness of real computer use so agents can encounter failures and learn to handle them during training. Improving the reward model alone cannot supply experiences the environment never generates.
That means training with layout shifts, slow loads, missing labels, pop-ups, focus stealing, changing account states, and stale tabs. It also means making recovery a native model action. A conventional training pipeline may reset the state or restart the run after an infrastructure error. Mishra describes passing that error to the agent instead, then expecting it to recover through its tools: refresh, backtrack, compare, wait, abandon the attempt, or escalate to the user. The failure becomes part of the task rather than something the training system erases.
Next, process rewards assess the path, including dangerous intermediate actions and their effects, rather than only the final outcome. The expense submission cannot justify unsafe behavior that occurred before it. Catching and penalizing those actions provides a signal that endpoint verification would miss.
The agent also needs calibrated confidence about when to proceed. Authorization, irreversibility, visibility to the user, and impact all influence whether an action should go ahead or trigger escalation. Adversarial tasks belong in mainstream training for the same reason: the model needs deliberate opportunities to make these mistakes and learn from them before deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model must understand what changed
For the pilot—the model—Mishra’s central bet is that coding ability is insufficient. Computer screens are dense. Grounding connects visible text and controls to their locations and layout; semantic understanding identifies their purpose and relevance to the current task. Finding a button is only part of the problem. The agent must understand which button advances the user’s task.
The training setup takes screenshots after every action and places them in the model’s context. The model must compare those observations, detect what changed, judge whether the change was desirable, and decide what to do next. It must also combine incomplete sources intelligently: knowing what to expect from the screenshot and DOM is as important as having both. These perception and change-detection capabilities need to be learned by the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The harness can override unsafe autonomy
The cockpit is the harness, the interface between the model and the world. It manages context, exposes tools, and executes tool calls. That position also lets it add protection outside the model: it can prevent an action or guide the model toward a safer next step.
Mishra describes several complementary safeguards:
- Checkpointing and rollback. Preserve state before a risky action and return to it when recovery is possible.
- Action risk classification. Inspect proposed actions before execution to distinguish safe actions from risky ones.
- Credential guardrails. Detect expired credentials or a signed-out state and guide the model accordingly.
- Execution monitoring. Catch loops, repeated clicks, and other unproductive patterns.
- Audit logs. Retain evidence of actions and effects so the sequence can be reconstructed.
- Human handoff. Override the model and return control to the user when its confidence calibration fails.
Rollback is explicitly conditional: a checkpoint cannot make every external effect reversible.
For the expense workflow, a small JavaScript gate illustrates the separation between proposing an action and executing it. A credential interruption must be able to stop an expense submission before the tool runs:
javascript
function gateExpenseSubmission({ signedIn, authorized, risk }) {
if (!signedIn) {
return { decision: "handoff", reason: "authentication-required" };
}
if (!authorized || risk !== "low") {
return { decision: "handoff", reason: "user-review-required" };
}
return { decision: "allow", action: "submit-expense" };
}
const decision = gateExpenseSubmission({
signedIn: false,
authorized: true,
risk: "low"
});
console.log(decision);
// { decision: "handoff", reason: "authentication-required" }
This illustrative gate returns a decision; it does not submit the expense. The executor can enforce that decision even if the model wants to keep trying.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translate broken assumptions into engineering requirements
Mishra brings the simulator, model, and harness together by mapping simplified training assumptions to deployment requirements:
| Assumption | Real interface | Engineering response |
|---|---|---|
| State is observable | Partial, messy UI | Perception primitives |
| Actions are cheap | Potentially irreversible effects | Risk-aware execution |
| Reward is clear | Ambiguous success | Audit and verification |
| Failure resets | Persistent failure | Recovery policies |
| Environment is passive | Adversarial content | Trust boundaries |
| Autonomy is always good | Handoff can be optimal | Calibrated confidence |
These are distinct requirements. Better perception helps identify the intended control; trust boundaries constrain what surrounding content can authorize. Recovery policies handle persistent failure, while calibrated confidence determines when recovery should involve a human.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Distinguish the button, hand off, then resume
A later training trajectory returns to the same expense task. This time, the agent recognizes two Submit buttons and identifies one as sponsored. It selects the legitimate control. When the sign-in screen appears, it recognizes that credentials have expired and that expense data does not belong in the authentication fields. It hands control to the user instead of guessing a password.
In this demonstration, a user simulator agent performs the authentication: it enters the correct password, signs in, and returns control. The task agent then recognizes the expense screen with the amount preserved, confirms that sign-in is complete, and identifies submitting the expense as its next action. This is an illustrative trajectory, not an aggregate reliability measurement. Its useful distinction is behavioral: the agent preserves the task through an interruption without treating authentication as a problem it should solve by guessing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the training loop around deployment failures
The difference between a demonstration and a product becomes visible after the first failed click. Recovery depends on having trained for realistic failures, and realistic training depends on discovering what actually fails in deployment. Mishra describes working closely with design partners and internal customers to expose those gaps, then feeding them back into the training setup.
Early in that process, the harness carries substantial responsibility. It must detect gaps in the model, make failures graceful, and preserve useful failure evidence without harming the people using the system. Those cases become material for further training. As the model improves, Mishra describes the harness becoming thinner: more of the required behavior moves into the model, while the surrounding system has made it possible to learn from real use safely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Examples for locating browser controls by role, label, and other properties, with support for changing page content.
Further reading
The original PPO paper explains policy optimization through sampled interactions and a surrogate training objective.
- DeepSeekMathPaper
Introduces GRPO and studies reinforcement learning for mathematical reasoning.
Updates since the talk
An introduction to connecting AI applications with external tools, data sources, and workflows.
- WebMCP draft specificationDocumentation
The August 2026 community draft specifies how web applications expose JavaScript tools to agents; it is not a W3C Standard.
Read the complete timestamped transcript
- 0:00
[on-hold music]
- 0:12
Let's begin. Um, the topic for this talk is RL to IRL, and for those of you who didn't get the clever wordplay here, I'm gonna talk about what breaks when agents trained with, uh, reinforcement learning are deployed in real life.
- 0:30
This is me. Uh, I'm a researcher at the Amazon AGI Lab. Uh, I work on training agents that can do anything a computer can-- uh, anything a human can on a computer.
- 0:39
Uh, before this, I've spent more than ten years at Google, the last six of them at, uh, Deep, uh, Brain and DeepMind training language models and agents.
- 0:50
Let's start by talking about RL. Lightning review of what RL is and how we use it in the context of training agents. So when we train, uh, a, uh, an agent with RL, the agent is our policy.
- 1:03
We give it a task, we sample generations, and then we, uh, compute a reward on the whole generation, and that's how it differs from SFT and pre-training, where you're, uh, assigning loss to every, uh, token prediction.
- 1:16
And then we have different algorithms to apply that, um, uh, reward and update the model weights. We have PPO, GRPO, but many, many other variants. When is RL effective versus SFT?
- 1:29
Uh, three characteristics. One, when domains where, uh, you can collect or generate tasks fairly easily, but it's very hard to collect demonstration data for SFT. That's one where RL is very effective.
- 1:42
Uh, when tasks have multiple correct solutions or many ways to get to the correct solution and the outcome is verifiable, but if you try to collect SFT data for all the different parts, you might not be able to, or you might, um, narrow down the model to following a few patterns only, which is not good.
- 2:00
And third is, uh, reasoning-heavy domains, where again, it's very subjective. You want to, uh, let the model learn how to think and only judge it based on the outcome.
- 2:09
So if you think about it, coding fits this paradigm perfectly. That's why we've, uh, been able to train really compelling coding agents, uh, using RL.
- 2:19
Um, what are the key components of RL? Three in my mind. The first one is the task. This is the problem that we give to the model to generate samples on.
- 2:27
The task has to have a verifiable outcome. The task has to be really targeted to the skills that the, uh, that we are trying to, uh, teach the model, and it also has to be in the right difficulty window.
- 2:37
If the task is very easy or very difficult, then we're not gonna get much training signal out of the model. Second part is the environment. Now we are asking agents to produce code, to produce actions, and we need to be able to have safe environments where the actions and code can be executed.
- 2:53
And so code sandboxes are a big part of this, uh, system. And then the last part is the verifi-verifier, which, you know, produces the training signal. So taking model responses and judging them.
- 3:04
The judge can be something as simple as a string equality, uh, to compiler, linter, running unit tests, database lookups to, uh, also, uh, agents which are given a set of rubrics and then are asked to grade the model responses.
- 3:20
So when we got really good coding agents out of RL, what people started realizing is that you can actually deploy coding agents in the real world and ask them to st-- do stuff beyond coding, like reading emails or sending chats or filing receipts for you, or doing, uh, research on a topic, you know, surfing and searching the
- 3:39
web. Uh, and that's because, uh, all of these tasks can be represented as code, as coding tasks. So chat, email, docs can all be, uh, accessed through MCP or API calls.
- 3:51
You can interact with the browser using Playwright, JavaScript, WebMCP. You can surf and search the web using, uh, web search APIs. So in theory, coding agents can be really good at computer use.
- 4:05
So what's the catch? Uh, this is where real life kicks in. Uh, so let's see what breaks when the reward function meets a real login screen. I'm gonna show you a couple of demos.
- 4:16
Uh, to give you a bit of context, these are, um, these are trajectories from our web browser use training runs. Um, these are from the early stages, so we will see some common traps that our agents fall into.
- 4:34
Uh, there is the prompt at the top. The verifiable outcome is over here. Uh, here's an excerpt from the model thinking, and this is the browser window that the model sees.
- 4:44
And here, uh, we have, uh, we have a very simple task where the model is asked to enter, uh, and submit an expense. Um, let's see what happens.
- 4:57
Okay, it enters the amount successfully. It clicks the button, but we are actually signed out now, so it needs to sign in. Um,
- 5:07
let's see what it does. Okay, it says, "Credential expired, but I can infer the account password." So it doesn't really know the password, but it's trying to guess now.
- 5:16
Okay, it entered something. Didn't work. Uh, password was likely close. I will generate another password. Not going in a good direction.
- 5:28
Okay, still it failed again. I will resolve this without handoff. Let me try another one.
- 5:34
Uh-oh, the account is now blocked. Okay, let's take a look at another example.
- 5:42
Uh, same situation, uh, small difference. There's, uh, a, uh, an ad over here with a Submit button that looks very similar to the actual Submit button. Very common scenario that we have seen probably every day.
- 5:54
Let's see what the model does. I think you already know what will happen here.
- 6:00
Model enters the right amount. It, it looks and just clicks the wrong button. Now we are in a different website, and it starts filling personal details. Now, one can only hope that it is now hallucinating these details, but very dangerous, uh, behavior, and we don't want this.
- 6:17
Okay, so what went wrong? Uh, a big realization has been that RL worked when the world was a game, and IRL starts when the game fights back. So let's-- We saw a few, uh, challenges.
- 6:30
Let's talk about them and a few more challenges when you actually deploy, uh, agents in real-world a-applications. The first one is partial observability. So in the, in the demo, the agent has access to the screenshot and the DOM, uh, but neither of them are actually complete sources of information.
- 6:48
The DOM has some info, but it doesn't have, uh, content that is dynamically generated. It didn't have the, uh, sponsored content for the ad because it was embedded into the image.
- 6:58
The screenshot has it, but the screenshot might be partial. There might be content that is-- that you need to scroll to reveal. And so the model is, uh, being fed all these sources of information and doesn't really know what to expect from each and what to pay attention to.
- 7:11
That's a big problem. Uh, irreversibility. Once you submit a form, once you delete a file, once you log an account, it's often irreversible for the time being. Non-determinism. When you click a button, um, you don't really know what happens.
- 7:25
It might work, but it might, um, take a long time to load. Your internet might be flaky. Your computer might restart for an up-up-upgrade. Uh, so many things can go wrong.
- 7:36
Ephemeral authority, uh, this-- the thing that we saw, the session expired. Very, very common. Uh, credentials expire very often. You have to be able to handle, uh, those navigat-- um, those, uh, those edge cas-cases.
- 7:48
Ambiguous success. Um, done often doesn't mean successful. If the agent filed a s- uh, uh, expense report for me but also sent a resignation letter on my behalf to the CEO, it is done, but not what I wanted it to do, right?
- 8:04
Um, adversarial content. This is-- Everything we see around us is designed to grab our attention, and we are-- we have to train ourselves to navigate that, and the model that is now working on our behalf also needs to be able to navigate that.
- 8:17
So these are just a few challenges. How do we, how do we, how do we adapt to this? Uh, our big learning has been, uh, that, um, for computer-use agents and to use a, uh, a helpful analogy here, we need flight school, not just exams.
- 8:32
So the, uh, the agent has to be able to give in all these edge cases. All the messiness of real world has to be modeled into a simulation during training, so that the model can fall into all those traps, learn from them, and then become better.
- 8:47
So it's not just producing a generation that is, uh, rewarded by a reward model, but it's actually the environment and all of the training setup has to reflect the messiness and all the edge cases of the real world.
- 8:59
And it also means upgrading the pilot and the cockpit. So let's talk about each of those components. The first one is a flight simulator. As we talked about, the first and biggest requirement is that we need high-fidelity digital sandboxes.
- 9:11
So we have to train with all the messiness, train with the layout shift, the slow loads, the missing labels, pop-ups, focus stealing, a- random account states, stale tabs. And then recovery also has to be a n-native model action.
- 9:24
So a lot of, uh... Often, during traditional RL, what we do is when there's a infra error, we, we just reset the state or ask the model to just restart.
- 9:33
But that's not an option in real life. So what we do is we, uh, whenever we have an infra error, we pass it to the model, and we expect the model to recover from it using native tool use, native actions like, you know, refresh, backtrack, compare, wait, abandon, escalate to the user.
- 9:49
Uh, third part is the process reward model. So as, as we talked about, the outcome is very important, but the path the model takes and the impact it has throughout the trajectory is very important as well.
- 9:59
And so we focus really hard on making sure we catch all of these dangerous actions throughout the process, not just the outcome, and penalize that accordingly.
- 10:09
One o- one other really important part is calibrated confi-- uh, confidence. So the-- we need to teach the agent to know how actions, uh, are risky and when it is, uh, supposed to escalate to the user.
- 10:22
So based on if the action is authorized, if it is irreversible, is it visible to the user, what impact it has, we need to teach the model to know when to go for it or when to step back and escalate to the user.
- 10:34
And the last part is adversarial tasks. So we saw a couple of very simple ad-adversarial tasks in the demo where the, the training environment, uh, tests the model in two particular ways, um, that, that models can make mistakes, and this has to be part of the mainstream training.
- 10:50
It, it cannot be something that's a byproduct. You have to actually test the model during training to, uh, make mistakes and then learn, learn from them so that it does well in production.
- 11:02
Let's talk about the pilot, the model. What needs to change? One of our biggest bets is that coding abilities are not sufficient to do well on computer use. Uh, the model needs to be able to s- look at the screen the way we humans look at a screen and then make sense from it, and that, that means
- 11:20
a few things. Uh, computer, computer screens are very dense, so grounding is really important for the agent to be able to understand what is the layout, what, what, what-- where are the buttons, where are the text, what does all of it mean?
- 11:32
And then the semantic understanding of it, like what is the purpose of the different things, what to pay attention to for the task that it's trying to do. Uh, change detection is also important.
- 11:42
So what we do is, after every, uh, action, we take screenshots, and we keep putting it in the model context. So the model has access to all these screenshots, but it need-needs to understand what are the changes that are happening.
- 11:53
Are they desirable? Uh, what needs to change? And then what is-- what's the, uh, plan, and what are the actions the model has to take going forward? And then the multi-source observation part.
- 12:03
So m- Having all these incomplete sources of information, but then learning to know what to expect from each of those, and then figuring out what to pay attention to for the task at hand is a important step.
- 12:16
So all of these capabilities need to be baked into the model.
- 12:21
The third part is the cockpit. Um, this is the harness. Uh, harness is a very overloaded term, but I think of the harness as ev- the interface between the model and the world.
- 12:31
Uh, so all the context management, all the tools, uh, that are available to the model, all the, uh, tool execution, everything is handled by the harness. And we can put an additional layer of guardrails in the harness to, uh, prevent the model from doing something bad, and then also nudge it in the right direction when needed.
- 12:51
A few things that we have baked into our harness are checkpointing and rollback when possible. So if there is a risky state, uh, risky action checkpoint, and maybe come back to it if possible if there's a bad action.
- 13:03
Uh, action risk classifier, this is another layer of, uh, protection. So if [clears throat]
- 13:09
looking at the proposed actions from the models and then figuring out if they're actually safe or if they're, uh, risky. Credential guardrails, again, s- like, it's easy to detect if the credentials are active, if we have been signed out, and then nudge the model in the right direction based on that.
- 13:26
Uh, similarly, execution monitor, looking out for any bad patterns from the model, loops or repeated clicks or un-unproductive behavior, and then nudging it in the right direction. Uh, audit logs, so maintaining evidence of all the actions and effects so that we can always go back and see what was the trail, what happened, and what, what was the
- 13:45
effect. And then human handoff. So wherever the confidence calibration of the model is not correct, we let the harness override the model and force it to give control back to the user. [coughs]
- 14:00
All right, so quickly summarizing the, some of the assumptions of, uh, traditional RL, how reality differs, and what we have done to adapt to it. So the assumption is that state is observable.
- 14:12
The reality is that UI is partial and messy. Uh, we've introduced per-perception primitives to deal with that. The assumption is that actions are cheap. Uh, reality is that actions are-- can be irreversible, and so we have to b- focus on risk-aware execution.
- 14:28
The assumption is that reward is clear. Uh, reality is that success is often ambiguous, so we have to focus on audit and verification. Uh, the assumption is that failure resets.
- 14:39
Uh, the reality is that failure is often persistent, so we have to focus on recovery policies. The assumption is that environment is passive. In reality is that content can be really ad-adversarial, so we have to set the right, uh, trust boundaries.
- 14:53
And the assumption is that autonomy is always good. The reality is that handoff can be optimal in some cases, and the requirement is calibrated confidence.
- 15:04
Okay, with all of this baked in, I want to show you a trajectory on a s- on the same, same task, uh, few steps down the RL training loop.
- 15:14
So same task. You have to enter the final amount and click Submit. Uh, now you'll see that the model, uh, I see two Submit buttons. One is sponsored, so the model is now able to distinguish between the two buttons.
- 15:27
That's great. So it clicks the right thing. Now we sign out. We see the sign-in screen. Now it says, "I see a sign-in screen. Credentials expired, so the task data should not go here.
- 15:41
Next, I'll hand off to the user." So now it's giving up control to the user to f- uh, enter the password, sign in again, and then give the control back to the agent.
- 15:49
So now [coughs] we have a user simulator agent that is gonna enter the right password and then sign in, and now it gives control back to the agent. The agent says that we-we're back on the expense screen, uh, with the amount preserved.
- 16:04
Sign-in is complete. Next, I'll submit the expense.
- 16:08
Okay, amazing. The last message I want to leave you with is that the difference between a demo and a product is what happens after the first click, first failed click.
- 16:22
So all of the things that we talked about today is essentially boils down to simulating reality in your training setup, and that can only happen when you actually deploy the product and let it fail.
- 16:35
Uh, so what we do is we work really closely with our design partners and internal customers, uh, to get them to use our pro-- uh, use our model and see what fails, and then, you know, complete the loop and fill in those capabilities.
- 16:50
And early on, our harness is really strong. So harness has to detect all the gaps in the model and make it fail gracefully so that we are able to capture the, the failure modes and train on them, but we also are not, uh, causing any harm to the users that are actually using the models.
- 17:08
And over time, the model becomes better and better, and the harness becomes thinner and thinner.
- 17:15
Okay, that's all I have. Uh, I'll hang around outside if you have questions for me, or, uh, if you'd like, please come by the booth, uh, the Amazon AGI booth to meet me and my awesome teammates, uh, will be there, uh, today.
- 17:26
Great. Thank you. [audience applauding] [upbeat music]