Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab

A browser agent can perform well in testing and still disappoint the people who need it to do real work. Felipe Blanes, a member of technical staff at Amazon AGI Lab, approaches that problem through his work helping customers use Nova Act, the lab’s tool for building agents that automate browser tasks.

Nova Act’s research preview launched in March of the preceding year with a deliberate goal: get the product into customers’ hands and collect feedback. The first few months exposed gaps in the developer experience, leading to a new experience in July. The service then launched on AWS in December. Blanes worked with customers throughout that progression, connecting what they were trying to accomplish to what the product actually helped them do.

The “benchmark illusion” starts with a familiar development sequence. A team receives requirements, builds a product, runs public benchmarks or evaluations against its own synthetic data, and improves the results until it feels ready to ship. Those tests can tell the team how well the product handles the scenarios it has chosen. Customers then introduce scenarios the team never anticipated.

The failure is in keeping the evaluations static while actual use changes. After shipping, a fixed test suite leaves the team hoping that its examples represent customer behavior. Unexpected tasks break the product without necessarily changing what the team measures. A higher score on the same tests cannot, by itself, close that gap. Production experience needs a route back into evaluation and development.

The eval flywheel begins by defining success in the customer’s terms. Before choosing a metric, understand the problem the customer needs solved. That definition determines which signals matter: collecting large amounts of telemetry is less useful if the measurements miss the intended outcome.

Instrumentation supplies one kind of signal; conversations supply another. Meetings and visits help uncover what customers are trying to do and why the observed behavior matters. Blanes’s instruction is direct: “talk to your customer.” Data can be misleading when the team lacks that context. The loop therefore combines measured behavior with insights from working directly with users.

Diagnosis then separates three kinds of work:

• Model failures: Identify where the model fails and feed those cases into model evaluations.
• Harness failures: Find problems in the surrounding agent software and send them to engineering for conventional bug fixing.
• Product gaps: Ask whether the product solves the intended problem, whether customers understand its purpose, and whether its positioning sets appropriate expectations.

This separation matters because a customer-visible failure does not automatically imply that the model needs improvement. It may require a software fix or a product decision. Once the gaps are categorized, the team prioritizes them and assigns work to science, engineering or product. The flywheel continues by repeating the process rather than treating the diagnosis as a one-time report.

How does customer experience reach the team that can change it? The cycle below shows the intermediate diagnosis step. Signals become useful when they lead to a specific kind of decision; repeating the cycle keeps the definition of success connected to what customers need.

Customers have their own progression through a product. Repeating the flywheel reveals different information at different stages, so the team should change what it is trying to learn as adoption grows.

• Use-case discovery: Establish the customer’s problem and whether the product can solve it.
• Early adoption: Explore what is possible. Customers try unexpected tasks, revealing both unsupported uses and surprising successes.
• Scaling: Look for recurring patterns across more customers. These expose the “hero use case”—the problem that repeatedly brings people to the product.
• Future product bets: Use gaps discovered during early adoption and scaling to decide what to build next.

An agent with 80% reliability can look ready to ship from the builder’s perspective. From the customer’s perspective, it can create another job. Someone must build the automation, monitor its execution and finish or repair the work when it fails. The agent’s successful attempts do not remove the need to supervise the workflow.

Blanes calls the change in willingness to delegate the “trust cliff.” In his customer experience, around 92% reliability changed the response from needing to watch the agent to feeling able to hand over tasks. These percentages illustrate his observed adoption pattern, rather than a universal threshold: the talk does not define a shared reliability metric or establish which workflows can safely be delegated at that level. The useful mechanism is the shift in supervision burden. Trust becomes a practical decision about whether the automation actually removes work.

That decision also needs honest information about the product’s limits. Customer discussions should distinguish three categories:

• What works: Explain the tasks the product handles well and the reliability customers can expect for them.
• What is being improved: Identify known gaps that the team is actively working on.
• What is out of scope: Say plainly which tasks the product cannot handle, before customers spend time discovering that themselves.

Clear limits help customers choose an appropriate use and avoid predictable disappointment. With many competing tools available, an unsuccessful trial can be enough for a customer to move on. Blanes’s qualitative lesson is that transparency about limitations can build more trust than higher benchmark scores: it lets the customer understand where confidence is warranted.

Amazon Leo, Amazon’s satellite internet business, supplied a concrete example of customer feedback changing the execution path. The resulting feature caches trajectories—the sequences of actions used to carry out a task—rather than requiring model inference every time. If the cached trajectory fails, execution falls back to the model.

The change moves repeated work onto a cheaper path while keeping model inference available when reuse does not work. Before caching, inference was needed on each use; afterward, a successful cached trajectory could avoid it. Blanes reports cost optimization for this customer, without quantifying the savings or specifying how trajectory failure is detected.

Where does inference remain in the cached workflow? The diagram shows that it becomes the fallback after a failed cached attempt. The cost benefit comes from successful reuse; the fallback preserves a route forward when reuse fails.

Two other customers exposed different kinds of gaps:

• Hertz: access for less technical QA staff. Some engineers using Nova Act for QA automation could write Python and other scripts. Others could not. That difference prompted a non-technical product experience intended to let more customers use the tool. The obstacle was how people built the automation, rather than simply how well the model performed it.
• Sola: flexibility across varied automation tasks. The process-automation startup served many different customer use cases. Its needs pointed toward more customization, particularly in the actuation stack—the part of the tool that carries out actions. The talk identifies the direction of that flexibility without specifying the customization interface.

These examples connect customer contact to three distinct decisions: reuse work to reduce inference costs, broaden who can build an automation, and let an automation provider customize execution. None requires interpreting every customer problem as a request for a better model.

Repeated use of the flywheel led to four operating principles:

• Derive scenarios from production: Synthetic data can help at the beginning, but customer signals should feed the evaluations once real usage exists.
• Categorize for action: Separate research, engineering and product issues so each can be prioritized and addressed by the appropriate team.
• Share capabilities honestly: Include limitations in customer communication, especially tasks the product does poorly.
• Run the loop quickly: Shorten the time between learning from customers and acting on what they reveal.

The expectation is that evaluations should get “smarter every week.” Here, smarter means learning from the problems customers are actually encountering, rather than continuing to test only the team’s original assumptions. If that learning stops, evaluations become stale even while the product keeps running.

The ending ties freshness to the customer journey. As customers move from exploration into broader adoption, what they attempt changes and can become more complex. Evaluations must keep reflecting that work. The two lasting requirements are therefore connected: measure what customers care about, and keep updating those measurements as their use evolves. A suite built once cannot keep answering a question that keeps changing.