AI Engineer World's Fair 2026
Computer Use at the Edge of the Statistical Precipice
Read the talk
Computer Use at the Edge of the Statistical Precipice
A blind script can outperform the model that produced it on a deterministic benchmark. Reliable computer-use evaluation needs verified variation and uncertainty estimates that account for it.
From a talk by Pierluca D'Oro
Before you start: Familiarity with agent benchmarks and task success rates is helpful; confidence-interval coverage is explained in the article.
When a recorded action sequence beats the model
Run a frontier model on a computer-use benchmark, keep a successful trajectory for each task, and turn those trajectories into a new agent. When a task arrives, this replay agent blindly repeats the recorded taps, typing and scrolling. It does not need to inspect the screen or decide what to do next. This is the opening experiment in Pierluca D’Oro’s Computer Use at the Edge of the Statistical Precipice, work conducted with collaborators at Meta Superintelligence Labs. For benchmarks with hundreds of tasks, he describes the resulting script as smaller than one megabyte.
In the presented experiments on OSWorld and MobileWorld, blind replay matches or exceeds the success rate of the frontier model that supplied its traces. The MobileWorld comparison uses Gemini 3 Pro as the source model and covers 96 of 201 tasks, rather than the full suite. These are the paper’s experimental results, not claims about today’s benchmark leaderboards.
The result is possible because the environment repeats itself. With a fixed starting state and deterministic execution, an action sequence that succeeded once can succeed again without adaptation. A high score can therefore measure reusable successful traces rather than robust computer use. The replay agent makes that weakness visible without requiring a sophisticated attack.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How pass@k inherits the replay problem
pass@k measures the probability that at least one of k attempts succeeds. That sounds useful: give an agent several chances and ask whether it can solve the task at all. But in a deterministic environment, finding one successful attempt also supplies a tape that can be replayed. The paper formalizes this connection: with fixed initial states and deterministic execution, the expected success of a replay agent extracted from k independent source rollouts equals the source agent’s pass@k. The expectation and execution conditions matter; this is not a claim that any recorded trajectory works in any environment.
Successful-trace selection consequently rewards the same property that made the blind script competitive. It establishes that a reusable success can be found, without establishing that the agent can cope with changed circumstances. Two separate design problems follow: build environments without this exploitable structure, and use evaluation metrics whose statistics do not create false confidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Introduce variation without introducing broken tasks
The first defense against replay is to make the benchmark multifactorial. Vary the data, appearance and initial state so that the agent cannot assume every run will present the same sequence of screens. Yet variation creates its own obligation: every generated combination must still describe a valid task. Otherwise, an apparent agent failure may actually be an environment failure.
Variation and validity sit alongside three other requirements: isolate the environment, provide the evaluator with privileged information for checking outcomes, and reproduce the relevant behavior of real systems faithfully. Together these form the PRISM principles.
| Principle | Requirement |
|---|---|
| Privileged | Support verifiers with access to relevant internal state. |
| Realistic | Faithfully reproduce the system being evaluated. |
| Integrity-checked | Check that generated combinations work as intended. |
| Sandboxed | Isolate task execution. |
| Multifactorial | Vary data, appearance and starting state. |
Existing benchmarks satisfy different subsets of these requirements. The aim is to combine them: diversity is useful only when tasks remain valid, and a valid task is useful only when its outcome means something beyond the benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
DigiWorld turns a few factors into many configurations
DigiWorld is the team’s attempt to meet all five principles with a collection of Android applications. Its reported scope is 15 apps, 387 verified scenarios and more than 3.2 million verified configurations. Those figures describe the available benchmark space, not an exhaustive evaluation of every agent on every configuration.
A task such as sending money can vary along several axes:
- Instance: change the amount to send or the task’s other specific parameters.
- Data profile: change the contacts, emails or other records available to the agent.
- Theme: change the application’s appearance.
- Starting screen: begin at login or at another valid page.
These factors multiply. Even a modest number of choices per factor creates a large space of combinations. Scaling the choices could produce billions of configurations; that is a growth possibility, distinct from the reported benchmark scope.
Because these axes are implemented in software, coding agents can help expand them—for example, by generating new instances or themes. But generating more software does not automatically produce a better evaluation environment. Every new combination also creates another opportunity for a task, its data and its interface to disagree.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compile tasks, verifiers and data together
Verification is what makes configuration generation scalable. Generate candidate combinations, reject the broken ones and retain valid configurations. DigiWorld implements this through a system resembling a compiler:
- Start with a parameterized task template, such as sending a specified amount to a specified recipient.
- Pair the template with a verifier for that task.
- Supply the mock data required for the task to be possible.
- Combine those inputs with base data and a base UI state to produce a valid configuration.
The task text, outcome checker and environment must agree on what success means.
A small Python example illustrates the binding between a payment task and its verifier. Here, the generated instruction and the outcome check derive from the same amount_cents and recipient_id; compiling the configuration does not execute the payment.
python
from dataclasses import dataclass
@dataclass(frozen=True)
class PaymentTask:
amount_cents: int
recipient_id: str
def compile(self, contacts: dict[str, str]) -> dict:
if self.amount_cents <= 0:
raise ValueError("Payment amount must be positive")
if self.recipient_id not in contacts:
raise ValueError("Recipient is absent from mock data")
name = contacts[self.recipient_id]
return {
"instruction": (
f"Send ${self.amount_cents / 100:.2f} to {name}."
),
"contacts": dict(contacts),
"starting_screen": "home",
}
def verify(self, new_payments: list[dict]) -> bool:
return any(
payment["amount_cents"] == self.amount_cents
and payment["recipient_id"] == self.recipient_id
and payment["status"] == "completed"
for payment in new_payments
)
task = PaymentTask(amount_cents=2500, recipient_id="contact_7")
configuration = task.compile({"contact_7": "Maya"})
The broader engineering problem includes validating the UI state and the interactions among factors. The compiler-like approach makes those checks part of environment construction, so diversity and correctness grow together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return to replay, then test the model itself
With verified variation in place, repeat the original experiment: evaluate a frontier model, extract its successful traces and evaluate the corresponding replay agent. D’Oro reports that replay agents now achieve little success. Some residual success is expected because certain tasks are naturally repeatable. The result addresses this particular replay exploit; it does not establish immunity to every way of gaming a benchmark.
The same infrastructure then supports a more useful question: how robust is the frontier model across each variation axis? A model that succeeds on a task might be expected to keep succeeding when only the theme or starting screen changes. In the reported experiments, that expectation often fails, with especially weak performance in the worst cases. Separating these axes makes it possible to measure fragility that a single base configuration hides.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repeated rollouts do not capture all uncertainty
A diverse environment fixes only half the evaluation problem. Its score still needs an honest estimate of uncertainty, and there are two sources of variation to capture:
- Action variation: repeated runs of the model can choose different actions and follow different trajectories.
- Environment variation: different configurations change the conditions under which those trajectories unfold.
Running the model repeatedly on one base configuration measures the first source while leaving the second largely unexplored. Deployment encounters both. The paper’s methodology therefore accounts for the structure of the benchmark rather than treating all uncertainty as rollout noise.
The diagnostic is confidence-interval coverage. Across repeated evaluations, a nominal 95% interval should contain the target performance approximately 95% of the time. Coverage is a property of the interval procedure; it is not the agent’s task success rate.
In the paper’s calibration simulation, the comparison is:
| Uncertainty included | Coverage of nominal 95% intervals |
|---|---|
| Rollouts only | 17% |
| Rollouts and configuration axes | 56% |
| Rollouts, configuration axes and scenario hierarchy | Approximately 95% |
The simulation uses 15 apps, eight scenarios per app, 27 configurations per scenario and three rollouts, across 200 experiments with 500 bootstrap replicates. D’Oro describes the rollout-only result as roughly 17–20% coverage. Accounting for configuration variation helps, but resampling at the scenario level is also needed to reach approximately nominal coverage. The estimator’s details are in the companion paper; the practical distinction is between narrow intervals and intervals that actually cover the intended performance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
False confidence becomes a deployment cost
Suppose an evaluation compares model A with model B to decide which to deploy. Very narrow confidence intervals can make the choice look settled. In the presentation’s example, the orange bars represent real performance, and they expose a choice that the apparently precise estimates got wrong. Precision around the wrong estimate is not useful certainty.
The cost example assumes one million tasks in a month, a four-percentage-point performance gap and an average cost of $12 per additional error. Those assumptions imply 40,000 additional errors and $480,000 in monthly cost.
This is illustrative arithmetic, not a reported deployment loss. It explains why a seemingly small model-selection error can justify substantial evaluation effort.
A reliable interval may instead reveal that the evidence is insufficient to choose confidently. That result has operational value: spend more time or money evaluating before committing to the more expensive mistake. An evaluation can be useful because it tells you that you do not yet know.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make rigor part of the evaluation design
These requirements belong in the evaluation’s design, before a score becomes a deployment decision. Apply PRISM to the environment, including initial-state variation—a practice D’Oro describes as uncommon despite its importance. Test whether successful traces can be replayed, and compute confidence intervals that respect the benchmark’s structure rather than understating total uncertainty.
Time pressure and prevailing practice make weak evaluation easy to accept: a benchmark is gameable, but widely used; error bars are missing, but others omit them too. Neither convention makes the evidence reliable. A misleading benchmark can direct a whole field toward maximizing a score that misses the capability it intended to measure. For an individual team, the same false confidence can turn directly into costly deployment decisions.
D’Oro closes by introducing Programma, describing its work on infrastructure for CUA-enabled verification and inviting hiring conversations. That closing connects the evaluation problem to the infrastructure needed to check software through computer use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The companion paper detailing DigiWorld, PRISM, replay equivalence and hierarchical uncertainty estimation.
Android benchmark with containerized environments, evaluation commands and trajectory inspection tools.
Desktop computer-use benchmark with environment setup, task definitions and evaluation infrastructure.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everyone. I am Pierluca D'Oro, and I'm the founder of Programma Labs.
- 0:17
And today, I'm gonna talk about Computer Use Agents Evaluation. And most of the work, um, and the details about it are in a, in a paper with this title.
- 0:28
And I did this work while at Meta Superintelligence Labs with, uh, the collaborators you see on this slide.
- 0:36
And so to start, uh, I want to introduce this type of, uh, agent. It's a weird type of agent that I call a replay agent. So imagine we run this process.
- 0:46
Uh, we run our frontier m- a frontier model, a good one, uh, on a benchmark we like. And then for every task, we collect, uh, a successful trace or a successful trajectory.
- 0:58
And we have our recorded tape of this type. So the actions might be tapping, uh, typing, scrolling. And we record this, and then we do this for all the tasks in the benchmark, and we com- sort of compile this into, uh, a replay agent that just when the tasks arrive, replays that sequence of actions blindly.
- 1:22
So if you do this for common benchmark with hundreds of tasks, this is gonna be a script that is, like, less than a megabyte. And this is a completely valid, like, type of agent that you can evaluate on the benchmark.
- 1:34
And if you try to evaluate this, uh, this kind of agent on standard benchmarks such as OSWorld or MobileWorld, you will see that the success rate of this agent compared to the, uh, frontier model from which the agent was extracted is actually, uh, the same or even better.
- 1:51
Um, so this is a kind of a weird but maybe trivial phenomenon. But I would argue that, uh, we shouldn't accept these kind of blind scripts beating the frontier models.
- 2:03
And the trick here or, like, the reason for why this happens is the, um, determinism of, of the existing-- of most existing benchmarks. And so, uh, if the benchmark is static, it's deterministic, then it is somehow gameable by this sort of strategy.
- 2:20
And it goes even deeper than this. Like, if you look at one of the metrics that people have been using in the past for evaluating computer use agents, um, uh, and it's pass@k.
- 2:31
This metric is defined as, you know, the probability of at least, uh, one of k attempts succeeding. Um, but if you look into, like, the details of how this metric works on a deterministic environment, you will see there is literally, um, and we prove it, like, formally in the paper, uh, like evaluating the success rate of the
- 2:51
replay agent that I've shown to you. Um, so, so basically if that replay agent felt weird, uh, to you, also pass@k on computer use tasks should somehow feel we-weird to you.
- 3:04
Um, or in other words, pass@k is sort of a metrification of that, uh, of that exploit of the replay agent. So these are two problems, specific problems, but they point at two general classes of problems in, in CUA benchmarks.
- 3:21
And, uh, these problems are around environments. So building environments and evaluation. Building good metrics to know if your agent is good or not. Uh, and in particular, we, we want to have environments that don't have exploitable structure, and we want to have metrics that are not fragile or they are not based on fragile statistics.
- 3:43
Uh, and so I'm going to talk about the-- both of the aspects now. So let's talk about building principled environments first. So the first aspect, um, that I, uh, that I worked on while, uh, you know, uh, working on environments is to try to design a set of principles that could be guiding principle when building environments so
- 4:05
that build, like, robust environments and trustworthy environments. Uh, and so if you think about the problem that I was describing with, uh, replay agents, uh, the first thing that you could think about as a solution not to have a replay agent to, like, hack your benchmark is, uh, to have your benchmark to be, uh, multifactorial.
- 4:24
So that means, uh, varying, um, uh, generating variation for your benchmark. So having stochasticity into the benchmark. Uh, and for computer use environment, that means, uh, varying stuff like, uh, data or appearance or s- or, or simply the initial state.
- 4:41
Uh, but if you do that, you want to make sure that all the combinations that you generate are valid. Uh, and so you want to have as a design principle in your environment also a system for checking and verifying that everything is working as, uh, as intended for every combination.
- 4:59
Uh, and of course you want, uh, the usual, uh, the usual things for your environment. So you want the-- your environment to be sandboxed and you want, uh, your environment to support, like, verifiers or privileged information.
- 5:12
And you want, of course, your environment to be realistic. So if it's a reproduction of a real system, you want, uh, that reproduction to, to be faithful so that the score that you get out is, um, uh, is a good one.
- 5:25
Um, and so if you sort them out, uh, you can remember these sort of principles as the PRISM principles for, for environment design. And we tried at Meta to build, uh, a benchmark that would be satisfying all of these principles.
- 5:40
And if you look at existing benchmarks, some of them do some things, uh, in a good way. Some of, uh, some others do other things in a good way.
- 5:48
But there is no unified benchmark that sort of matches all of these boxes. And we built one that is called DigiWorld. So the way DigiWorld, uh, in practice was built is as a set of like mobile apps, um, for, for Android devices.
- 6:03
So it's fifteen apps on spanning different domains, uh, with-
- 6:07
387, uh, verified scenarios and, uh, a number of configurations. So, uh, these configurations, they are in a large number, so 3.2 million. Uh, but the important thing is that they are verified.
- 6:22
Um, and indeed, uh, the axes are the ones that I was mentioning before. So you can imagine for each one of the tasks, you can vary, um, things like the, the instance.
- 6:33
So what is the exact amount of money that you're sending, for instance, or the data profile, like which kind of contacts or emails you have in the data for your, uh, for your task, or like the theme or the starting screen.
- 6:47
So do you start from the login page or do you start from another valid page? So if you do the math, even if you start from a relatively low number of, uh, base cases for each one of these variables, you end up having many, many combinations.
- 7:02
So you, you can get to like millions of combinations and if you scale this up, you can get to easily to billions of, uh, of combinations. Um, and all of these, uh, you know, different axes can be, can be manipulated by coding agents because in the end they are like forms of software.
- 7:19
So you can have a coding agent to generate different instances, different themes and such. Um, so you might think maybe it's easy to build an environment. You just generate as much software as you can, uh, with a coding agent, and then you have like a diverse environment, but it's a little bit trickier than that.
- 7:36
And indeed, coding agents can generate a lot of software, but a lot of software is not the same as an effective CUA environment. And the reason for this is that you need to verify the correctness of, uh, of, of your, uh, combination, right?
- 7:53
Um, and so the, the, the key to scale this up is to have a verification strategy for the variations of, of your tasks. Um, and so the kind of verification strategy to follow is, uh, is this one.
- 8:06
So you, you can generate many configs, all the combinations, uh, of dif- different factors that I, that I've explained before. Uh, and you can then have a system that rejects the broken ones, the ones that are not valid, and just keeps the, the valid configs.
- 8:23
Um, and so in the case of DigiWorld, uh, we did this by building a system that looks a little bit like a compiler and that works in the, in the following way.
- 8:32
So you start from a parameterized task template. Um, and so, uh, this might look like something like this. So you have send a certain amount or certain recipient, and then you have a verifier that corresponds to that template, and then you have mock data, uh, for the task.
- 8:50
So data that you need for that specific task to happen. Uh, and then we have a system that is like a DigiWorld compiler that takes all of this and given a base case of data, base case of UI state, puts all of this together and creates like a valid configuration.
- 9:08
Um, and so you can build systems like this in which the, the main craft is good software engineering, uh, to make sure that actually the combinations that you have are both diverse and valid.
- 9:21
And so if you build a benchmark in this way, um, and you follow the, the PRISM principles that I was talking about before, you actually can have a benchmark that is not hackable in that way.
- 9:33
Uh, and so if you do the same process we did before, you evaluate your frontier model, and then you evaluate the corresponding replay agent, uh, you will see that the replay agent doesn't get a lot of performance.
- 9:45
Um, it gets a little bit of performance. That is probably what you want. Sometimes some tasks maybe are repeatable by nature, but on average, you, you shouldn't expect a replay agent to have good performance on the benchmark.
- 9:59
Once you build like these diverse, um, combinations, you also can do other things like, um, measuring the, um, robustness of frontier models on different, o- over different axes of variation.
- 10:14
So the axes of variation, uh, I described before are here represented there. Um, and you can see that in the worst case, frontier models are pretty well actually at, um, being robust to these variations.
- 10:27
And so for instance, if you have a model, uh, that seems to be good at, at a given task, you would expect that if you just vary, you know, which screen the task is starting from or like what is the theme of the app, the model should pretty much have the same performance.
- 10:42
But this is actually not the case for most frontier models. And so if you have infrastructure like this, you can actually measure, measure that and like tailor your expectation, uh, about this kind of, uh, robustness.
- 10:56
So this was about the first aspect. This was building an environment that supports, um, diversity, uh, and that is robust enough to evaluate models. But the second aspect is as important as the first one is to measure uncertainty honestly.
- 11:14
Uh, so once you have all of this variation, how do you handle like computing, um, the real performance of your agent? Uh, and basically, there are two sources of, of stochasticity, of variation.
- 11:27
Um, uh, they are not exactly the same, but they are equally important. And so the one that we usually think about is the one about the actions, right? And so you, you run your model multiple times.
- 11:39
Um, in many cases you can have even quite different trajectories out of it because the action at each step will be different. Uh, but if you have a benchmark like the one that I've described with multiple combinations, with multiple variations, then also the variability from the environment becomes important.
- 11:58
And we wanna capture that because that is what we are gonna find, uh, in the real world. And so we need a methodology that captures, uh, both of these types of variation.
- 12:09
And, uh, in the paper there are the details. But, but basically we build a meth- we, we built a methodology that can accurately, um, capture these two types of variation, taking into account the structure of the benchmark.
- 12:23
And so if you start, like, in, in practice, it, it is useful to use this concept of coverage. Uh, so when you, when you compute a confidence interval, uh, basically you have some confidence that the performance of the model is inside of that range.
- 12:39
Um, and so you would expect that a 95% confidence interval would say that, you know, 95% of the time the performance of the model is in that range. But if you only use rollouts, so you only use the base case and what people would use normally, actually in realistic cases you have something like 17% or 20% coverage.
- 13:01
And that means that basically you only 20% of the time you, uh, guess the right performance of the agent, which can be pretty bad. But, like, if you take into, into account the hierarchy and you use the proper way of computing confidence intervals, you can get to the, to the full confidence interval and be 97, m- 95%
- 13:19
accurate. And so if this seems quite abstract, uh, you know, in practice that means that if you wanna make a decision about which model to deploy, maybe you have model A, model B, and you do an eval for, for those two models.
- 13:35
Um, you can have cases in which the confidence intervals seem really, really small. Uh, and so you make a decision based on those small confidence intervals, but they are actually overconfident and so, uh, this was the wrong decision.
- 13:50
So the, the orange bars are the real performance here. Uh, so you make this decision, and if a mistake is pretty costly for you, uh, and you have many tasks.
- 14:00
Like, if you have one million tasks and there is a 4% mismatch in performance for real in the models, um, each mistake is, like, $20, uh, like, $12 on average.
- 14:12
Uh, it can cost you, like, hundreds, uh, of thousands of dollars, uh, in a single month. So it can be, like, super costly as a mistake just for a confidence interval being overconfident.
- 14:23
Um, but if you have, like, a reliable way of computing the confidence interval, the method would tell to you, um, "I'm not confident enough to make an informed decision."
- 14:33
And so you can choose, like, to spend more money, to spend more time on evaluating models and avoid the co- costly mistake. So we don't wanna, like, delude ourselves with, like, wrong confidence intervals because there's, uh, you know, money on the table essentially.
- 14:50
And so this is sort of a final checklist of, uh, the things that I've, uh, that I've discussed so far. Uh, so again, to recap, some of the things that are important in building a benchmark are about the environment and some other things are about the metrics.
- 15:06
So on the environment you can follow, uh, the principles, uh, that I described before, like PRISM principles. So some of these things are rather common, but some things like, um, varying initial state across runs, they are pretty rare across existing benchmarks, but they are very important.
- 15:25
And so I would suggest you to try to incorporate these into your evals. Um, and things about the metrics, uh, you can of course read the paper for the details.
- 15:36
But essentially, um, it's very important to avoid, uh, replayability as something that, uh, you can have in your benchmark and also to focus on having accurate confidence intervals. Uh, so respecting the benchmark structure and trying to avoid, um, underestimating the uncertainty overall.
- 15:57
And so I've heard many times sentences like this, "This benchmark can be gamed, but everybody's still using it," or like, "There is no error bar, but I don't see people using them."
- 16:06
So they are sort of, uh, things that we can think when we don't have enough time. But actually a non-rigorous benchmark, uh, is misleading. Um, you know, it can be misleading for the field because everybody could be seeking, you know, um, maximizing a score on a benchmark that maybe is not capturing what we care about.
- 16:25
But especially it can be misleading for, uh, you know, your own decisions. And so if you, um, are deluding yourself on thinking that a score is, like, confident and that it's confidently telling that your model is good, actually you are gonna pay for those mistakes.
- 16:43
And so I think it's very good, uh, usually to be honest with, uh, with yourself and to try to be rigorous in the evaluations that you have. Um, as a last slide, uh, I just started, uh, this company Programma.
- 16:57
And we are building, uh, the best infrastructure for CUA-enabled verification. Uh, and so we are hiring if you're interested or want to chat. Um, this is our website. Thank you very much. [audience applauding] [upbeat music]