Computer Use at the Edge of the Statistical Precipice

Run a frontier model on a computer-use benchmark, keep a successful trajectory for each task, and turn those trajectories into a new agent. When a task arrives, this replay agent blindly repeats the recorded taps, typing and scrolling. It does not need to inspect the screen or decide what to do next. This is the opening experiment in Pierluca D’Oro’s Computer Use at the Edge of the Statistical Precipice, work conducted with collaborators at Meta Superintelligence Labs. For benchmarks with hundreds of tasks, he describes the resulting script as smaller than one megabyte.

In the presented experiments on OSWorld and MobileWorld, blind replay matches or exceeds the success rate of the frontier model that supplied its traces. The MobileWorld comparison uses Gemini 3 Pro as the source model and covers 96 of 201 tasks, rather than the full suite. These are the paper’s experimental results, not claims about today’s benchmark leaderboards.

The result is possible because the environment repeats itself. With a fixed starting state and deterministic execution, an action sequence that succeeded once can succeed again without adaptation. A high score can therefore measure reusable successful traces rather than robust computer use. The replay agent makes that weakness visible without requiring a sophisticated attack.

pass@k measures the probability that at least one of k attempts succeeds. That sounds useful: give an agent several chances and ask whether it can solve the task at all. But in a deterministic environment, finding one successful attempt also supplies a tape that can be replayed. The paper formalizes this connection: with fixed initial states and deterministic execution, the expected success of a replay agent extracted from k independent source rollouts equals the source agent’s pass@k. The expectation and execution conditions matter; this is not a claim that any recorded trajectory works in any environment.

Successful-trace selection consequently rewards the same property that made the blind script competitive. It establishes that a reusable success can be found, without establishing that the agent can cope with changed circumstances. Two separate design problems follow: build environments without this exploitable structure, and use evaluation metrics whose statistics do not create false confidence.

The first defense against replay is to make the benchmark multifactorial. Vary the data, appearance and initial state so that the agent cannot assume every run will present the same sequence of screens. Yet variation creates its own obligation: every generated combination must still describe a valid task. Otherwise, an apparent agent failure may actually be an environment failure.

Variation and validity sit alongside three other requirements: isolate the environment, provide the evaluator with privileged information for checking outcomes, and reproduce the relevant behavior of real systems faithfully. Together these form the PRISM principles.

Existing benchmarks satisfy different subsets of these requirements. The aim is to combine them: diversity is useful only when tasks remain valid, and a valid task is useful only when its outcome means something beyond the benchmark.

DigiWorld is the team’s attempt to meet all five principles with a collection of Android applications. Its reported scope is 15 apps, 387 verified scenarios and more than 3.2 million verified configurations. Those figures describe the available benchmark space, not an exhaustive evaluation of every agent on every configuration.

A task such as sending money can vary along several axes:

• Instance: change the amount to send or the task’s other specific parameters.
• Data profile: change the contacts, emails or other records available to the agent.
• Theme: change the application’s appearance.
• Starting screen: begin at login or at another valid page.

These factors multiply. Even a modest number of choices per factor creates a large space of combinations. Scaling the choices could produce billions of configurations; that is a growth possibility, distinct from the reported benchmark scope.

Because these axes are implemented in software, coding agents can help expand them—for example, by generating new instances or themes. But generating more software does not automatically produce a better evaluation environment. Every new combination also creates another opportunity for a task, its data and its interface to disagree.

Verification is what makes configuration generation scalable. Generate candidate combinations, reject the broken ones and retain valid configurations. DigiWorld implements this through a system resembling a compiler:

1. Start with a parameterized task template, such as sending a specified amount to a specified recipient.
2. Pair the template with a verifier for that task.
3. Supply the mock data required for the task to be possible.
4. Combine those inputs with base data and a base UI state to produce a valid configuration.

The task text, outcome checker and environment must agree on what success means.

A small Python example illustrates the binding between a payment task and its verifier. Here, the generated instruction and the outcome check derive from the same amount_cents and recipient_id; compiling the configuration does not execute the payment.

from dataclasses import dataclass

@dataclass(frozen=True)
class PaymentTask:
    amount_cents: int
    recipient_id: str

    def compile(self, contacts: dict[str, str]) -> dict:
        if self.amount_cents <= 0:
            raise ValueError("Payment amount must be positive")
        if self.recipient_id not in contacts:
            raise ValueError("Recipient is absent from mock data")

        name = contacts[self.recipient_id]
        return {
            "instruction": (
                f"Send ${self.amount_cents / 100:.2f} to {name}."
            ),
            "contacts": dict(contacts),
            "starting_screen": "home",
        }

    def verify(self, new_payments: list[dict]) -> bool:
        return any(
            payment["amount_cents"] == self.amount_cents
            and payment["recipient_id"] == self.recipient_id
            and payment["status"] == "completed"
            for payment in new_payments
        )

task = PaymentTask(amount_cents=2500, recipient_id="contact_7")
configuration = task.compile({"contact_7": "Maya"})

The broader engineering problem includes validating the UI state and the interactions among factors. The compiler-like approach makes those checks part of environment construction, so diversity and correctness grow together.

With verified variation in place, repeat the original experiment: evaluate a frontier model, extract its successful traces and evaluate the corresponding replay agent. D’Oro reports that replay agents now achieve little success. Some residual success is expected because certain tasks are naturally repeatable. The result addresses this particular replay exploit; it does not establish immunity to every way of gaming a benchmark.

The same infrastructure then supports a more useful question: how robust is the frontier model across each variation axis? A model that succeeds on a task might be expected to keep succeeding when only the theme or starting screen changes. In the reported experiments, that expectation often fails, with especially weak performance in the worst cases. Separating these axes makes it possible to measure fragility that a single base configuration hides.

A diverse environment fixes only half the evaluation problem. Its score still needs an honest estimate of uncertainty, and there are two sources of variation to capture:

• Action variation: repeated runs of the model can choose different actions and follow different trajectories.
• Environment variation: different configurations change the conditions under which those trajectories unfold.

Running the model repeatedly on one base configuration measures the first source while leaving the second largely unexplored. Deployment encounters both. The paper’s methodology therefore accounts for the structure of the benchmark rather than treating all uncertainty as rollout noise.

The diagnostic is confidence-interval coverage. Across repeated evaluations, a nominal 95% interval should contain the target performance approximately 95% of the time. Coverage is a property of the interval procedure; it is not the agent’s task success rate.

In the paper’s calibration simulation, the comparison is:

The simulation uses 15 apps, eight scenarios per app, 27 configurations per scenario and three rollouts, across 200 experiments with 500 bootstrap replicates. D’Oro describes the rollout-only result as roughly 17–20% coverage. Accounting for configuration variation helps, but resampling at the scenario level is also needed to reach approximately nominal coverage. The estimator’s details are in the companion paper; the practical distinction is between narrow intervals and intervals that actually cover the intended performance.

Suppose an evaluation compares model A with model B to decide which to deploy. Very narrow confidence intervals can make the choice look settled. In the presentation’s example, the orange bars represent real performance, and they expose a choice that the apparently precise estimates got wrong. Precision around the wrong estimate is not useful certainty.

The cost example assumes one million tasks in a month, a four-percentage-point performance gap and an average cost of $12 per additional error. Those assumptions imply 40,000 additional errors and $480,000 in monthly cost.

This is illustrative arithmetic, not a reported deployment loss. It explains why a seemingly small model-selection error can justify substantial evaluation effort.

A reliable interval may instead reveal that the evidence is insufficient to choose confidently. That result has operational value: spend more time or money evaluating before committing to the more expensive mistake. An evaluation can be useful because it tells you that you do not yet know.

These requirements belong in the evaluation’s design, before a score becomes a deployment decision. Apply PRISM to the environment, including initial-state variation—a practice D’Oro describes as uncommon despite its importance. Test whether successful traces can be replayed, and compute confidence intervals that respect the benchmark’s structure rather than understating total uncertainty.

Time pressure and prevailing practice make weak evaluation easy to accept: a benchmark is gameable, but widely used; error bars are missing, but others omit them too. Neither convention makes the evidence reliable. A misleading benchmark can direct a whole field toward maximizing a score that misses the capability it intended to measure. For an individual team, the same false confidence can turn directly into costly deployment decisions.

D’Oro closes by introducing Programma, describing its work on infrastructure for CUA-enabled verification and inviting hiring conversations. That closing connects the evaluation problem to the infrastructure needed to check software through computer use.