Computer-Use 2.0: Agents Just Got Multi-Cursor

An agent takes a screenshot, reasons about what it sees, and clicks, types, or scrolls. That familiar loop creates a practical problem: the agent operates the same desktop the human wants to use. Drawing on earlier GUI-agent work at Microsoft, Francesco Bonacci calls this screenshot–reason–act pattern computer use 1.0. The opening slide makes it concrete with a PDF export action: observe the interface, identify the next action, and select a click target.

Cua Driver moves computer interaction into the background so the agent can work without continually taking over the user's screen. Francesco dates its open-source release to roughly two months before the talk. He connects the initial effort to Codex's computer-use release; the corresponding OpenAI announcement introduced background computer use in the Codex app, rather than a separately named model. The team put its first version together over a weekend.

On macOS, the approach relies on undocumented APIs in Apple's shipped frameworks. The calendar-and-terminal demonstration illustrates the intended result: an agent interacting with an application while the person retains control of the laptop. The slide names this the SkyLight recipe. Francesco describes Driver support across macOS, Windows, and Linux, exposing both accessibility trees and screenshots as ways for an agent to observe the operating system.

After installation, the agent observes a particular window and chooses an action route:

1. Call getWindowState to obtain an accessibility-tree representation and a screenshot.
2. Attempt the action through accessibility-based background execution.
3. If that fails, attempt a pixel-based background click.

The two observations serve different purposes: the accessibility tree supplies a structured representation of the interface, while the screenshot supports actions located visually. The driver handles some of the operating-system differences, but background behavior is not identical across platforms. The speakers' companion article also describes a final fallback that briefly raises the window and then restores focus, so background operation is a preferred route rather than a guarantee for every action.

Keeping those routes working across releases requires application-level regression tests. Francesco reports approximately eight application harnesses used to check for breakage. He also credits early adopters with upstream contributions: Clicky, Hermes, Qwen Code, H Company, and Factory's Droid.

Once an agent has a way to act, the next question is whether it uses that capability correctly and leaves the environment intact. Dillon DuPont introduces Cua-Bench through a comparison with Terminal-Bench and Harbor: a task needs a reproducible starting point, a known solution, and a way to check the result.

The oracle consists of GUI actions, written in a style Dillon compares to PyAutoGUI. The evaluator answers a different question from the oracle: it checks whether the environment satisfies the goal, rather than merely supplying the actions expected to reach it.

A small Python runner makes that separation explicit. Each callback closes over the GUI environment; the same setup and evaluator can check either the oracle or an agent attempt:

from collections.abc import Callable
from dataclasses import dataclass

@dataclass
class Task:
    setup: Callable[[], None]
    oracle: Callable[[], None]
    evaluator: Callable[[], bool]

    def check(self, attempt: Callable[[], None]) -> bool:
        self.setup()
        attempt()
        return self.evaluator()

    def check_oracle(self) -> bool:
        return self.check(self.oracle)

This expresses the three-part task structure without requiring the evaluator to depend on a particular action sequence.

The authoring problem is portability. Dillon describes targeting more than five desktop platforms and collapsing GUI construction into a single Python file. The Cua-Bench SDK defines the GUI and probes it for usable agent data; either a person or an agent can author a task. At the time of the talk, Dillon reports over 130 verifiable tasks in 42 environments across five platforms, reproducible through the CLI.

Cua-Bench KiCad, built with Snorkel AI, moves evaluation into professional electrical-engineering software. Here, completing a sequence of clicks is insufficient: the resulting circuit needs a meaningful check. Dillon describes evaluators that simulate circuits. The companion article instead describes exporting netlists and comparing components and connections with reference designs; those are distinct checks, and the supplied descriptions do not establish which tasks use simulation.

Dillon reports that the top tested agent fully passed six of 25 KiCad tasks, all six involving edits to existing schematics. He reports 0% success on tasks starting from a blank schematic. The June evaluation identifies GPT-5.5 as the six-solve model and describes design-from-scratch timeouts within a 200-step budget. The blank-schematic result therefore concerns completion within that budget, rather than an unrestricted claim about whether a model can design circuits.

Across the tested models, Dillon reports that none exceeded 30% reward. Full passes and mean reward are different measurements: partial progress can raise reward without producing another complete solve. In the dated evaluation, Gemini 3.5 Flash leads mean reward while GPT-5.5 leads full solves. The seven models share a 200-step budget, although the Gemini 3.1 Pro and Gemini 3 Flash baselines use function calling rather than native computer use.

Evaluation can also reveal improvements that come from the interface between model and environment. On Cua-Bench basic scaled to 4K resolution, Dillon reports that replacing the built-in computer tool with Cua Driver raised pass rate from approximately 62% to 80% while using 34% fewer tokens. He attributes the gain primarily to focusing on a window instead of the entire desktop. The talk does not identify the model or provide the task count and token-accounting details for this comparison.

A score is useful only if the task's success check deserves trust. Before admitting a task to the dataset, Cua runs a matrix of agents against it, asking them to attempt reward hacking and break the environment. Their findings become a CodeRabbit-style code review. The review screenshot beside the question about trusting an evaluator shows findings and supporting evidence—the output of testing the test itself.

Only tasks that survive that pipeline enter the dataset. Dillon calls this “evals all the way down”: an agent's performance depends on an evaluator, and confidence in that evaluator depends on another layer of adversarial testing.

Successful actions are only one part of intelligence. To test what an agent understands about its environment, Dillon describes forking recorded trajectories and asking a model to predict the reward, internal state, or another observation, then comparing that prediction with the recovered state. The companion makes the reconstruction scope more concrete: captured action and file state at recorded steps, rather than arbitrary restoration of every part of a running machine. This turns a world-model prediction into something that can be checked against a recorded environment.

Robert Wendt shifts from measuring agents to keeping their reinforcement-learning infrastructure busy. In the training diagram, the blue portions represent GPUs generating tokens. Once a task finishes in a sandbox, the worker may wait for a fresh sandbox to start or for the existing one to reset. During that interval, the GPU is available but produces no useful work. The worker timelines make the interruption visible as alternating generation and idle periods.

As training scales, those waits compound. Reducing startup time helps, but some desktop environments are inherently substantial: Robert gives the example of a necessary 40-gigabyte research environment that takes time to download and start. The infrastructure problem is therefore how to keep environment preparation from blocking GPU work even when the environment itself cannot start quickly.

The solution is a warm sandbox pool. The intended animation does not play, so Robert explains the mechanism verbally: an autoscaler observes how many GPUs currently need sandboxes and grows the pool in response. The operator does not have to choose a fixed warm-pool size upfront.

Demand can change during a multi-day training run. If generations become longer, workers may need new sandboxes less often, reducing the required pool size. Robert says sandbox resources can be two to four times cheaper than GPU resources, making some redundant sandbox capacity economical. The trade is to spend on cheaper readiness so that expensive generation workers spend less time waiting.

With spare environments ready, startup happens on the sandbox infrastructure while GPU workers continue using available environments. Robert describes the result as full GPU-worker utilization and instant sandbox availability; these are workload-dependent claims, with immediate allocation depending on available warm capacity, rather than measured utilization results presented in the talk.

The historical platform list here is Windows, Linux, and Android, with macOS forthcoming. In the transition to questions, Francesco names this infrastructure offering Cua Fleet. Its role is separate from Driver's local interaction layer: Fleet supplies environments to workers, while Driver determines how an agent observes and acts within an environment.

The closing audience question asks whether background computer use can extend from the Mac to mobile. Francesco points to Android and discussions with the Hermes team about its Android harness. One possible route is to isolate the workload in a container, running an Ubuntu-like environment or a GUI Docker container within Android.

He describes Android as more accommodating than iOS to this form of background operation, but draws a boundary around the capability: it is closer to background tool use than general control of a mobile GUI. The team's approach uses the activity framework to perform that background work. Extending the idea to phones therefore changes the execution mechanism; it does not simply reproduce desktop cursor control on a smaller screen.