AI Engineer World's Fair 2026
Computer-Use 2.0: Agents Just Got Multi-Cursor
Read the talk
Computer-Use 2.0: Background Actions, Verifiable Tasks, and Warm Sandboxes
Running agents beside a human requires more than another cursor: window-level control, trustworthy task evaluation, and infrastructure that keeps GPUs from waiting on desktops.
From a talk by Francesco Bonacci, Dillon DuPont and Robert Wendt
Before you start: Familiarity with agent tool calls, basic Python, and the role of GPUs in model training will help; no prior Cua experience is required.
When the agent needs your desktop
An agent takes a screenshot, reasons about what it sees, and clicks, types, or scrolls. That familiar loop creates a practical problem: the agent operates the same desktop the human wants to use. Drawing on earlier GUI-agent work at Microsoft, Francesco Bonacci calls this screenshot–reason–act pattern computer use 1.0. The opening slide makes it concrete with a PDF export action: observe the interface, identify the next action, and select a click target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cua Driver runs beside the user
Cua Driver moves computer interaction into the background so the agent can work without continually taking over the user's screen. Francesco dates its open-source release to roughly two months before the talk. He connects the initial effort to Codex's computer-use release; the corresponding OpenAI announcement introduced background computer use in the Codex app, rather than a separately named model. The team put its first version together over a weekend.
On macOS, the approach relies on undocumented APIs in Apple's shipped frameworks. The calendar-and-terminal demonstration illustrates the intended result: an agent interacting with an application while the person retains control of the laptop. The slide names this the SkyLight recipe. Francesco describes Driver support across macOS, Windows, and Linux, exposing both accessibility trees and screenshots as ways for an agent to observe the operating system.
After installation, the agent observes a particular window and chooses an action route:
- Call
getWindowStateto obtain an accessibility-tree representation and a screenshot. - Attempt the action through accessibility-based background execution.
- If that fails, attempt a pixel-based background click.
The two observations serve different purposes: the accessibility tree supplies a structured representation of the interface, while the screenshot supports actions located visually. The driver handles some of the operating-system differences, but background behavior is not identical across platforms. The speakers' companion article also describes a final fallback that briefly raises the window and then restores focus, so background operation is a preferred route rather than a guarantee for every action.
Keeping those routes working across releases requires application-level regression tests. Francesco reports approximately eight application harnesses used to check for breakage. He also credits early adopters with upstream contributions: Clicky, Hermes, Qwen Code, H Company, and Factory's Droid.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give each GUI task an initial state and a success check
Once an agent has a way to act, the next question is whether it uses that capability correctly and leaves the environment intact. Dillon DuPont introduces Cua-Bench through a comparison with Terminal-Bench and Harbor: a task needs a reproducible starting point, a known solution, and a way to check the result.
| Task component | Responsibility |
|---|---|
setup | Put the machine into its initial state. |
oracle | Supply a golden trajectory that solves the task. |
evaluator | Probe the environment to check completion. |
The oracle consists of GUI actions, written in a style Dillon compares to PyAutoGUI. The evaluator answers a different question from the oracle: it checks whether the environment satisfies the goal, rather than merely supplying the actions expected to reach it.
A small Python runner makes that separation explicit. Each callback closes over the GUI environment; the same setup and evaluator can check either the oracle or an agent attempt:
python
from collections.abc import Callable
from dataclasses import dataclass
@dataclass
class Task:
setup: Callable[[], None]
oracle: Callable[[], None]
evaluator: Callable[[], bool]
def check(self, attempt: Callable[[], None]) -> bool:
self.setup()
attempt()
return self.evaluator()
def check_oracle(self) -> bool:
return self.check(self.oracle)
This expresses the three-part task structure without requiring the evaluator to depend on a particular action sequence.
The authoring problem is portability. Dillon describes targeting more than five desktop platforms and collapsing GUI construction into a single Python file. The Cua-Bench SDK defines the GUI and probes it for usable agent data; either a person or an agent can author a task. At the time of the talk, Dillon reports over 130 verifiable tasks in 42 environments across five platforms, reproducible through the CLI.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Editing a circuit is easier than starting with a blank schematic
Cua-Bench KiCad, built with Snorkel AI, moves evaluation into professional electrical-engineering software. Here, completing a sequence of clicks is insufficient: the resulting circuit needs a meaningful check. Dillon describes evaluators that simulate circuits. The companion article instead describes exporting netlists and comparing components and connections with reference designs; those are distinct checks, and the supplied descriptions do not establish which tasks use simulation.
Dillon reports that the top tested agent fully passed six of 25 KiCad tasks, all six involving edits to existing schematics. He reports 0% success on tasks starting from a blank schematic. The June evaluation identifies GPT-5.5 as the six-solve model and describes design-from-scratch timeouts within a 200-step budget. The blank-schematic result therefore concerns completion within that budget, rather than an unrestricted claim about whether a model can design circuits.
Across the tested models, Dillon reports that none exceeded 30% reward. Full passes and mean reward are different measurements: partial progress can raise reward without producing another complete solve. In the dated evaluation, Gemini 3.5 Flash leads mean reward while GPT-5.5 leads full solves. The seven models share a 200-step budget, although the Gemini 3.1 Pro and Gemini 3 Flash baselines use function calling rather than native computer use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The computer tool changes the result
Evaluation can also reveal improvements that come from the interface between model and environment. On Cua-Bench basic scaled to 4K resolution, Dillon reports that replacing the built-in computer tool with Cua Driver raised pass rate from approximately 62% to 80% while using 34% fewer tokens. He attributes the gain primarily to focusing on a window instead of the entire desktop. The talk does not identify the model or provide the task count and token-accounting details for this comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try to break the evaluator before trusting its score
A score is useful only if the task's success check deserves trust. Before admitting a task to the dataset, Cua runs a matrix of agents against it, asking them to attempt reward hacking and break the environment. Their findings become a CodeRabbit-style code review. The review screenshot beside the question about trusting an evaluator shows findings and supporting evidence—the output of testing the test itself.
Only tasks that survive that pipeline enter the dataset. Dillon calls this “evals all the way down”: an agent's performance depends on an evaluator, and confidence in that evaluator depends on another layer of adversarial testing.
Successful actions are only one part of intelligence. To test what an agent understands about its environment, Dillon describes forking recorded trajectories and asking a model to predict the reward, internal state, or another observation, then comparing that prediction with the recovered state. The companion makes the reconstruction scope more concrete: captured action and file state at recorded steps, rather than arbitrary restoration of every part of a running machine. This turns a world-model prediction into something that can be checked against a recorded environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A desktop reset can leave an expensive GPU idle
Robert Wendt shifts from measuring agents to keeping their reinforcement-learning infrastructure busy. In the training diagram, the blue portions represent GPUs generating tokens. Once a task finishes in a sandbox, the worker may wait for a fresh sandbox to start or for the existing one to reset. During that interval, the GPU is available but produces no useful work. The worker timelines make the interruption visible as alternating generation and idle periods.
As training scales, those waits compound. Reducing startup time helps, but some desktop environments are inherently substantial: Robert gives the example of a necessary 40-gigabyte research environment that takes time to download and start. The infrastructure problem is therefore how to keep environment preparation from blocking GPU work even when the environment itself cannot start quickly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare sandboxes before workers need them
The solution is a warm sandbox pool. The intended animation does not play, so Robert explains the mechanism verbally: an autoscaler observes how many GPUs currently need sandboxes and grows the pool in response. The operator does not have to choose a fixed warm-pool size upfront.
Demand can change during a multi-day training run. If generations become longer, workers may need new sandboxes less often, reducing the required pool size. Robert says sandbox resources can be two to four times cheaper than GPU resources, making some redundant sandbox capacity economical. The trade is to spend on cheaper readiness so that expensive generation workers spend less time waiting.
With spare environments ready, startup happens on the sandbox infrastructure while GPU workers continue using available environments. Robert describes the result as full GPU-worker utilization and instant sandbox availability; these are workload-dependent claims, with immediate allocation depending on available warm capacity, rather than measured utilization results presented in the talk.
The historical platform list here is Windows, Linux, and Android, with macOS forthcoming. In the transition to questions, Francesco names this infrastructure offering Cua Fleet. Its role is separate from Driver's local interaction layer: Fleet supplies environments to workers, while Driver determines how an agent observes and acts within an environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
On mobile, background work may mean tool use
The closing audience question asks whether background computer use can extend from the Mac to mobile. Francesco points to Android and discussions with the Hermes team about its Android harness. One possible route is to isolate the workload in a container, running an Ubuntu-like environment or a GUI Docker container within Android.
He describes Android as more accommodating than iOS to this form of background operation, but draws a boundary around the capability: it is closer to background tool use than general control of a mobile GUI. The team's approach uses the activity framework to perform that background work. Extending the idea to phones therefore changes the execution mechanism; it does not simply reproduce desktop cursor control on a smaller screen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code for Cua's computer-use drivers, sandbox infrastructure and evaluation tools.
Desktop and mobile evaluation tasks, with links to the registry and current leaderboard.
Further reading
- Computer-Use 2.0Article
The speakers explain window-level control, benchmark evaluation and warm sandbox pools in their companion article.
Seven-model evaluation on 25 KiCad tasks, including the 200-step budget, full solves and partial-credit rewards.
OpenAI's April 2026 announcement of parallel background computer use in the Codex macOS app.
Updates since the talk
Current installation instructions, operating-system requirements and permission setup.
Read the complete timestamped transcript
- 0:00
[upbeat music] Thank you for taking the time for coming over here.
- 0:17
Um, I'm Francesco. I'm the CEO of the company. Uh, alongside me, a couple of other folks, um, my CTO, Dillon, and my chief of infra, Rob. They're gonna walk on the stage in a while.
- 0:27
Uh, but before we do that, who's excited for some computer-using agent talk happening now? Are you guys excited? Lovely. Um, if I were to ask, like, what, what was a computer-using agent like one year ago?
- 0:39
Probably half the crowd would say, "I don't have any idea what really computer use mean." Um, so, um, today I'm gonna take you to a journey, um, basically like from our vision where, um, we come from so far on computer user, um, like this new shape of like agents that are talking, um, and, uh, up to
- 1:04
model intelligence. Um, so we're gonna start like with the vision of, uh, Cua Driver, where we're coming from. And, uh,
- 1:15
if you-- How many of you guys have been work- have been working with computer use for one year?
- 1:22
How about like three years? Lovely. Okay. So our team has plenty of experience. Like we go all the way back, uh, our time on Microsoft. We were working on this type of GUI agents we were calling them, uh, back in the days.
- 1:36
Um, and, uh, there is a, um, there is an example of like old-fashioned human, uh, agent loop. Um, we basically refer, um, refer this as, uh, human, uh, loop, where you will have like an agent loop.
- 1:55
You will have a, um, uh, you will take a screenshot that the agents will have to reason and plan through, and then, uh, um, you will basically work with an action space, uh, in terms of like clicking, typing, scrolling around.
- 2:07
So this is what we refer as, uh, as, um, um, the old-fashioned like computer use 1.0 just to set the tone for, for, um, for this talk. And, uh, we come like a long, um, a long like way since, uh, this type of like, um, computer-using agents.
- 2:27
'Cause this is, this is again, like I'm gonna skim over these slides, but that's like the old-fashioned way of like representing this agents loop as a human would do.
- 2:34
Um, we, um... Here we go. Um, over like two months ago, we released a project in open source. It's called Cua Driver. And, uh,
- 2:51
um, we, we, um, we made it working like in the background. That means that your computer user will ta- will not take over your, uh, screen as like the computer use p- 1.0, um, kind of like agent loop was doing back in the days.
- 3:07
And, uh, um, it all like started from, uh, um, from like a, um, from Codex rel-releasing their computer-user, um, model, uh, two months ago. So we kind of like take the challenge because we were really like working with, uh, these, uh, this type of background computer user.
- 3:26
So over one weekend, uh, we hack something together, and, uh, the trick here is really, um, not like having your agents like take over your screen. So there is a lot of like dark magic like happening behind the wood.
- 3:40
Just to, uh, give you some context, uh, there are like some undocumented API, um, living in, uh, um, in the Apple framework and basically ships with your laptop. And as you can see here, like is in the demo, you have like an AI agent, uh, that is not taking over, uh, control of your, over your laptop.
- 3:58
Um, we made it working not only for macOS but also spanning like across Windows and Linux. We were-- Uh, this is like the very first like driver that is, uh, living on your, um, laptop and, uh, it lets, uh, really any AI agents connect to the underlying operating system, uh, either like using accessibility trees or like a
- 4:20
screenshot-level approach. We kind of like take all both.
- 4:26
Um, this is what really the agency for what it concerns. Um, you will have to install Cua Driver. Um, the agents will, uh, will take a snapshot of the windows state, and, uh, you will have to observe.
- 4:40
Um, and, uh, we really like take, take like one, uh, like different like a, action path to really make the ground computer use happening. So you really, um, have, uh, to observe the space.
- 4:54
Uh, in this case, uh, just by calling like getWindowState, you get, uh, an accessibility tree representation plus a screenshot. And then, uh, you will go and, uh, um, try a background execution using accessibility tree, and if that doesn't work, we go all the way and, uh, make the heavy lifting for you and just try a pixel background
- 5:16
click. This is like, uh, kinda like best step for background at this stage. It's not like behaving the same way on macOS, Windows and Linux. So, uh, we, we do like some of the heavy lifting for you so that your AI agent can, uh, can run undisturbed on your, uh, on your MacBook.
- 5:34
Um, how we manage like to not break anything between like release cycles. We have ex- We have like a lot of investment happening, uh, behind the scene, uh, when we test like new releases.
- 5:49
Uh, we have about like eight different f- uh, application harnesses that, uh, are, um, that, that, that we, that we use for making sure that we don't break anything, uh, among different releases.
- 6:05
Um, among our early adopters, you can see, like, Cliqi, Hermes, Quancode, H. Company, and, uh, DroidFactory. Uh, like, huge thanks to them for using Cua Driver and, like, um, basically releasing a lot of, like, upstream contribution, like, in, uh, in our framework.
- 6:22
Um, without further ado, I'm just gonna move to the next part of the presentation, which is gonna be intelligence. Um, and I'm gonna have our CTO, Dillon, uh, cover that.
- 6:34
Hello. So thank you, Francesco. Um, with Cua Driver, we gave an agent hands. But then the question becomes: How can you trust the agent to use those hands correctly and not leave anything broken behind?
- 6:46
And to answer that, we had to build Cua-Bench.
- 6:49
So for a show of hands, who here has heard of Terminal Bench or Harbor?
- 6:54
Yeah, so a few of y- few of you have, have heard of it. And, uh, if you've ever authored a task for Terminal Bench, then this might look familiar.
- 7:03
But in Cua-Bench, a task is made of three pieces: The setup function, which sets up the machine into initial state; the oracle function, which provides a golden trajectory for the task; and the evaluator, which probes the environment to check if the agent successfully completed the task.
- 7:21
Uh, unlike Terminal Bench, the oracle here is, uh, GUI actions, so it looks kind of like piled GUI when you write that.
- 7:31
And writing environments takes skill and expertise. On desktop, there's more than, uh, five platforms that we target, and, um, we try to collapse that into a single Python file.
- 7:43
So using the Cua-Bench SDK, you can write a GUI that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data.
- 7:55
Anyone or any agent can author one of these tasks.
- 7:59
And when you put that to work, you get a real catalog. We have currently over 130 verifiable tasks, 42 environments, and across five platforms, and each of these are easily reproducible using our CLI.
- 8:13
And the latest addition to our datasets is one that we're proud of. Uh, with collaboration with Snorkel AI, we built Cua-Bench KiCad, which tests computer-use agents on electrical engineering tasks using software by real professionals and evaluator functions that actually simulate the circuits.
- 8:34
But the results are humbling. The top agent that we tested only got a full pass on six out of 25 of these tasks. Of those six, 100% of them involved editing an existing schematic.
- 8:46
And when we start the task from a blank schematic, the success rate drops to 0%.
- 8:54
And across all the models that we tested, the leaderboard is flat. No model has achieved more than 30% reward.
- 9:04
But once you can score something, you can improve it. If we take a look at the Cua-Bench basic dataset, scale it up to 4K resolution, uh, testing an agent, they typically get around 62% pass rate.
- 9:18
But when you switch the agent computer tool from the built-in one to Cua Driver, the pass rate jumps from 62% to 80% using 34% less tokens. And this is primarily because Cua Driver focuses on a window rather than the entire desktop.
- 9:34
But our evals might say you can trust model XYZ at task whatever. But how can you know that the task-- How can you know that the eval can be trusted?
- 9:44
So before we test a task against any agent, we first try to break the environment ourselves. We have a matrix of agents attempt to do reward hacking and attempting to break the environment, and we take all that data and we compile it into a nice code rabbit style code review.
- 10:02
And only tasks that survive our pipeline can enter the dataset. And if you ask us how we trust that agent, the answer is that it's just evals all the way down.
- 10:12
But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions. You also have to measure its ability to understand the world that it's operating in.
- 10:22
Every run that we record can be forked through any moment in its trajectory to give you the state of the computer at that moment. From there, we can probe a model asking to predict the reward, the internal state, or any other observation of the computer and compare it against the fork.
- 10:41
And that prediction is the world model of the agent made measurable. And with that, um, I'll let Robert take the stage.
- 10:50
Thank you, Dillon. Uh, hello, everybody. Uh, I am the chief infra officer at Cua, and I'm here to talk to you about, um, how you're probably leaving a lot of money on the table, uh, with idle GPUs if you do RL training, uh, for computer-use agents.
- 11:07
So I kinda wanna introduce this, uh, diagram to y'all. Uh, can I get like... Is, is there, like, general familiarity with this diagram, or is this, like, something that most of us haven't seen before?
- 11:18
Like, any- anyone? Awesome. Very niche. Um, almost everything on this is not really important for what we're talking about, but the blue portions are. Um, and what those basically represent are GPUs, uh, generating tokens for, um, RL, uh, training.
- 11:37
And if you zoom in on this a little bit, you can kinda see, like, how this typically looks, like, with a sandbox environment, is you're gonna be generating some tokens, um, and then you finish your task on a sandbox, and then you're waiting for either, like, a new sandbox to spin up or for your existing one to
- 11:57
reset. Uh, the problem here is that, like, this is just pure cost. Um, your GPU really isn't doing anything useful here. And, you know, I don't know if you've heard, but GPU time is pretty expensive right now.
- 12:10
So, um, as you're scaling, this cost really compounds a lot, and you really wanna focus on minimizing this if possible. So one thing that you might try to do is, uh, minimize the startup time of your sandbox, and I mean, you should do that.
- 12:27
Like, that's a great thing to do, but, uh, you know, especially for computer use style environments, sometimes this can be a little bit impractical. Um, you know, your researchers might give you like a forty-gigabyte environment, and that might just be necessary, and it takes a long time to pull that down and start it up.
- 12:43
So, you know, how do you, how do you design your training infrastructure so that you can minimize the GPU startup or the minimize the startup time of the sandbox, uh, even when the sandbox is like not well designed to be start up quickly?
- 13:00
Um, so the way we do that... Ah, man, is it not?
- 13:06
So the way we do this is a p-pool, and this is supposed to be animated, but it's not animating. So, um, I guess I'll just explain it to you orally.
- 13:16
And what, what that is is, uh, so we have like a, a set of GPUs here which all want to use a sandbox. And what we will do is that we use a demand-based autoscaler to detect, um, how many GPUs, like, currently need a sandbox, and we can grow the pool to be that size, uh, on demand.
- 13:38
And what that means is that, uh, if you have... Let's say like you have a warm pool that you wanna allocate to your GPU cluster, you don't actually need to know upfront what that warm pool size is.
- 13:50
We can figure out what that warm pool size should be for you, um, on demand. And that might even change over the course of your multi-day training run. Uh, you might start needing a lot of sandboxes, but then as your generations get longer, you might need less.
- 14:05
So these also could be like, you know, easily, uh, two to f-four times cheaper than your GPUs. So having a little bit of redundancy here, uh, you still wind up saving money because you're maximizing the use of your GPU time.
- 14:21
Um, yeah, come see me after if you wanna see the animation 'cause it's, it's cool. Um, so yeah. So now when you have like this like, uh, redundancy in your pool, you're paying the cost of that startup time on the infrastructure side, not on the GPU side.
- 14:38
So your GPU workers have full utilization. Um, yeah. And then because we use this, we can give you instant sandboxes for your GPUs for Windows, Windows, Linux, Android, uh, and macOS is coming up.
- 14:55
Um, and I'm gonna hand it back to Francesco to, uh, close it out for us.
- 15:02
Lovely. Uh, thank you, Dillon. Thank you, Rob, for taking this over. Um, we do have like plenty of time for Q&A, so if you guys like have any questions, like happy to, um, take them either for Cua Driver, Cua Bench, basically what Dillon presented, or Cua Fleet, uh, which is like what, uh, Robert covered.
- 15:30
Any questions? Otherwise, we can wrap this up. Oh, I see.
- 15:35
Would it be possible to operate like, uh, computer-use agents in the background? I know you do this for Mac-
- 15:41
Mm-hmm
- 15:41
... but like in something like, uh, uh, mobile?
- 15:44
Um, so the story for mobile, Android, there is very far you can go. Um, we are talking with the Hermes team 'cause they do have like an harness that runs on Android.
- 15:55
I guess like if you're talking about background, there is some level of like background that can happen if you containerize a workload and basically on Android you can even like run your own container or like sort of like Ubuntu or like GUI Docker container, uh, within An-Android.
- 16:14
Um, but yeah, the Android ecosystem, especially compared to iOS, is more inclined to that form of like background, uh, computer use, but it's more towards like tool use than really like controlling GUI interface.
- 16:27
Um, we work with the activity framework and, uh, do tool use, uh, in the background.
- 16:38
Thank you.
- 16:38
Cool. Thank you, guys. [audience applauding]