Computer-Use 2.0: Agents Just Got Multi-Cursor

Talk transcript

1. 0:00

   [upbeat music] Thank you for taking the time for coming over here.

2. 0:17

   Um, I'm Francesco. I'm the CEO of the company. Uh, alongside me, a couple of other folks, um, my CTO, Dillon, and my chief of infra, Rob. They're gonna walk on the stage in a while.

3. 0:27

   Uh, but before we do that, who's excited for some computer-using agent talk happening now? Are you guys excited? Lovely. Um, if I were to ask, like, what, what was a computer-using agent like one year ago?

4. 0:39

   Probably half the crowd would say, "I don't have any idea what really computer use mean." Um, so, um, today I'm gonna take you to a journey, um, basically like from our vision where, um, we come from so far on computer user, um, like this new shape of like agents that are talking, um, and, uh, up to

5. 1:04

   model intelligence. Um, so we're gonna start like with the vision of, uh, Cua Driver, where we're coming from. And, uh,

6. 1:15

   if you-- How many of you guys have been work- have been working with computer use for one year?

7. 1:22

   How about like three years? Lovely. Okay. So our team has plenty of experience. Like we go all the way back, uh, our time on Microsoft. We were working on this type of GUI agents we were calling them, uh, back in the days.

8. 1:36

   Um, and, uh, there is a, um, there is an example of like old-fashioned human, uh, agent loop. Um, we basically refer, um, refer this as, uh, human, uh, loop, where you will have like an agent loop.

9. 1:55

   You will have a, um, uh, you will take a screenshot that the agents will have to reason and plan through, and then, uh, um, you will basically work with an action space, uh, in terms of like clicking, typing, scrolling around.

10. 2:07

    So this is what we refer as, uh, as, um, um, the old-fashioned like computer use 1.0 just to set the tone for, for, um, for this talk. And, uh, we come like a long, um, a long like way since, uh, this type of like, um, computer-using agents.

11. 2:27

    'Cause this is, this is again, like I'm gonna skim over these slides, but that's like the old-fashioned way of like representing this agents loop as a human would do.

12. 2:34

    Um, we, um... Here we go. Um, over like two months ago, we released a project in open source. It's called Cua Driver. And, uh,

13. 2:51

    um, we, we, um, we made it working like in the background. That means that your computer user will ta- will not take over your, uh, screen as like the computer use p- 1.0, um, kind of like agent loop was doing back in the days.

14. 3:07

    And, uh, um, it all like started from, uh, um, from like a, um, from Codex rel-releasing their computer-user, um, model, uh, two months ago. So we kind of like take the challenge because we were really like working with, uh, these, uh, this type of background computer user.

15. 3:26

    So over one weekend, uh, we hack something together, and, uh, the trick here is really, um, not like having your agents like take over your screen. So there is a lot of like dark magic like happening behind the wood.

16. 3:40

    Just to, uh, give you some context, uh, there are like some undocumented API, um, living in, uh, um, in the Apple framework and basically ships with your laptop. And as you can see here, like is in the demo, you have like an AI agent, uh, that is not taking over, uh, control of your, over your laptop.

17. 3:58

    Um, we made it working not only for macOS but also spanning like across Windows and Linux. We were-- Uh, this is like the very first like driver that is, uh, living on your, um, laptop and, uh, it lets, uh, really any AI agents connect to the underlying operating system, uh, either like using accessibility trees or like a

18. 4:20

    screenshot-level approach. We kind of like take all both.

19. 4:26

    Um, this is what really the agency for what it concerns. Um, you will have to install Cua Driver. Um, the agents will, uh, will take a snapshot of the windows state, and, uh, you will have to observe.

20. 4:40

    Um, and, uh, we really like take, take like one, uh, like different like a, action path to really make the ground computer use happening. So you really, um, have, uh, to observe the space.

21. 4:54

    Uh, in this case, uh, just by calling like getWindowState, you get, uh, an accessibility tree representation plus a screenshot. And then, uh, you will go and, uh, um, try a background execution using accessibility tree, and if that doesn't work, we go all the way and, uh, make the heavy lifting for you and just try a pixel background

22. 5:16

    click. This is like, uh, kinda like best step for background at this stage. It's not like behaving the same way on macOS, Windows and Linux. So, uh, we, we do like some of the heavy lifting for you so that your AI agent can, uh, can run undisturbed on your, uh, on your MacBook.

23. 5:34

    Um, how we manage like to not break anything between like release cycles. We have ex- We have like a lot of investment happening, uh, behind the scene, uh, when we test like new releases.

24. 5:49

    Uh, we have about like eight different f- uh, application harnesses that, uh, are, um, that, that, that we, that we use for making sure that we don't break anything, uh, among different releases.

25. 6:05

    Um, among our early adopters, you can see, like, Cliqi, Hermes, Quancode, H. Company, and, uh, DroidFactory. Uh, like, huge thanks to them for using Cua Driver and, like, um, basically releasing a lot of, like, upstream contribution, like, in, uh, in our framework.

26. 6:22

    Um, without further ado, I'm just gonna move to the next part of the presentation, which is gonna be intelligence. Um, and I'm gonna have our CTO, Dillon, uh, cover that.

27. 6:34

    Hello. So thank you, Francesco. Um, with Cua Driver, we gave an agent hands. But then the question becomes: How can you trust the agent to use those hands correctly and not leave anything broken behind?

28. 6:46

    And to answer that, we had to build Cua-Bench.

29. 6:49

    So for a show of hands, who here has heard of Terminal Bench or Harbor?

30. 6:54

    Yeah, so a few of y- few of you have, have heard of it. And, uh, if you've ever authored a task for Terminal Bench, then this might look familiar.

31. 7:03

    But in Cua-Bench, a task is made of three pieces: The setup function, which sets up the machine into initial state; the oracle function, which provides a golden trajectory for the task; and the evaluator, which probes the environment to check if the agent successfully completed the task.

32. 7:21

    Uh, unlike Terminal Bench, the oracle here is, uh, GUI actions, so it looks kind of like piled GUI when you write that.

33. 7:31

    And writing environments takes skill and expertise. On desktop, there's more than, uh, five platforms that we target, and, um, we try to collapse that into a single Python file.

34. 7:43

    So using the Cua-Bench SDK, you can write a GUI that works across every desktop platform in a single Python file and use the same SDK to probe that GUI to get usable agent data.

35. 7:55

    Anyone or any agent can author one of these tasks.

36. 7:59

    And when you put that to work, you get a real catalog. We have currently over 130 verifiable tasks, 42 environments, and across five platforms, and each of these are easily reproducible using our CLI.

37. 8:13

    And the latest addition to our datasets is one that we're proud of. Uh, with collaboration with Snorkel AI, we built Cua-Bench KiCad, which tests computer-use agents on electrical engineering tasks using software by real professionals and evaluator functions that actually simulate the circuits.

38. 8:34

    But the results are humbling. The top agent that we tested only got a full pass on six out of 25 of these tasks. Of those six, 100% of them involved editing an existing schematic.

39. 8:46

    And when we start the task from a blank schematic, the success rate drops to 0%.

40. 8:54

    And across all the models that we tested, the leaderboard is flat. No model has achieved more than 30% reward.

41. 9:04

    But once you can score something, you can improve it. If we take a look at the Cua-Bench basic dataset, scale it up to 4K resolution, uh, testing an agent, they typically get around 62% pass rate.

42. 9:18

    But when you switch the agent computer tool from the built-in one to Cua Driver, the pass rate jumps from 62% to 80% using 34% less tokens. And this is primarily because Cua Driver focuses on a window rather than the entire desktop.

43. 9:34

    But our evals might say you can trust model XYZ at task whatever. But how can you know that the task-- How can you know that the eval can be trusted?

44. 9:44

    So before we test a task against any agent, we first try to break the environment ourselves. We have a matrix of agents attempt to do reward hacking and attempting to break the environment, and we take all that data and we compile it into a nice code rabbit style code review.

45. 10:02

    And only tasks that survive our pipeline can enter the dataset. And if you ask us how we trust that agent, the answer is that it's just evals all the way down.

46. 10:12

    But to measure the intelligence of an agent, you can't just measure its ability to successfully perform actions. You also have to measure its ability to understand the world that it's operating in.

47. 10:22

    Every run that we record can be forked through any moment in its trajectory to give you the state of the computer at that moment. From there, we can probe a model asking to predict the reward, the internal state, or any other observation of the computer and compare it against the fork.

48. 10:41

    And that prediction is the world model of the agent made measurable. And with that, um, I'll let Robert take the stage.

49. 10:50

    Thank you, Dillon. Uh, hello, everybody. Uh, I am the chief infra officer at Cua, and I'm here to talk to you about, um, how you're probably leaving a lot of money on the table, uh, with idle GPUs if you do RL training, uh, for computer-use agents.

50. 11:07

    So I kinda wanna introduce this, uh, diagram to y'all. Uh, can I get like... Is, is there, like, general familiarity with this diagram, or is this, like, something that most of us haven't seen before?

51. 11:18

    Like, any- anyone? Awesome. Very niche. Um, almost everything on this is not really important for what we're talking about, but the blue portions are. Um, and what those basically represent are GPUs, uh, generating tokens for, um, RL, uh, training.

52. 11:37

    And if you zoom in on this a little bit, you can kinda see, like, how this typically looks, like, with a sandbox environment, is you're gonna be generating some tokens, um, and then you finish your task on a sandbox, and then you're waiting for either, like, a new sandbox to spin up or for your existing one to

53. 11:57

    reset. Uh, the problem here is that, like, this is just pure cost. Um, your GPU really isn't doing anything useful here. And, you know, I don't know if you've heard, but GPU time is pretty expensive right now.

54. 12:10

    So, um, as you're scaling, this cost really compounds a lot, and you really wanna focus on minimizing this if possible. So one thing that you might try to do is, uh, minimize the startup time of your sandbox, and I mean, you should do that.

55. 12:27

    Like, that's a great thing to do, but, uh, you know, especially for computer use style environments, sometimes this can be a little bit impractical. Um, you know, your researchers might give you like a forty-gigabyte environment, and that might just be necessary, and it takes a long time to pull that down and start it up.

56. 12:43

    So, you know, how do you, how do you design your training infrastructure so that you can minimize the GPU startup or the minimize the startup time of the sandbox, uh, even when the sandbox is like not well designed to be start up quickly?

57. 13:00

    Um, so the way we do that... Ah, man, is it not?

58. 13:06

    So the way we do this is a p-pool, and this is supposed to be animated, but it's not animating. So, um, I guess I'll just explain it to you orally.

59. 13:16

    And what, what that is is, uh, so we have like a, a set of GPUs here which all want to use a sandbox. And what we will do is that we use a demand-based autoscaler to detect, um, how many GPUs, like, currently need a sandbox, and we can grow the pool to be that size, uh, on demand.

60. 13:38

    And what that means is that, uh, if you have... Let's say like you have a warm pool that you wanna allocate to your GPU cluster, you don't actually need to know upfront what that warm pool size is.

61. 13:50

    We can figure out what that warm pool size should be for you, um, on demand. And that might even change over the course of your multi-day training run. Uh, you might start needing a lot of sandboxes, but then as your generations get longer, you might need less.

62. 14:05

    So these also could be like, you know, easily, uh, two to f-four times cheaper than your GPUs. So having a little bit of redundancy here, uh, you still wind up saving money because you're maximizing the use of your GPU time.

63. 14:21

    Um, yeah, come see me after if you wanna see the animation 'cause it's, it's cool. Um, so yeah. So now when you have like this like, uh, redundancy in your pool, you're paying the cost of that startup time on the infrastructure side, not on the GPU side.

64. 14:38

    So your GPU workers have full utilization. Um, yeah. And then because we use this, we can give you instant sandboxes for your GPUs for Windows, Windows, Linux, Android, uh, and macOS is coming up.

65. 14:55

    Um, and I'm gonna hand it back to Francesco to, uh, close it out for us.

66. 15:02

    Lovely. Uh, thank you, Dillon. Thank you, Rob, for taking this over. Um, we do have like plenty of time for Q&A, so if you guys like have any questions, like happy to, um, take them either for Cua Driver, Cua Bench, basically what Dillon presented, or Cua Fleet, uh, which is like what, uh, Robert covered.

67. 15:30

    Any questions? Otherwise, we can wrap this up. Oh, I see.

68. 15:35

    Would it be possible to operate like, uh, computer-use agents in the background? I know you do this for Mac-

69. 15:41

    Mm-hmm

70. 15:41

    ... but like in something like, uh, uh, mobile?

71. 15:44

    Um, so the story for mobile, Android, there is very far you can go. Um, we are talking with the Hermes team 'cause they do have like an harness that runs on Android.

72. 15:55

    I guess like if you're talking about background, there is some level of like background that can happen if you containerize a workload and basically on Android you can even like run your own container or like sort of like Ubuntu or like GUI Docker container, uh, within An-Android.

73. 16:14

    Um, but yeah, the Android ecosystem, especially compared to iOS, is more inclined to that form of like background, uh, computer use, but it's more towards like tool use than really like controlling GUI interface.

74. 16:27

    Um, we work with the activity framework and, uh, do tool use, uh, in the background.

75. 16:38

    Thank you.

76. 16:38

    Cool. Thank you, guys. [audience applauding]