Training Frontier Models to Out-Think Hackers — Uri Rolls, Arithmetic & Thom Wolf, Hugging Face

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone. Okay, thanks for showing up at this data quality.

2. 0:17

   So as you saw, we'll probably talk a little bit about other things than data quality. And, uh, and first I actually tell you about why I'm very excited about this talk and why actually I, uh, accepted to, uh, to come present it with Uri.

3. 0:30

   Um, there is two reason for that. And the first reason is, uh, I think, and what you will see today, is that cybersecurity is a much wider field for AI, a much wider playing field and exploration field than you might think.

4. 0:45

   And in particular, what we'll show you today is, uh, a benchmark that, uh, Arithmetic and Uri has been developing and I've been a little bit advising, which I think is even close to things like ARC-AGI-3 for people who have been following progress around AGI, in which that, um...

5. 1:02

   By which I mean that this benchmark ask for people-- Oh yeah, cool.

6. 1:07

   Yeah, yeah.

7. 1:07

   To do... If you have been playing with ARC-AGI-3, some of you, who knows here ARC-AGI-3 or ARC-AGI in general?

8. 1:15

   One person? Okay, we're in very data quality, uh, field. [laughs] So basically, this ask models to try to understand what's happening in the world, and it's ac-actually small games that the models need to understand basically what's the current state, how we can play with it, and how we can actually change the state of the game.

9. 1:35

   So, uh, uh, it's, it's actually something you can play yourself. So basically, it ask a mo- it ask a model to understand when you click somewhere, something happening at another place.

10. 1:43

    And you may think this is very simple and that we should be past that, uh, on our way to AGI. And the, the thing you will discover if you play with this benchmark is, no.

11. 1:54

    Like, models have one to two person, uh, success rate on, on this generic benchmark. And the reason is the current model, even though they're really good, they build-- they can't really build a dynamic model of what's happening in the world or what's happening in any type of world.

12. 2:08

    And I think the, the benchmark that, uh, Arithmetic has been developed is a benchmark that's also extremely challenging for a model, in that they need to understand what's happening and to act accordingly to the world model they've been building on the fly.

13. 2:21

    So that's the first reason I think this benchmark is really interesting and why I'm actually, uh, very happy to show you that. And the second reason is, um, I think there's a lot of things that, uh, open source model can bring in cybersecurity, and we tend to have this very binary view of closed source model are good

14. 2:37

    for cyber, open source model are bad. And I think what we want to show today is that, uh, open source model are one part of the solution to, uh, cybersecurity challenges today.

15. 2:48

    Uh, and in particular, if you think in terms of attack and defense and how it will be in the future, this balance, uh, we think that cyber, uh, and open source model used in cybersecurity will be key to actually be able to, to, to, uh, solve the defense solution.

16. 3:04

    So there is a future, I think, where cyber is alive, and every-everyone is well-protected, and I'm pretty sure this future involve open source model. Now, Uri's also a kind of an impressive, uh, impressive person, so I'm really happy I met him.

17. 3:17

    Uh, he was studying at Harvard, dropped to, like, build this idea of what, uh, the future of cyber should be, and, uh, it's an honor to have you on stage with me.

18. 3:27

    Uh, thank you so much, Thomas, and thank you everyone who came, and I'm really grateful. And I'm also grateful for the work we've done together to build this benchmark, which is, uh, I think, incredibly difficult for the models and actually shows some really, really interesting leaps in places that I think we still have way to go.

19. 3:44

    Um, I guess the way we've been thinking about the problem and the reason we set out to do this is it's very clear that the economics of cyber are fundamentally shifting.

20. 3:52

    There's this, um, inherent thing that, that is inherent to cyber, which is that attackers need to choose their resource really wisely. And if you sort of think about cyber as, as a house in a way, then my job is to block every door and close every window and make sure that there's no way in.

21. 4:08

    And the attacker's job is to find at least one seam, one crack, one thing I missed. And then once inside, my job is to put sensors and anything I can to keep them out.

22. 4:16

    Um, and the whole stack, the entire world of cyber that we've been building for the past twenty-ish years has been based on this economics, that the attackers have to choose their targets, and we do everything we can across it to protect ourselves.

23. 4:29

    It is true that that is changing in really dramatic ways. The models are incredibly powerful. Um, they're able to find a ton of primitives. They're able to find a bunch of zero-day exploits.

24. 4:38

    We're seeing this. There's so many [laughs] news and chaos around this point. Um, and on the other hand, it seems like we, as defenders, don't seem to be prepared for, for tho- this world and the way that it's coming.

25. 4:49

    And I, um, actually think that I might be on the, the wrong one. What I wanna show is sort of this idea of, um... So, so if you think about the way cyber has been, this is definitely, I think, the way we've been thinking about AI and cyber for the past many, many, many, uh, months, and it's,

26. 5:08

    it's freaky, and it's getting really scary. And this is my point on the house. And part of the reason we think this is happening is if cyber is this game of skill and speed, then a very skilled attacker using something like [REDACTED:username] can now choose a bunch of targets all at once.

27. 5:23

    That is a, a real reality that is quite scary. Um, and the truth is the picture really doesn't change that much when we move to a really strong open source model as well.

28. 5:31

    So the question is, what do we do in this world where the economics of cyber offense are shifting so much? Um, and part of the problem with the existing stack is that defensive systems have to operate at scale.

29. 5:43

    And that means that we have always very limited human intervention, and so we're sort of bound by what the models can do out of the box. So if we live in this world where the models are becoming so powerful and advancing so fast- We think, uh, like Thomas said, the solution also has to be the models themselves.

30. 5:58

    Um, this is our first solution. We can just all go live in the woods and lock ourselves out. Um, another solution is to figure out how we get the models to be much, much, much more capable.

31. 6:09

    We have no doubt that open source models have to be part of the solution because, um, they allow for many things that we'll talk about as well, and there's this deep need for collaboration.

32. 6:18

    And so what I wanna show you today, and I see the, the clock ticking on me, is, um, sort of this theory that if we've already done this before for coding, we can do this for cyber.

33. 6:28

    And there's a reason to be optimistic, which is really controversial in the context of AI and cyber lately. Um, we've done this before. I think if you go back a year, it was very clear that all of coding was going to be transformed thanks to, um, the models getting better and better and better, and we're sort of

34. 6:43

    seeing the early innings of that with cyber right now, where everyone's talking about cyber and everyone's freaking out. But what if through very high quality, uh, evals, very high quality data, good benchmarks, we could get to a place where the attackers are, um, simply outperformed by very, very, very good defenders?

35. 6:59

    And so our goal in Arithmetic is to be able to get the models to be really capable at cybersecurity to the point where we can rebuild this new stack, um, that's based on the models, uh, winning the models on the other side.

36. 7:12

    So I'm really excited to show you Mask Off. It's our first benchmark that we're releasing. Um, our first fundamental idea is that we can't capture all of cyber in one singular benchmark.

37. 7:22

    That's a bit like saying that swimming, an F1 driver, and a basketball player is the same thing. It doesn't work. Um, and so we focus specifically on access control.

38. 7:30

    Really quickly, why access control? Um, it's sort of the first door to any target in cyber, begins with my ability to get a foothold. So if you think about an attacker on the one hand, they're trying to get to some privileged thing.

39. 7:42

    Uh, if Thomas and I are working in the same company, I'm some ML engineer, he's an admin, uh, what are the things-- Can I find a way to do things that I'm not allowed to do, uh, in my current privileged position?

40. 7:53

    It actually leads to being the number one on the OS list, and has created this sort of thirty billion dollar industry. Um, and for years, these are number one vulnerabilities.

41. 8:01

    The reason they exist, and this is to the RKGI point, these are logic-based vulnerabilities, so it's not just about bugs in the code that I find and I need to patch.

42. 8:10

    It's about very, very, very big systems, and somewhere between them, there's these logic breaks where it's possible that one thing checks for something specific in the code, another checks for something else, and that sort of leads to everything breaking.

43. 8:21

    And so what we're trying to do is we're trying to figure out how we get the models to really reason very, very hard and not just do pattern matching.

44. 8:28

    That's where a lot of work goes into the data quality. Uh, our data is created first by humans. I think that's really important right now. To find out the distribution things, we need humans to go do the search.

45. 8:38

    Our team is all, uh, based on very deep vulnerability researchers and nerds who love to hack, who are trying to get really, really good at cyber-- at, at AI capabilities.

46. 8:47

    So we find our own zero-days in widely, uh, distributed open-source software. We use that to create these ta-- these real live, huge environments of a bunch of different, um, applications chained together.

47. 8:59

    That then allows us to basically create this black box setting where the model doesn't see the code, and it doesn't know about the zero-day, 'cause we found it ourselves, and it has to find a way, uh, to reason across this entire surface and understand exactly what the exploitation is.

48. 9:13

    And so to do that, we don't give it access to the internet or the code base, but we do give it sort of all the basic tooling it would need to be able to execute a task well.

49. 9:21

    And everything, because the tasks are so difficult, everything has a deterministic grader. And so across the entire exploitation and the discovery chain, every single step can be deterministically verified, allowing us to see how deep it got within the chain.

50. 9:34

    And finally, this is credit to Eugene from Anthropic, who we slightly stole this graphic from, but it really does capture really well the way we've set up our evals, where basically you have, uh, inputs on the one hand based on our real zero-days, the agents, so we're thinking about it as the model plus its harness, um, and

51. 9:49

    some black box tooling that it has to find. Then we have a verifiable grader, which is a binary pass. Was the, uh, model able to do something it wasn't allowed to do as an underprivileged user?

52. 9:58

    And then we have our deterministic grading in every step along the way. So what I wanna show you now is an illustration. For all the security buffs in the room, it's an illustration, guys.

53. 10:09

    Um, but the idea is how a real solve looks, and fundamentally what we have here is a real, uh, task of ours, where it's a chain between Keycloak, Vault, and a broker, and I start as a very low privileged user.

54. 10:21

    I need to figure out how to get to production code. So what we're gonna see is the solve, each one of our tasks has a solve script, of what a real solution looks like.

55. 10:28

    There's a real zero-day that we found that we submitted for verification to the maintainers where, um, there's a check whether I'm an admin or not. It only checks my name, and another, uh, aspect of this checks whether-- by ID.

56. 10:40

    So that allows me as a user to change the name of, um, the real admin, inherit their, um, their, uh, privilege, and then use that to escalate myself. And what I'm really trying to illustrate is sort of this long chain, sixteen step type of logic that the model has to do, and if it's not able to understand

57. 10:59

    inherently the system, it's way too bi-- uh, wide for it to test everything, sort of shoot across the space. So what we're gonna see now is a real attempt by, uh, GPT-5.5 and then Opus as well, trying to solve this task.

58. 11:11

    And what you're going to see is a sort of chaotic, trying everything, ha-- jumping between everything, probing a bunch of different stuff. It does even reach the check, but it never makes the logical leap that it's supposed to f-be able to change the admin's own permission, uh, their own name, in order to bypass this permissioning.

59. 11:27

    So this is exactly what we're trying to test. Can the model understand, uh, leaps, logic leaps that are inherent to the system? Uh, a really important point, just like RKGI, everything you do in a live system in permissioning changes other stuff in the system.

60. 11:41

    So the model needs to be able to hold this model of the world it's living in and iterate through it. Um, and then fundamentally at the end of this, it writes code.

61. 11:50

    It writes exploitation code and it needs to be able to do, um, to reason really, really thinly and understand exactly what the exploitation is in order to be able to execute.

62. 11:58

    And we can really see the difference between models that have succeeded some of the tests and models that haven't. And then now we're gonna do something that I've been told to never do, which is show a live demo of a real system on stage [laughs].

63. 12:10

    And so let's hope, um, let's hope we don't get, uh, there, and I, I know we're basically out of time. So what we're seeing here is what's called Bach.

64. 12:18

    It's our internal system. It's an orchestrator. This is how we run our actual eval. Um, what we see is the actual results of the benchmark. The benchmark right now is incredibly hard.

65. 12:27

    There's only one solve at K1. Um, and then at K5, uh, there is, um, it, it remains only GPT in the public models is, is able to solve this.

66. 12:36

    That's why the partial graders are so critical, to be able to really see what the model is able to do and what-- and how deep within the exploitation chain they can get.

67. 12:44

    Um, I'm going to load quickly the sort of the way we think about these environments, which is because we're looking for performance over time, we really measure how capable is the model at making specific leaps.

68. 12:56

    And so what you'll see is the results of exploitation on a specific one of our environments. Um, and then if we zoom in, then you can really see how sort of GPT-5.5 is the only model that's able to make this leap.

69. 13:09

    The mo- other models sort of have been able to reason across everything. If you look at the discovery phase, they do capture, uh, nearly all the different information they need, and they never are able to make the leap into what is the exploitation they need to do.

70. 13:21

    This is exactly the type of capability that we believe if every model in the world could get really, really, really good at doing this and very fast, that should give a lasting defense, uh, and capabilities to the defenders that the attackers simply don't have right now.

71. 13:34

    Um, and then finally, I think the way we sort of reason through these and work through them, um, is, is quite cool, and I wanted to show you. So again, if we go into one of our tasks called Fall- Falltime, we can really see sort of the way [laughs] we spend our days, which is really, really, really trying

72. 13:53

    to understand what are the specific failure modes a model does. You can see that this is long horizon, not because it's waiting for code to run. It's constantly working over three hours, and it's still been unable to solve the tasks.

73. 14:04

    And then I guess the way I spend all my day is, is quite literally going through all the traces of what the models did, why, and how. Um, and yeah, and I think my -- I think the final point I wanna make, and I'll pass it back to Thomas, is sort of

74. 14:18

    next, what if we can, uh, not just have this chart, which is really cool, sort of have this chart, um, of a really super cool future model that's very, very fast in its understanding of what the capability leaps need to be.

75. 14:34

    Uh, I really believe that with everything happening now, it is critical that we get cyber capabilities to the point where, um, we can defend much, much faster. The only way to replace the old stack is through the models.

76. 14:47

    Um, and I'll pass it back to Thomas. I think also the only way to do that is through a real array of strong open source models and collaboration that we can post train on, and that we can post train to each network and to each environment as well.

77. 14:58

    Yeah. So as you saw, this benchmark is quite different from [REDACTED:username] type of, uh, we read the code, and we find the vulnerabilities. Here, basically the model is operating in a real environment where, you know, there is a authentication place somewhere.

78. 15:11

    It doesn't know what's there. Then there's another, like, network, like, um, like microservices you need to access and use, and it has basically zero information of that. And it all start, like you show, by a zero, zero-day vulnerability that we have.

79. 15:25

    So everything is new. Uh, so I think there's, there's a lot of research, uh, to be done, understand how models can actually, uh, understand and work on that. And so the first step is getting some good data.

80. 15:36

    So the idea is to have some good benchmarks, then have some good data on how this is operating. And I think the second step is being able to fine-tune models and try to understand how we can pro-protect against that, right?

81. 15:46

    And, and a big challenge here is gonna be speed, uh, so Uri told that, uh, several times. Gonna be the speed of attacker versus defense, right? When they start to enter, you have to be able to see what's happening and then catch them.

82. 15:58

    And speed will be where, you know, you, you want to have a specialized model that's maybe running on specialized hardware and actually is, is gonna be very important. And here, I think the, the danger is to say, we're just gonna rely on two company that everyone knows here to solve all of that for us.

83. 16:13

    I think the solution is just to take our future and say, "Well, it's gonna be a speed challenge, and we're gonna train our model. We're gonna run them fast and make them available to basically every company who wants to be protected."

84. 16:24

    So exciting. I would say it's, uh, as everything in cybersecurity, it's both very interesting, but also a, a big challenge and, and, um, well, something you, you have to not mess up, I would say.

85. 16:39

    Awesome. Um, yeah. Thank you very much. Um, [audience clapping] for anyone that wants to collaborate, work on this work, we have a forum. Uh, we are going to, uh, work with people on this data directly.

86. 16:52

    We'd love to hear from you. Anyone who's really passionate about any other field in cyber, like Thomas said, we have to do this across sort of not just access control, a bunch of different things.

87. 17:01

    And, uh, the only way this is gonna work is through a lot of, um, a lot of post-training data and really, really capable models.

88. 17:07

    Congrats for your first presentation.

89. 17:08

    Thanks. [laughs] [audience clapping] [upbeat instrumental music]