Safety and security for code-executing agents

Talk transcript

1. 0:00

   [on-hold music] Hi, everyone.

2. 0:15

   I'm Fouad, and I'm here to talk about safety and security for code-executing agents. And a little intro about myself. I actually started on the OpenAI Security team, um, after running a startup for about six years, a security company.

3. 0:29

   Um, and now I work on Agent Robustness and Control as part of post-training. Uh, one of the things I did in the last couple of months is work on Codex and Codex CLI, which is our open source library for actually running Codex directly on your computer.

4. 0:43

   And there's a lot of things we learned in building Codex that I'm excited to share with you all. But, um, there's definitely a lot more work for us to do, and excited to hear what you think, um, afterwards.

5. 0:53

   Um, one high-level point I want to start with is that every frontier research lab is focusing on how to push the benchmarks around coding, and not just the benchmarks, but also usability and actually deployability of these agents.

6. 1:05

   So they're making them really good at writing and executing code, and as a result, every agent will become a code-executing agent. It's not just ab- actually about writing code, but it's about achieving the objective most efficiently.

7. 1:17

   And if you look at where the models were just even a, a year ago or under a year ago, um, with, with o1, it showed us a very early preview of what these reasoning models can do.

8. 1:27

   But with more recent models like o3, o4-mini, and other models in the space, um, you can see higher reliability and more capabilities. And now the new constraint isn't just can these models do things, but actually what should they be able to do and what should the guardrails be when you allow them, um, to work, um, in your

9. 1:43

   environments. And as I mentioned, code isn't just for SWE tasks, which is candidly what I thought initially when I, when I started at OpenAI, um, but it actually helps across the stack.

10. 1:53

    Here's an example, um, from our o3 release around multimodal reasoning, where, um, previously o1 would, you know, look at the image and try to just reason about it based on the image as it's given.

11. 2:03

    But what we've noticed with code-executing agents, even outside of a SWE scenario, they're able to actually run code to decipher the text that's on the page using OCR or to crop images.

12. 2:15

    There's some really exciting, um, behaviors that we've seen from models when you just give them the ability to run code. Uh, we didn't tell it in this prompt that it should run code.

13. 2:23

    It just knew that with that tool as an option, it's able to do it more efficiently.

14. 2:29

    And what we'll, I think, observe when it comes to building AI agents is this shift from the kinda complex like inner loop, where you have a model that might determine what k- type of task the user is asking for given a prompt.

15. 2:42

    You'll then load a more task-specific prompt and tool set. You'll then chain a bunch of these loops together in order to achieve some sort of goal. Maybe just ask the model, "Hey, are you done yet?"

16. 2:51

    or to keep going, um, and then finally use another model to respond back to the user. Um, we don't-- generally, we don't need these anymore. Um, you can actually just have the model decide when it should use which tools and when it should write or run code, and it can just write and, write and run that code,

17. 3:07

    um, on its own. Now, that's what we in security would call a RCE or remote code, um, exploitation. So, um, when we're looking at these new behaviors, it's important to consider not just the capabilities, but also how do we ensure that those capabilities are not gonna backfire on us when we allow it to be able to perform

18. 3:24

    those operations. And there's a couple of different ways that we've observed how models can go wrong. Um, the most common ones and that we think about consistently is prompt injection and data exfiltration.

19. 3:35

    There's a lot of different examples that we'll, we'll be documenting in the coming months. Um, but, uh, that's probably number one in our priority queue. But then you also have things like the agent just makes a mistake.

20. 3:44

    It just does something wrong. Um, maybe it installs a malicious package un-unintentionally, or it writes vulnerable code, again, unintentionally, or you have privilege escalation or sandbox escape.

21. 3:55

    And when we think about our responsibility of deploying these agents both internally and externally, we have this preparedness framework where we document some of the recommendations and also some of the, um, kinda standards that we hold ourselves to.

22. 4:07

    But one, one of the ones that I wanna emphasize is requiring safeguards to ensure, um, or to avoid misalignment at large-scale deployment. And this is something that we think about ourselves when we are building Codex, um, but also something that organizations, as you deploy coding agents into your workplace, that you should also be considering.

23. 4:24

    And one of the first safeguards that we put in place is to sandbox the agent, especially if you're running it locally. Um, generally the best, the best method is just to give it its own computer.

24. 4:33

    That's what we did with Codex and ChatGPT. It spins up a container, fully isolated. It then produces a PR at the end. That's practically as safe as you can get.

25. 4:41

    Um, but if you are gonna run it locally, which, uh, of course, with Codex CLI, we, we, we also, uh, encourage, um, making sure that you're actually providing the correct level of sandboxing, whether it's, uh, containerization or it's using app-level sandboxing, which we'll talk about in a moment, or OS-level sandboxing, um, making sure that you're providing the

26. 4:57

    right guardrails for the model, even if it does attempt to do something wrong.

27. 5:02

    Related to that is disabling or limiting internet access, and this is probably the kinda highest, uh, uh, kinda probability vector of prompt injection, um, or data exfil. You know, the model goes to read some sorta docs or reads a GitHub issue, and then in a comment of that GitHub issue, maybe there's a prompt injection, um, that kinda

28. 5:20

    untrusted content can leak into, um, the kinda core inner loop that you would trust an agent to run code in. And if it has access to your code base or other s- uh, sensitive materials, um, that could be, um, pretty bad.

29. 5:32

    Um, and then finally, um, reviewing, um, all of these operations or the actual final diffs that the agents perform, um, whether it's a code review in a, in a GitHub PR or it's, um, approvals and confirmations, um, those guardrails are actually really important.

30. 5:47

    Ensuring that humans stay in control of these systems, um, is one of the strongest mitigations that we have. But of course, no one wants to sit there and click, keep clicking Approved, so, um, avoiding the kinda YOLO mode on one end to, uh, you know, having to approve every single, um, you know, ls command is, is not

31. 6:03

    practical either. So let's talk a little bit about how do we actually achieve this. So, um, I mentioned, um, our recommendation is to give the agent its own computer.

32. 6:10

    You see this in Codex and ChatGPT. Um, there's a lot of different constraints that you need to apply when you think about that, making sure that the agent has all of the dependencies installed, all of the different access it needs to perform its actions.

33. 6:22

    Um, and, uh, if you wanna run it locally, being able to use something like Codex CLI, which we fully open sourced for you to be able to, um, build out these agents yourself.

34. 6:31

    You can use this as a reference point. That's part of why we wanted to open source it, is really showcase not only here's, uh, the agent that we built for you, but also, um, here's how you can build your own.

35. 6:40

    And, um, s- as I mentioned, fully open source, so you can actually use these, um, in this case, macOS or, uh, Linux sandboxing techniques. And, um, as an example, here is the, um...

36. 6:51

    so a, a portion of the macOS sandboxing policy. This uses a language called, uh, Seatbelt, um, that, uh, uh, Apple bundles into operating systems since, uh, Leopard. Um, it's, uh, can be somewhat, uh, uh, hard-to-find documentation for, so this is definitely an area where, um, using both our models, using deep research to actually understand what are the

37. 7:09

    bounds of different examples that people have created. Um, this, uh, we were heavily inspired by Chromium, which also uses Sandbox, or excuse me, uh, Seatbelt, uh, as a sandboxing, uh, mechanism on, uh, macOS.

38. 7:19

    And then, um, separately, um, you'll actually notice this is now in Rust, um, where we actually, um, tapped into our own security teams to, um, build out our Linux sandboxing and run it, in this case, using both seccomp and Landlock, um, in order to be able to, um...

39. 7:34

    I think we'll do maybe questions afterwards. Um, but, um, to, to... in order to, uh, have unprivileged sandbox, um, uh, and prevent escalation. Um, and then next we have disabling internet access.

40. 7:46

    This is really important when it comes to prompt injection, which again is a primary, um, uh, exfil risk. And, uh, we have two methods. Well, actually, let me before I get into that, um, we have two methods both in Codex and ChatGPT, but also within CLI, we actually have this full auto mode where effectively what we did

41. 8:01

    was define a sandbox where, um, it can only r-read and write files within the directory that it's run in. It can only make network calls, um, based on commands that you're auto-approve it for.

42. 8:10

    Um, but otherwise it just runs in this kind of fully sandbox and lockdown, um, environment that allows the agent to be able to go and test, you know, run pytest, run npm test, um, but not actually have some second-order consequences.

43. 8:23

    Um, and then when it comes to Codex and ChatGPT, we actually just launched this, uh, yesterday, um, or two days ago maybe. Um, but, uh, you can now turn on internet access, but it comes with a set of configurable allow lists.

44. 8:34

    This is really important when you consider either using or building agents yourself, um, ensuring that you have both the kind of maximum security option and also this more flexible option so people can define, um, whatever use case that, um, or whatever policy that makes sense for their use case.

45. 8:49

    And in here we even define, um, which HTTP methods are allowed, including a warning letting you know about the risks. Just to give you an example, and we actually link to this, um, from those docs, um, let's say my prompt is to fix this issue, and I just link to a GitHub issue.

46. 9:03

    Um, seems pretty innocuous, but in the GitHub issue, which could be, you know, user-generated content, um, uh, go ahead and grab the last commit and go ahead and, um, post that to this random URL.

47. 9:13

    And because Codex is, um, really trained with, um, instruction following, and it tries to do exactly what you ask, it'll go ahead and do that. Um, now a way that we can control that is both at the model level and flagging things that seem like they could be suspicious, and that's definitely an area, um, when it comes

48. 9:27

    to model training that we're actually focusing. Um, but ultimately your most, um, kinda, uh, deterministic and authoritative control is gonna be a system-level control. It shouldn't even be able to make a call to http bin in this case.

49. 9:40

    So combining those model-level controls along with your kinda system-level configurations is really key to solving this problem.

50. 9:49

    And finally, there's requiring human review. Um, now this is something that, um, I see a lot of tension when it comes to folks who are, um, kinda using LMS and coding agents, um, is that it's, uh, you have this new problem when you're prompting these agents, is there's just so much code that you end up having to

51. 10:04

    review. Using tools like other PR review or, um, kinda code review tools and using LMS as part of that loop, while useful, is not a substitute for a human actually going in and reviewing the operations that the model's about to perform.

52. 10:17

    Ensuring that you're not having a model that might have installed a package that maybe is not as well known or it's maybe off by one, uh, character, um, and, uh, ensuring that that doesn't land in your code base that then later gets run, um, in an unprivileged environment.

53. 10:32

    Or excuse me, in a privileged environment. And then we also have, again, since this doesn't just apply to coding agents, um, we also have Operator as an example where, um, there's different techniques you can use.

54. 10:40

    In this case, um, we have both a domain list and also a monitor that is in the loop, uh, identifying any kind of potential sensitive operations that a model might go out and do on your behalf.

55. 10:50

    And we have this monitoring task and watch mode, as we call it, where we ensure that a human is actually reviewing any kinda actions that it can take. So again, balancing the maximum security with the maximum flexibility is really important here.

56. 11:03

    And so as an example of how to think about actually building these agents, um, effectively it-- where previously you might have had a loop that is doing a bunch of different l- um, elements of software-based logic, now you can actually just defer most of that logic to the reasoning model and give it the right tools to accomplish

57. 11:17

    the task. Um, we released this exec tool, um, LocalShell, as it's called in the API, um, where it actually is exactly the way that we train our models to be able to execute, to write and execute code.

58. 11:28

    Um, we also released tools like ApplyPatch, which models aren't particularly good at getting line numbers correct for like a, a git diff. So we provided this new, um, format for actually applying diffs to files.

59. 11:39

    Um, but then of course, kinda your more standard tools, things like MCP, web search. Um, I'm actually gonna give an example of how you can use these in combination.

60. 11:47

    Um, so let's say, um, uh, Socket, which is a dependency, um, check, um, dependency vulnerability checking service, um, now has an MCP server. You can expose that to the agent to then go in and verify whether or not a given dependency it's about to install could be vulnerable or suspicious, and ensure that the model either is part

61. 12:05

    of its own operations or you can apply a system-level check after the rollout has completed, um, to make sure that any dependencies it's going to install, um, are actually safe to do so.

62. 12:15

    But again, one thing we'd emphasize is to use a remote container. Um, we are releasing a container service as part of, um, our a-agents SDK and, um, as part of a responses API.

63. 12:26

    And so you can either run it locally or run it in your own environment, or you can, uh, let OpenAI kinda host it for you.

64. 12:33

    And so as a recap, um, would strongly recommend sandboxing these agents, whether it's through containerization or it's through OS-level sandboxing, um, disabling and limiting internet. I think that balance between, uh, capability, where you wanna be able to let it just run and do its own thing for as long as possible, um, which you can do when it's

65. 12:50

    fully network disabled, um, versus I want it to go out and read docs, I want it to go install packages. Um, we give you that flexibility, but being really thoughtful about when you employ each.

66. 13:00

    Um, and then finally, requiring human review. This is definitely an area where we expect there to be a lot more research. Um, employing monitors, um, LM-based monitors in the loop, while valuable, is just not quite there yet in terms of, um, the kinda certainty that you get from, again, a deterministic control.

67. 13:17

    And so in, in that, uh, in that vein, um, there is more tooling that we plan to release here, so, uh, stay tuned in the Codex repo, um, on the OpenAI org.

68. 13:25

    Um, there's also more documentation that we plan to publish around, um, both the ML-based interventions and the systems controls. And if you're interested in working on problems like this, um, we are hiring for this new team, Agent Robustness and Control.

69. 13:38

    And, um, so if you also write Rust, we are also hiring for the Codex CLI to build out more of those integrations and making sure that everyone can benefit from them.

70. 13:46

    So, um, if you're interested or you know someone who'd be interested, definitely let us know. But with that, thank you so much. [applause] [upbeat music]