Codex, Behind the Harness

Talk transcript

1. 0:00

   [upbeat music] Hi, everyone. Uh, we're gonna start right on time because, uh, I'm gonna speak basically at two X.

2. 0:17

   I'm sorry, I have a lot of to- content. I'm trying to get you out of here on time. [laughs]

3. 0:22

   I'm gonna start with a quick raise of hands, though. How many of you have built your own agents or are currently building your own agents? Perfectly. You're the right audience for this.

4. 0:30

   Um, over the next twenty minutes, I wanna talk to you about a couple of different things that we're doing in the Codex Harness that hopefully you can learn to apply to your own, uh, use cases or even just use the Codex Harness with this in, um, uh, in your own project or at bare minimum, learn what happens

5. 0:46

   when you actually use Codex. Uh, since we're at AI Engineer World-World's Fair, and we're actually on an agentic engineering track, I'm gonna, like, stop bothering you with, like, how does an agent work?

6. 0:57

   What is an agent? And instead, I wanna talk a bit more specifically about some key features that we have in the agent that I think are particularly interesting and are challenges you have to solve.

7. 1:08

   And, uh, so let's cover it from the a- lens of, like, what actually happens when you send off a message. Uh, also, a quick reminder, if you're unaware, the Codex Harness and everything I'm showing you is actually open source.

8. 1:21

   Uh, it's MI-- it's Apache-2.0 licensed, and the harness was written in Rust. So feel free to either learn from it, ask Codex deeper questions about what I'm covering, uh, or fork it and make it your own.

9. 1:34

   Also, disclaimer before we dive deep into it, this is a current state of affairs. Like, things change so quickly. Um, you can always refer back to asking Codex what the current state is, but especially with new model releases, we often release new APIs and change sort of how the harness works.

10. 1:50

    Uh, so feel free to follow along as new models come out.

11. 1:55

    If we wanna talk about how the Codex agent works, we first need to talk about what actually happens when you send off your message. Um, namely, there's two protocols that are involved with, uh, Coda-- uh, with the Codex agent.

12. 2:07

    The first one is what happens when you send it off in the UI, and it goes to the harness. We call that the app server. I talked about that yesterday, so we're not gonna spend too much time about it.

13. 2:15

    There will be a talk online that you can follow along. Um, the second part is the Responses API, which is-- handles the communication between the harness and the inference.

14. 2:25

    Both of these, though, are designed for an open ecosystem, meaning if you're building your own UI, you're building your own, uh, agent interface, you can actually build on top of the Codex, uh, harness using the app server protocol.

15. 2:39

    We use that same app server to power the Codex app, so it has really all of that functionality, uh, that you might expect from, uh, Codex, as well as, like, we have a lot of third-party community projects that build on top of it, including Theos T3 Code or, uh, RemoteX, for example.

16. 2:56

    I even use that same app server to put Codex into Claude Code. So if you're a Claude Code user and you wanna leverage Co- uh, Codex, you can use that plugin.

17. 3:05

    And if you joined my talk yesterday, you saw me using that same protocol to actually put Codex into Doom, um, which was, was a fun adventure as well.

18. 3:15

    I mentioned the other part is the Responses API. So the Responses API was released last year as, like, a rethinking of the Chat Completions API in a more agentic world, meaning, um, we redesigned the-- slightly the structure, but more importantly, we added a lot of, like, building capabilities that are important for agents like web search, image gen,

19. 3:34

    or other, other more complex capabilities that you will see as part of this talk. We also want to make sure that this is, like, an open ecosystem, so we worked with a lot of partners, including Ollama, LM Studio, NVIDIA, and others, to, uh, codify an open responses schema and have a governance body for that so that other,

20. 3:53

    uh, companies and, and platforms can actually build on that same responses API, and you can use any responses API-compatible, um, harness, uh, model provider, and actually plug it into the Codex Harness.

21. 4:08

    So that's an overview of how these protocols work. We're going from the UI to the harness with the app server protocol and then from the harness to the LM inference using responses.

22. 4:18

    But what happens in the actual harness? The first step, and arguably one of the most important ones, is context construction.

23. 4:26

    And during that, we care about three things, uh, quite a lot. The first one is size. We wanna make sure that, you know, we don't blast through your token budgets and throw in a bunch of unnecessary content.

24. 4:37

    But also, the more context you have in your, uh, in your context, um, the higher it is that you have contradicting information and, uh, it causes confusion for the model.

25. 4:48

    The other part is flexibility. We wanna make sure that regardless of how many or how little skills you're using, you have a great experience, um, uh, regardless of how many plugins and MCPs you install.

26. 4:58

    And of course, we wanna make sure that things are performant and cost-- uh, we know you're cost sensitive, so cacheability is important as well.

27. 5:06

    To show you this and a couple of other things, I actually built this little nano Codex here, um, which functions the same way. It's built on, uh, built using the same code that is on the public repo, just turned into TypeScript.

28. 5:19

    But we can see here, when we send off a message, um, we have a couple of different parts of the co-- uh, of the actual, uh, context that gets assembled, and some of these are fairly standard and predictable, like the model instructions.

29. 5:34

    Um, again, these are open source if you actually wanna read them. But these are fairly structured and don't really change in size or k-- uh, like, mess around with cacheability.

30. 5:43

    But there are some parts that are harder to predict, namely, for example, how many skills do you have available? Or the tool registry where, uh, especially if you install MCPs, for example, you might have additional context that is, like, growing as you're installing more MCPs.

31. 5:59

    For that reason, we do two things to maintain cost. The first one is-- Or the size of the context. The first one is having deferred tools. So, uh, we're marking some of these tools as deferred, and that means that they are not added directly to the context window, but instead are available through tool search later on.

32. 6:16

    Um-

33. 6:18

    And then the other part is that for available skills, we actually cap the available skills list at two percent of your context, uh, tot-- like, maximum context window. And that means that, um, if it gets longer, we're reducing slowly the amount of, like, description that we're putting in there.

34. 6:36

    Tool search specifically is actually something that is available in the Responses API, so even if you're building your own harness, you can leverage this. Since GPT 5.4, you can mark any tool as deferred loading, um, and that means that these tools are only available if you're using tool search.

35. 6:52

    And then you can give the model either our build-in tool search tool or implement your own if you feel like you can better do that, uh, discovery yourself.

36. 7:01

    Great. We talked about how we're building the context, but an agent really only becomes an agent if it performs actions, and there's three common actions I wanted to talk about.

37. 7:10

    The first one is async actions, things that are happening while the agent has to continue to do work, computer use, and then the file system.

38. 7:18

    For async a- uh, actions, a good example is subagents, where we wanna be able to delegate tasks off and then have the main a- agent continue to do work if necessary.

39. 7:29

    The way that works in practice is that we give the agent a spawn agent tool, which then allows the agent to create new a- uh, new agent instances and then use a send input tool to either send new content to that a, uh, to those new, newly created agents, wait for an agent, or shut it back down.

40. 7:49

    Um, we use that same concept actually for background terminals as well. So the Codex agent has a tool to spin up a new, uh, background terminal and then continuously interact with it by sending new data through standard in to that new agent or wait for a specific amount of time for that agent to finish a task.

41. 8:12

    Computer use is an interesting one because we actually introduced, uh, computer use in the Responses API last year, and it was fairly limited. It only allowed you to do one action at a time, and, um, you had to basically declare that you wanted it to do computer use, and from there you were up to actually implementing specifically

42. 8:32

    the type of actions that were exposed to that tool. This was gr- uh, great in terms of, like, that sta- uh, that point in the journey of, of building agents, but since then we've evolved it and, uh, the recent models and the recent API shapes allow you to use code execution instead to actually do computer use, which

43. 8:51

    means that the agent can script its own interactions with the, uh, with whatever computer implementation you wanna have. You can choose the language, like JavaScript or Python, and you are, you have a much more flexible harness.

44. 9:04

    In fact, that's what we use for browser use. Um, so the... What happens when, uh, Codex uses browser use is it actually interacts with a persistent Node REPL that gets persisted throughout different, uh, throughout the turns, and then it writes JavaScript, um, essentially Playwright code, to interact with that, uh, browser instance in the Node REPL.

45. 9:28

    Um, so here's an example where on the right side we have a Chromium browser, and the first time, um, it writes some code to get the overall status and, like, pulls up the right tab.

46. 9:39

    And then on subsequent turns, we can see here that it is able to, like, reference those new tabs, pull in information, and script the respective actions, um, to understand what actually has to be done.

47. 9:53

    And so that ma- speeds up these actions significantly because, uh, Codex and browser use can actually look at, for example, one page, understand the structure, and then write a script to perform, like, scraping, for example, on subsequent pages more easily.

48. 10:08

    The third thing is file system interactions. I think if you're using Codex or any coding agent, of course you want it to be able to interact with the file system.

49. 10:15

    In the case of, um, our models, all, all of the recent models starting with GPT 5 have been trained on the concept of an apply patch tool to do file editing, which means that they're used to using that to change files by giving it a, uh, diff and then, um, also using that same thing to create new

50. 10:34

    files. And then for everything else, it uses a shell tool where it will, uh, do file search or other file system navigation. Uh, you will see the model naturally trying to use ripgrep, um, since that's what, what it got used to during training.

51. 10:49

    So we're actually in the Codex harness shipping ripgrep with, uh, with the harness if you don't have it installed on your own. On Windows, we also trained the model to use PowerShell natively, so if we're running it on Windows, you will see it, uh, start writing, uh, PowerShell code instead.

52. 11:06

    One of the things you're seeing here is, um, the sandbox pulling in. So all of the interactions that you see with the file system actually go through our sandbox layer.

53. 11:17

    Um, and on macOS, we use Seatbelt for that, similar to most agents, and on Linux we use Bubblewrap. On, uh, Windows it's slightly different, where we actually had to build our own custom, um, open, open source Windows sandbox.

54. 11:34

    It's in the same GitHub repository if you wanna take a look. And there's, um, many reasons why we had to do this, and I could probably fill a whole talk about that, so instead I would recommend you to actually check out the t- uh, check out the article that David wrote about this.

55. 11:48

    Um, it goes into all of the other Windows alternatives and why we actually had to build our own, uh, sandbox there. Highly fascinating if you really wanna nerd out about Windows internals.

56. 11:59

    With sandboxes though, [laughs] uh, one of the biggest complaints is always approval fatigue. Um, how many of you get, like, annoyed by approvals, especially for long tasks, and, like, have run in full access?

57. 12:11

    All right. And I don't think the audience gets filmed, so, like, we're gonna do this again. Like, how many of you know that, like, IT and security really hates that you turn on full access?

58. 12:19

    Yeah. Yeah, yeah, yeah. I know. Uh, [laughs] so for that reason and, um, you know, our own security team wanting to be happy as well, um, we worked on, um, an auto-approval mode.

59. 12:31

    And the idea here is really to, um- De-risk some of the, like, full access things because there are things that can go wrong. As much as these models have gotten better, especially if you're trying to push the model through prompting to really have high agency, that can be misinterpreted by the a- uh, by the agent to not

60. 12:49

    match up with your expectation. Like, you asking a model to send out a file to-- through an email, pushing it to have high agency, and it realizes it can't attach the file, so it uploads it to a file share, um, and, like, uses that instead or messes up some escaping and accidentally deletes too much data.

61. 13:06

    Those things can still happen even with the models getting better. And so you want to still be careful with these, um, full access modes. And so instead, we build Auto Review with this goal to cover all of the things that are harder to predict that an agent might do, and especially in the long run, and, um, approve

62. 13:25

    those automatically without having to escalate them to you while keeping things like, uh, data exfiltration outside of it.

63. 13:34

    The way this works is when we, um, kick off a task that runs into a sandbox, like deleting a file in this case, um, the model, when it tries to escalate, spins up an Auto Review, uh, sub-agent.

64. 13:49

    And, uh, let's see. Didn't do it yet. There we go. So it spins up an Auto Review sub-agent, and this sub-agent runs entirely separate. It can't spin up other sub-agents.

65. 14:00

    It has read permissions only. Um, but it-- we're giving it a bunch of context around sort of how-- what, what is user authorization, which we'll see in a second.

66. 14:10

    How do things like, uh, risk taxonomies work? How do we want it, want it to judge these things? Um, and then eventually, we give it, um, the transcript as well as sort of the tool calls that are actually happening.

67. 14:24

    And this is important because the context matters, right? Like, in some cases, you want the agent to actually delete a file. In other cases, you don't. If you ask it to or if it, like, is part of the project, it makes sense, especially, like, things like if you ask it to delete a .git folder, great.

68. 14:40

    If you didn't ask it to, it should probably not touch that part and, like, completely delete your history. Um, and so we're giving it all of this context and ask it to then come up with, um, a judgment based on, like, the user authorization.

69. 14:55

    In this case, it's high because we explicitly told it to delete the file. Um, what is the impact of the deletion or, like, the action itself, and what should we do?

70. 15:04

    So this could apply both to file system actions but also to network calls, for example, where, like, curling Google to see if it-- if the internet works is fine but maybe not uploading a file, for example.

71. 15:18

    This is a gross oversimplification of all of the thoughts and, uh, work that went into this by engineers and, and the research team. So if you do wanna learn more about it, check out the Auto Review, um, blog post that we wrote.

72. 15:32

    The next thing I wanted to talk about is speed, um, because agents can do a lot of tool calls. And while we're doing a lot of work on speeding up inference, it's only part of the equation.

73. 15:44

    We noticed this when we, uh, launched GPT 5.3 Codex Spark, uh, and it's running on Cerebras at a thousand tokens per second. So with that, we realized that with all of these tool calls and the interactions, inference wasn't no longer-- uh, was no longer the bottleneck.

74. 16:00

    It was actually the network. So for that reason, we introduced WebSocket mode, which means that the responses API doesn't run through server-side events and HTP, but instead uses we-- uh, uses a persistent WebSocket connection, which allows us to save both on network overhead but also provide stateful context, which means we only have to send the data that

75. 16:22

    actually changed. So, for example, if there's a tool call, we only send back the result of the tool call rather than sending all of the items back. So in practice...

76. 16:32

    Oh, great. In this case, my demo server crashed. Um, let me see if I have a backup here.

77. 16:43

    Um, oh, spoiling the rest of the demo. Nope. Uh, [laughs] all right. Um, in practice, you will see that it's gonna send, um, one-- like, it's gonna send only one item after another.

78. 16:58

    Oh, wait. Sorry, I pressed the wrong button for this one. There you go. Here's the backup. Um, so it's gonna only send, like, one item after another rather than sending, like, in this case, for example, like, nine items back.

79. 17:12

    Um, while for the other one, we're only sending one back. And so over time, this actually speeds up things quite significantly, um, and can have a pretty drastic impact on the performance.

80. 17:25

    All right. We're at AI Engineer World's Fair in twenty twenty-six. The only thing, like, I have to talk legally about is loops. [laughs] Um, so we're briefly gonna talk about it, um, specifically slash goal because I got the questions a couple of times during this event.

81. 17:39

    How does this actually work? Um, and of course,

82. 17:45

    since the demo server crashed, which also shows that that was real data, uh, [laughs] we're gonna show you the demo here. This is a hosted demo, by the way, so you'll be able to replay this afterwards.

83. 17:57

    Um, but basically what happens is, in this case, we're having it try to guess a number, and it's only achieving that goal when it's actually done with guessing that number.

84. 18:07

    And while it-- uh, until it's done with that, um, it will actually automatically, the harness will inject this continuation prompt. And this continuation prompt includes, among other things, your objective.

85. 18:18

    That's the goal that you set. And then, uh, uh, we continue to do this until the model itself calls an update plan-- uh, update goal tool, which specifies that the plan was actually-- or the goal was actually achieved.

86. 18:32

    So that is the reason why you actually don't want to, you know, write full essays like I know a lot of you have been trying to, um, into your goal.

87. 18:43

    But instead, have very concrete and verifiable, um, prompts so that, uh, it's easy to detect when things are done. The last thing I wanna talk about is compaction. If we have these agents run for hours or days at a time, you don't wanna stand there and actually approve everything all the time.

88. 19:01

    Um, and because of that, we introduced, uh, end, end of last year, auto compaction, and this has been used by Codex since then to automatically trigger compaction on the server side in a way that the model got trained with so that the performance stays the same.

89. 19:17

    Um, the way this works is you can either trigger this manually or automatically, and it will turn your transcription into... or your, your previous context window into a new one that you're gonna use instead on subsequent turns, and it contains a compaction item that has all of the necessary information in it that you need.

90. 19:37

    All right. To wrap things up, thank you for staying with me speed running through these slides. [laughs] Um, three things I want you to take away from this. Uh, the first one is the Codex app server and the harness are open source.

91. 19:50

    You can use this as a blueprint to learn how we build our agents, or you can actually use it as the harness itself that you can build on top of.

92. 19:58

    The other part is that most of the features that are standout for Codex are actually features that are exposed in the Responses API. So even if you wanna build your own agent, you can leverage these things like tool search, apply patch, web sockets, or server-side compaction.

93. 20:13

    You can use this directly regardless of what harness you're using. And the last thing is, as models evolve, keep an eye out on how we're evolving the Responses API, how we're evolving Codex, and, uh, use that as a way to understand how you wanna update your own agents to make, uh, make use of these new capabilities.

94. 20:32

    And with that, thank you so much. That's the link to the slides, and I'll be heading down to the booth if you have any questions. [clapping] [outro music]