Future-Proof Coding Agents

Talk transcript

1. 0:00

   [upbeat electronic music] Hello, everyone.

2. 0:22

   Um, today we'll be talking about how to build coding agents.

3. 0:27

   And, uh, I'm Bill. I work on the Applied AI Startups team at OpenAI.

4. 0:32

   And I'm Brian. I work with Bill on the OpenAI Startups team.

5. 0:35

   And we specifically, uh, focus on, uh, building coding agents here at OpenAI.

6. 0:41

   Um, yeah, so why are we talk- giving this talk? Why, why are we, you know, uh, talking about coding agents? Well, it's really quite interesting because it's been booming for the, the, the past year.

7. 0:54

   Actually, it's just ... If you think about it, it's not that much time ago, like only been a year or so. The ground keeps shifting really under the, uh, harness on, on the coding agents.

8. 1:04

   But if you think about it, it's really, like, why it's interesting is because it's really a signal on how close we are to AGI. Software engineering can be set as a universal medium for problem-solving.

9. 1:15

   But because the ground is shifting so fast, uh, we ha- kept having to rebuild the agent on top of the model whenever a model is released. And today, we're going to talk a little bit about how we might be able to get around that.

10. 1:29

    So here's what we're gonna go over today. We'll start with the anatomy of a coding agent, especially going into the details of models and harnesses and how they work together.

11. 1:39

    We'll share some lessons that we learned from putting them together ourselves, and we're specifically gonna talk about Codex here, which is our own coding agent. We'll talk a little bit about emerging patterns that we're seeing from all of you for using agents like Codex in your own products.

12. 1:56

    And lastly, we'll talk a little bit about what to expect from Codex in the future so that you can build along with us if you want to.

13. 2:06

    To start, let's talk a little bit about what makes a coding agent an agent as a whole. Um, it really is quite simple. I think, you know, people kind of overcomplicate things a little bit these days.

14. 2:19

    It's made out of three parts. It's a user interface, it has a model, it's a harness, right? Uh, the interface, quite self-explanatory. Could be a, a computer, uh, a c- uh, like a CLI tool, or it could be an, a, uh, integrated developer environment.

15. 2:34

    Could be also cloud or a background agent. Um, models, also very quite self-explanatory, are, you know, the things like the latest and the greatest, the GPT-5.1-Codex, uh, Max that we just released yesterday, uh, or the GPT-5.1 series of models or other, uh, models from other providers as well.

16. 2:56

    And the harness, uh, is an, a little bit more of an interesting part. This is the part that directly interacts with the model. Uh, in the most reductive way, you can sort of think of it as a collection of prompts and tools combined in a core agent loop, which provides input and outputs, uh, from a model.

17. 3:15

    Uh, the last part will be our focus for today.

18. 3:21

    As touched on a bit earlier, coding is one of the most active frontiers in applied AI, and, uh, how models are constantly getting released, and we're not making the problem, uh, easier for everybody [laughs], is that people have to constantly adapt, uh, the agents to the new models.

19. 3:44

    So, um, Bill's done a great job of giving us an overview of coding agents, what they're made up of. So let's zoom in a little bit on the harness.

20. 3:55

    Um, turns out that's a little bit tricky. So what is a harness? A harness is really the interface layer to the model. It's the surface area the model uses to talk to users and the code and perform actions with tools.

21. 4:10

    It's made up of all of the pieces that the model needs to work over many turns, call tools, and, and really write code for you and interpret to, what the user is actually asking.

22. 4:22

    Um, for some, the harness might actually be the special sauce of the product. But as we're gonna go into a little bit more, it's really challenging work to build a good harness, and we'll talk about how we did that.

23. 4:39

    So let's see. What are some of these challenges? Um, just to name a few, AV is one. Um, your [laughs] um, your brand-new innovative custom tool that you're giving to your agent might not actually be something the model is using, is used to using.

24. 4:55

    It may not have ever seen that tool before in training. And even if it is, you need to spend time tuning your prompt to that particular model and the habits that it comes with.

25. 5:07

    And new models are coming out all of the time. What about latency? Like, does the model take a while to think about certain things? Which things? Do you prompt it not to?

26. 5:17

    How do you expose the UX of what a thinking model is doing while it's thinking? Is it communicating with you while it's thinking, or do you have to summarize it?

27. 5:27

    Managing the context window and compaction can be really challenging. We just launched Codex Max that does that out of the box for you. You don't have to worry about compaction and context window management.

28. 5:40

    It's really hard to do. Um, and so if you were to do it yourself, have fun. Um, and then also, like, the APIs keep changing, right? So we have completions, we have responses, we have whatever else is coming in the future.

29. 5:52

    What does the model know how to use and get, to get the most intelligence out of the box?

30. 5:59

    And so- This is the interesting part. Fitting a model into a harness takes a lot of prompting. [laughs] It turns out that how the model is trained has side effects.

31. 6:13

    I like to think about it this way,

32. 6:15

    intelligence plus habit. Intelligence, what is the model good at? What languages does it know really well? What is, what is its capabilities in terms of, like, how well it can write code in certain frameworks?

33. 6:30

    And then what habits did it learn to s- to use to solve those problems? We've trained our models to have habits of, like, planning a solution, looking around, gathering context, and, and thinking about a problem before diving in and writing code, and then testing its work at the end.

34. 6:51

    Developing a feel for these habits is how you become a good prompt engineer. If you don't instruct the model in ways that it's familiar with, you can have problems.

35. 7:03

    We saw this when we launched GPT-5. A lot of people who weren't used to using our models in coding tried to take prompts that existed for other models and put them into their harness and have GPT-5 follow those instructions.

36. 7:17

    And it turned out that we taught our model to do some of the things that the other models didn't really do out of the box. And so when they were prompting them to look really hard at the context and, like, examine every single file before making a, a code edit, our model was being very kind of thorough

37. 7:37

    about that, and it was taking a really long time, and they weren't seeing the best performance. And so we figured out that if you let the model just do the behaviors that it's used to and don't over-prompt it, it'll actually perform really better.

38. 7:50

    We found out by asking. I was literally like, "Hey, like, I like the solution, but it took you a long time to get there. What can I do differently in your instructions to help you get there faster next time?"

39. 8:01

    And literally it said, "Uh, you're telling me to go look at everything, and I don't really need to, so that's what's taking forever." [laughs]

40. 8:12

    And so you can actually see the advantages of building both the model and the harness together because you just, like, know all of that while you're building it, and that's why Codex is both a model and a harness combined.

41. 8:26

    So let's dig deeper into Codex and what it can actually do.

42. 8:31

    So we built Codex to be an agent for everywhere that you code. It's a VS Code plugin. It's a CLI. You can call it in the cloud from the VS Code plugin or from ChatGPT from your phone.

43. 8:43

    Um, at its very basic, you can use it to turn your specs into runnable code, starting from a prompt, um, having a plan. It navigates your repo to edit files, it runs commands, executes tasks, and you can call it from Slack, or you can have it review PRs in GitHub.

44. 9:02

    So all of the things that you would expect.

45. 9:06

    And that means that the c- that Codex, um, the harness of Codex needs to be able to do a lot of really complex things. Uh, when I talked to a member of the Codex team about this slide and what should be on it, he was like, "It's way harder than you think." [laughs]

46. 9:21

    You have to manage parallel tool calls, like thread merging and all of the things involved in that. Think about all of the security considerations you have with sandboxing, prompt forwarding, permissions, uh, port management.

47. 9:33

    Um, compaction is a whole thing, um, and doing that well is really complex. When do you trigger compaction? When do you reinject? How do you worry about con- uh, cache optimization during that?

48. 9:45

    MCP, [laughs] right? Like, all of the th- the, uh, plumbing you have to build for MCP support into the harness, uh, and then not even mentioning images and what is the resolution that you need to compress them to to send them to the model.

49. 9:58

    All this, all of this is, like, work that you have to do if you're gonna build this from scratch, and keep it updated as new features come online.

50. 10:07

    So since we've bundled all of these features together for you in an agent that can safely write its own tools to solve new problems that it encounters...

51. 10:20

    Oops. [laughs] Uh, we actually have here, uh, a computer use agent for the terminal.

52. 10:33

    Wow. That sounds quite a bit powerful than just plain old coding agent, doesn't it? Um, but just think about it again. Well, before browser and graphic user interface was a thing, wasn't that how we always operated a computer, with the writing code and chaining them together in a command line interface?

53. 10:51

    Uh, so that means if you can express your task in command line as well as files, tasks, Codex will be able to know what to do. Um, the example is, I like to use Codex to organize a lot of the photos from my desktop into a folder.

54. 11:06

    N- and that's a very simple use case. But what it can also do is it can analyze huge amounts of CSV files inside of a folder, uh, doing data analysis.

55. 11:17

    It does not have to be a coding task, and if it can be accomplished by running tools from command line, you can use Codex.

56. 11:24

    So now that we see Codex as such a cool harness, um, I want to also share a little bit about how you can use it to build your own agents.

57. 11:33

    And what you can do is you can use Codex the agent inside of your own agent.

58. 11:41

    Um, how does that work? Well, if you want to build, uh, a coding, uh, an- the next coding startup, we don't really have all the answers, but we do have a few patterns, uh, that we s- thought, uh, might help you, having worked with some of the top coding customers, uh, like Cursor and VS Code.

59. 12:01

    Uh, one of those patterns is, uh, harness becoming the new abstraction layer. The benefits of this is quite obvious. Um, you no longer have to care about prioritize, uh, optimizing the prompt and tools with every mo- model upgrade.

60. 12:18

    But, um, does that mean you're just building a wrapper?

61. 12:21

    Well, I disagree with that take. [laughs] I disagree.

62. 12:26

    I would disagree with my colleague here. Um, [laughs] just like how building wrappers on top of models, I think, is really reductive on, uh,

63. 12:35

    on, on the whole value prop of the infrastructure layer.

64. 12:37

    Sorry, I used to be a VC. [laughs]

65. 12:40

    Focusing most of your efforts on differentiating your product is what this pattern allows you to do, and that's where most of the value lies.

66. 12:52

    Exactly. Okay, so let's look at some of these patterns that we've seen and actually have helped our customers build, um, along with them.

67. 13:01

    Codex is an SDK. It can be called through a TypeScript library. You can call it programmatically in a Python exec. There's a GitHub Action that you can plug into to have it merge, merge conflicts [laughs] on PRs that everybody hates doing.

68. 13:16

    Then, uh, you can also add it to the agent's SDK and give it MCP connectors back to your product. So now you have an agent, I like to say we started with chatbots that you can talk to, then we gave the chatbots tools to use, and then now you can give, uh, a tool to your chatbot that

69. 13:37

    can make other tools that it doesn't have. And so now you can actually build out enterprise software that does its, that writes its own plugin connectors to the API level for each customer on the spot.

70. 13:51

    That's something that a professional services team used to have to do. Um, so you have fully customizable software that can now talk back to itself. Um, I made a Kanban board for Dev Day that can actually fix its own bugs. [laughs]

71. 14:03

    Um, it's pretty fun. And then lastly, um, you can actually do something like what Zed has done. They have just decided to wrap Codex inside of a layer and give it an interface to the IDE for talking back and forth for the user and making code edits.

72. 14:20

    And now they don't actually have to do all the work of staying on top of all of the things that we're good at doing, and they can focus on building, like, the best code editor.

73. 14:32

    Uh, so our top coding cu- pa- partners, like GitHub, has used this, uh, to great effect, and well, uh, we've created an SDK, uh, for it that they use to directly integrate, uh, with Codex.

74. 14:45

    You can also use the SDK to, uh, control Codex as part of your CI/CD pipeline, as well as use it as an agent that directly interacts with your own agent as well.

75. 14:55

    Uh, if you really want to customize the agent layer, you can do it too. As an example of this, we worked with, closely, with the Cursor team to get the best performance out of the Codex, the model, not the agent.

76. 15:07

    We're bad at naming things. The model is different from the agent. [laughs] They did so by aligning their tools to be in distribution with how the model is trained, and they did so by aligning, uh, the harness with our open source, uh, implementation of Codex CLI.

77. 15:22

    All of this is publicly available. Uh, you can fork the repo, you can use it, our source code, you can use it, uh, go nuts.

78. 15:34

    So what does the future hold for Codex? It hasn't even been out for a year. Um, and especially with the last launch of Codex Max yesterday, like, things are really changing fast.

79. 15:46

    Uh, it's the fastest growing model in usage, now serving dozens of trillions of tokens per week, which has actually doubled since Dev Day.

80. 15:57

    It's always good to build where the models are going. It's safe to assume that the models will get better. They'll be able to get to work on much longer horizon tasks unsupervised.

81. 16:09

    New models will raise the trust ceiling. I trust these models now to do some way harder work than I would've six months ago, and that's gonna keep increasing. The future is about sprawling code bases and non-standard libraries and knowing how to work in closed source environments, matching existing templates and practices.

82. 16:29

    And the models, uh... And, and, and so you can imagine that the SDK will evolve to better support these model capabilities, letting the model learn as it goes and not repeat mistakes, and generally provide more surface area for an agent that writes code and uses a terminal to solve whatever problems it encounters.

83. 16:50

    And you can use that in your products via the SDK.

84. 16:56

    So what have we learned? Harnesses are really complicated and take a lot of work to maintain, especially with all the new models coming out. So we've built one for you inside of Codex that you can use off the shelf, or look at the source if you want to.

85. 17:12

    And you can use it to build new things outside of coding. And let us do all the work, making sure that you have the most capable computer agent,

86. 17:21

    and we're really excited to see what you craft. [outro music]