Creating Agents That Co-Create

Talk transcript

1. 0:00

   [on-hold electronic music] Hey, everyone.

2. 0:18

   My name's Karina, and I'm an AI researcher at OpenAI. Before that, I worked at Anthropic for about two years, uh, working on Claude. So today, I would love to chat more about what kind of scaling paradigms that has happened in the past two to four years in AI research and how those paradigms

3. 0:43

   unlocked new frontier product research. I'm also gonna share some of the vignettes from some of the lessons learned by developing Claude and ChatGPT products, some design challenges and lessons, um, and how do I think about the future of agents as they become from collaborators to co-innovators,

4. 1:08

   um, in the future. I would also love to invite you to engage, um, in the conversation, so I'd be more than happy to answer some of the questions at the end.

5. 1:20

   Cool. So not sure if probably the majority of you know this, but I think there are two scaling paradigms that has happened in AI research over the past few years.

6. 1:33

   The first paradigm is the next-token prediction, and you might have heard this as called pre-training. And what's really amazing about the next-token prediction is that it's a world-building machine.

7. 1:50

   The model learns to understand the world by predicting the next word. And I think fundamentally, if you think about it, this is happening because certain sequence is caused by initial action and this is irreversible.

8. 2:08

   And so the model learns some of the phys-physics of the world to understand. And the token can be anything, right? The tokens that we pre-trained are strings, words, pixels.

9. 2:19

   It could be anything. And so to predict what will happen next, the model needs to understand how the world works, and this is why pre-training works. And so you can imagine the next-token prediction is the massive multi-task learning.

10. 2:39

    And what's amazing about this is that during pre-training, some tasks are really easy to learn, such as translation, right? Like the word boarding in French is...

11. 2:53

    The model also learns a lot about the world. The capital of France is Paris. And because some of the information is much more present on the Internet and in some of the knowledge artifacts, the model has much easier time to learn this.

12. 3:10

    But we... Actually, the reason why compute is so important and scaling compute in the pre-training stage is so, so important is because there is a new class. There is a class of tasks that is, uh, really, really hard to learn.

13. 3:25

    And for example, the model learns a lot about the physics. It learns so much about the problem-solving, generation, and the logical expressions. It learns some of the spatial reasoning, although it's not perfect.

14. 3:40

    Um, but we are getting to the complexity of the tasks, such as math. When the model has to compute this number during the next-token prediction is actually really hard.

15. 3:53

    So that's why you need chain of thought or might spend more compute on the chain of thought to help the model to reason through more computational such tasks. Another class of tasks that I was thinking a lot about is creative writing.

16. 4:11

    It's actually really, really hard. And the reason why it's so hard for the model is because, you know, you can predict very nicely the style of the writing, but a lot of the creative writing is actually world-building and storytelling and the plot, and it's much, much easier for the model to make a mistake for the next to-token

17. 4:30

    prediction in such a way where it will completely deteriorate the plot coherence, which is really important for the stories. And this is an open-ended, like, research problem, um, creative writing in itself.

18. 4:44

    And the reason why it's because it's really, really hard to measure what is a good creative writing, what is not a creative writing. And obviously we would love for the models to invent new forms of writing and be extremely creative in their generations.

19. 5:00

    But this is actually one of the hardest AI research problems today is, um, how do we make mo-models to, like, write novels and have coherent

20. 5:13

    stories over the long course of the period of time?

21. 5:18

    So I think the era of twenty-twenty to twenty-twenty-one, that was an era of pre-scaling, pre-training a lot, both at, at Anthropic and at, at OpenAI. And actually the first...

22. 5:34

    At that time, one of the first products was GitHub Copilot, and I thought it was completely interesting product, the autocomplete, because it's so... In pertaining, the model has learned so much about the code and the next-token prediction for the code by the billions code tokens using from GitHub, open source projects, et cetera.

23. 5:57

    And what has happened for the autocomplete tab, tab, tab in the cursor or GitHub Copilot is that-

24. 6:05

    The researchers post-trained via RLHF, reinforcement learning from human feedback, and then reinforcement learning from AI feedback to make it extremely a little bit more useful to use. And this is where the era of post-training has gone off.

25. 6:21

    So in post-training, we teach the model how to complete function bodies, understanding docstrings, how to complete, generating multi-line completions, predicting the next diffs, apply the next diffs. And I think we are still in that era where there is so much more to be explored in the post-training stage of RLHF, RLAF to push the capabilities of

26. 6:46

    models to reason through complex code ba- code bases.

27. 6:52

    The next paradigm in AI research, which has happened last year, um, and it was published by OpenAI with a new model, o1, is scaling reinforcement learning on chain of thought.

28. 7:07

    And this is why we call them... It's highly complex reasoning. And you can imagine you, you spend a lot more test-time compute

29. 7:19

    on training in r- to scale reinforcement learning. And the reason why it works is because the model learns how to think during the training and learn from the feedback by having really good signals in RL.

30. 7:35

    So on the left, you can see the output of normal GPT-4o or GPT-4, and on the right, you can see the entire chain of thought that has been... that the model has thought about to solve the complex problems.

31. 7:52

    And as we think about harder and harder tasks, if you want the model to go from, you know, translation towards solving medical problems, you actually need t-to spend... You actually need the model to, like, spend a lot of time just thinking through the problem and completely creating more complex environments with tools and other, other tools

32. 8:17

    and more complex environments to think through and verify its outputs during the chain of thought. So as you can see, the chain of thought itself is very interesting, and the model is, has certain words that it does.

33. 8:33

    Um, but, um, there is a lot of science to be done in terms of, you know, faithfulness in the chain of thought. How do you measure the faithfulness? What happens if the model goes into, like, wrong direction?

34. 8:48

    Can it backtrack itself? I think there is a lot of science around that, and we are only at the beginning of it.

35. 8:55

    One of the first projects that I've done at OpenAI is actually how do we...

36. 9:03

    W- Uh, the interaction paradigm is very different now. So the interaction paradigm is the models thinks a lot to solve the problem if the problem is hard. So but how do we create this interaction, new interaction paradigm with humans such that it'll be much easier so that humans don't have to wait for fifteen seconds or thirty minutes

37. 9:25

    for a model to come back? And one of the things that we did, um, as a simple approach is to have, like, a streaming model thoughts to a user, and that way we had to communicate what exactly the summaries of the thoughts for the model and communicate very wisely to human.

38. 9:44

    But I think it's still one of the design challenges. Like, as the model's capabilities and interaction paradigms change, you have, like, new design challenges that you need to solve, um, for these types of models.

39. 9:58

    So and I guess, like, this year at OpenAI is the year of agents, and the way we think about it is highly complex reasoners, such as models trained on RL and chain of thought using real-world tools such as browsing, search, computer use, uh, over a long horizon period of time, over a long context.

40. 10:25

    But what's the next stage? In my view, the next stage level is co-innovators. And the way I'm thinking about is it's agents that is built upon all the things that we've done with reasoning and tool use and long context, plus creativity.

41. 10:43

    And creativity is enabled only through human-AI collaboration. And I think this is where I'm really, really excited about in the future, is to

42. 10:53

    create new affordances for humans to collaborate better with AI such that we both can co-create the future that we want.

43. 11:03

    And so those two scaling paradigms in AI research has unlocked us new kind of product research. And, you know, you can imagine product research being, oh, we have API from the model, and now we have to integrate in the products.

44. 11:20

    But it's actually what's happening on the ground is we have, like, a very... now we have, like, a very nice rapid iteration cycle of the product development. And the reason why is because we can use those highly reasoning models to distill back to smaller models or the models that we can iterate very, very fast.

45. 11:42

    And we can use those highly complex reasoning models to synthetically generate new data such that we can create new post-training, new datasets, new reinforcement learning environments. So, okay, um,

46. 11:56

    so one of the things that we can do is creating new, completely new class of tasks. And, um, you know, if the task is Um, how do we create a multiplayer collaboration, uh, with a human and AI?

47. 12:12

    You might want to simulate different users. Uh, and how do you do that? You might wanna, like, synthetically generate datasets of different users, conditioned on the different users, and post-train on that.

48. 12:26

    So it actually highly depends on, like, what kind of product experiences that you want to create, and extrapolate that to a new class of tasks that you want to post-train the models.

49. 12:37

    Um, I think we are moving towards more complex reinforcement learning environments, uh, which means we can allow models to use search or browsing or much more collaborative tools like Canvas during RL, such that they can learn how to be-- how to become better ca- at collaborating.

50. 12:58

    Um, we can leverage things like in-context learning. I think models are extremely, extremely good. So you can essentially create something, a new tool, and then the model will learn just via a few short examples.

51. 13:11

    And this is extremely rapid iteration cycle for any developer.

52. 13:16

    As I mentioned before, synthetic data via distillation is another thing. I think we can also invent new model behavior and interactions to utilize user feedback. So now we're gonna go through some of the vignettes that, um,

53. 13:32

    that has happened, um, from Anthropic to OpenAI. Um, I think the first concept that I've learned, um, is how do we bring unfamiliar capability into a familiar form factor?

54. 13:44

    And the reason why 100K context, uh, was successful is because we found, you know, file uploads is extremely familiar form factor. Everybody is working on documents. But you can imagine we could have deployed 100K context via infinite chats, such that it's, like, one huge long chat that you can interact with.

55. 14:05

    But I think finding the simplest form factor sometimes for unfamiliar capability, uh, is one of the design challenges in this new era. The next project that I worked, uh, well, the second project I was, r- I worked on at OpenAI is called ChatGPT Tasks.

56. 14:24

    Um, and actually, I did not realize about this until it was shipped. Um, you know, reminders and tasks, scheduled tasks, is actually, is a very familiar thing that people do almost every day.

57. 14:38

    But what's amazing about this product is that you can scale this with new kind of capabilities of the models. So ChatGPT Task is not just scheduled reminders and to-do lists.

58. 14:50

    It's actually you can create... You can ask the model to continue the story every day for you. Uh, or you can ask the model to search

59. 15:01

    everything that you are interested in every day or every other, every other day. So in a way, you can also, like, help yourself to, like, learn new language by having, like, extremely multimodal and interactive visualizations that ChatGPT can create.

60. 15:17

    And so I, I think this concept of your product feature should enable modular compositions that will scale very nicely in the future as the models will develop much higher capabilities, um, is one of the, is, is something that I've learned by doing ChatGPT Tasks.

61. 15:36

    Um, I think another design challenge that we have here is how do we bridge together real-time interaction with models to asynchronous task completion, and where we can ask the model to go off for, like, ten hours to research or write code and then come back with a solution.

62. 15:57

    And the bottleneck here is trust. And I believe that giving trust can be solved by giving humans new collaborative affordances to verify, edit model outputs and having them to give models real-time feedback so that the model can self-improve.

63. 16:18

    And, you know, one of the first products, uh, from Anthropic was actually Claude+Slack, and it was the first attempt to have a virtual teammate in your organization. And it was an amazing concept because Slack had all the affordances with tools and image uploads and multiplayer collaboration that you can create.

64. 16:39

    And there is something in there that is still... I think there is still a lot of things that we can do here and take the lessons from Claude+Slack to the next generational products.

65. 16:51

    Um, again, the task was also, like, very much inspired by Claude+Slack prototypes when Claude could just summarize channels, Slack channels across the organization every Friday and have the summaries for everybody.

66. 17:09

    My first project at OpenAI was Canvas, and

67. 17:15

    I thought that human-AI collaborative affordances could scale and create new creative capabilities. And what I really loved about the Canvas and the way we operate in Canvas as a team is that it was extremely flexible interface that we could come up with, a-and here, here are some of the vignettes that we had.

68. 17:38

    So the Canvas itself can become like co-creator and co-editor, and you can have, like, a very, very fine-grained editing, uh, interaction. The model can also do search in order to generate the report, and then you can also ask a question back, "Hi, h-verify this output."

69. 17:54

    And you can imagine this interface scales to multiplayer when other people can join your document or even multi-agents. If, uh, I can create a model critic or an editor, they can use You have like a multi-agentic and multiplay collaboration at the same time.

70. 18:11

    And so this is like a new design challenge that we need to navigate. How do we do that?

71. 18:16

    I'm also excited for personalized tutors. I think the models can-- are becoming extremely multimodal, extremely flexible, that you can learn new things in a new way, the way you like.

72. 18:29

    If I'm a visual learner and you're a more auditory learner, the model can adapt to your personalization. [instrumental music] Sorry. Um, one thing that, uh, I did [chuckles] yesterday is that, um, uh, I was on the plane, and actually I used Canvas to create me a game.

73. 18:49

    And so I really like this generative entertainment on the fly.

74. 18:55

    Anyone can create their own tools and web apps now, and I'm not sure what the future will be look like, but I think it will be extremely amazing if a non...

75. 19:07

    a person who never had to touch code ever before in their life, for the first time can create the tool that they really wanted and deploy that, um, for themselves or to start a business from scratch.

76. 19:20

    And I think there's something around pair programming and code creators that we can use in order to create the future that we want. And so Canvas has also become more of a pair programmer.

77. 19:36

    So the reason why Canvas is so flexible is because it was adap-- it was trained both to become collaborative for writing and coding.

78. 19:46

    Because it has tools, such as search, and can search for API documentation, it can become a data scientist too. And especially if you upload the entire CSV docs, um, it can generate a, a real-time, um, analysis.

79. 20:08

    And finally, what I'm really, really excited and what everybody is excited in AI is how do we actually help models to become better at research, um, and creating new knowledge.

80. 20:22

    And here you can see that the model, that the model and the human can co-create a document or co-create a new artifact that has never been h- happened before.

81. 20:35

    So here's a demo of the paper that, uh, I've co-published, and then, um, I'm asking the model to kind of reproduce it. And you can imagine this is like one of the, maybe one of the most common tasks in research is to reproduce or you can imagine reproduce like open source GitHub repo.

82. 20:55

    And you, you have like this very nice interactive

83. 21:00

    paradigm where because the model can also leverage its own internal knowledge, uh, you and a, you and a AI can work together to come up with new research hypotheses and verify certain like research, um, directions together.

84. 21:20

    And you can also handle, uh, delegate the tasks to an AI assistant to do that.

85. 21:28

    Finally, what I'm really excited about the future is that there will be this layer of invisible software creation for all. Um, and especially what I'm really excited is like f- from the mobile itself, people can just create their own software tools.

86. 21:45

    I think the way you interact with AI fundamentally changes in the way that the way you access the internet will also change. My prediction is that you will click less and less on the internet cl- links, and the way you will access the internet will be via models lens, which will be much cleaner and in a much

87. 22:06

    more personalized way. And, uh, you can imagine have like very personalized multimodal outputs. Let's say if I say, "I want to learn more about the solar system," instead of it giving me a text output, it should give you a three.js interactive visualization of the solar system, and you can have like highly richly interactive features to learn more.

88. 22:30

    And I think there is-- there'll be this like kind of cool future, like generative entertainment on the fly

89. 22:36

    for the people to learn and share new games with other people. I think the way I'm thinking about it is the kind of interface to AGI is, is blank canvas that kind of self morphs into your intent.

90. 22:51

    So for example, you come to the work today and your intention is to just write code. Then the canvas becomes more of an IDE, like a cursor or like a coding IDE, although the future programming might change.

91. 23:07

    Or if you're a writer and you decided to write a novel together, the model can start creating tools on the fly for you, such that it would be much easier for you to brainstorm or edit the writing or create character plots and visualize the structure of the plot itself.

92. 23:28

    And finally, I think the co-innovation is actually gonna happen with co-direction, creative co-direction with the models itself, and it's through collaboration with highly reasoning agentic systems, uh, that will be extremely capable of superhuman tasks to create new novels, films, games, uh, and essentially new

93. 23:53

    science, new knowledge creation. [instrumental music] Cool. Um, thank you so much. I think that's the end of my talk. Um- [clapping] [instrumental music]