A Taxonomy for Next-Generation Reasoning Models

Talk transcript

1. 0:00

   [upbeat music] I really came to this thinking about trying to reflect on six months into this, like, reinforcement learning with verifiable rewards, post o1, post DeepSeek.

2. 0:23

   And I think that a lot of this stuff is somewhat boring because everybody has a reasoning model. Um, we all know the basics of you can scale RL at training time, and the numbers will go up, and that's deeply correlated with being able to then do this inference-time scaling.

3. 0:38

   Um, but really in AI right now, everybody- there's a lot of people who are up to speed, but the crucial question is, like, where are things going to go, and how do you skate where the puck is going?

4. 0:47

   So a lot of this talk is really me trying to process is, like, where is this going besides getting high benchmark scores with using ten thousand tokens per answer?

5. 0:57

   And, like, what do we need to do to actually train these models? And what are the things that OpenAI, et cetera, are probably already doing, but it's increasingly hard to get that, uh, signal out of them.

6. 1:08

   So if we look at this, like, reasoning is really also unlocking really new language model applications. I think I-I... This is the same search query, which is, like, I-- as a RL researcher, I need to find this all the time.

7. 1:20

   I forget that it's called CoastRunners, and you Google, like, over-optimization twenty times to find it. But I tried asking o3, and it, like, literally gave me the download link directly, so I didn't even have to do anything.

8. 1:31

   And that's a very unusual use case to just pop out of this reasoning training where math and code was the real thing to start with. And o3 is great.

9. 1:41

   It's the model that I use the most for finding information, and this just really is the signal that I have that a lot of new, interesting things are coming down the pipe.

10. 1:49

    Um, I would say it's starting to unlock a lot of new language model applications that I use some of these. So this is a screenshot of Deep Research. It's great.

11. 1:58

    You can use it in really creative ways, like, uh, prompt it to look at your website and find typos or look at ev- only the material on your website and things like this.

12. 2:07

    It's actually more steerable than you, than you may expect. Um, Claude Code, which I describe as just the, the vibes are very good. It's fun. I'm not a serious software engineer, so I don't use it on hard things.

13. 2:18

    But I use it for fun things because I can. I can put the company API key in and just kind of mess around, like helping me build my-- the website for this book that I wrote online.

14. 2:27

    And then there's the really serious things, which are like Codex and these fully autonomous agents that are starting to come. If you play with it, it's obvious that the form factor is going to be able to work.

15. 2:37

    I'm sure there are people that are getting a lot of value out of it right now. I think for ML tasks, it's like there's no GPUs in it right now, and if you are dealing with open models, it's like they just added internet.

16. 2:48

    So, like, it wasn't gonna be able to go back and forth and look at, like, Hugging Face configs or something, and all these headaches that you don't wanna deal with.

17. 2:54

    But in the six months, like, all of these things are gonna be stuff you should be using on a day-to-day basis, and this is all downstream of this kind of step change in performance from reasoning models.

18. 3:05

    And then this is kind of like another plot that's been talked about. And when I look at this, it's like through twenty twenty-four, if we look at like GPT-4o, it-- things a lot-- it really were saturating then, and then there's these new Sonnet models in o1, which really helped push out the frontier and time horizon.

19. 3:22

    So this is the y-axis is how long a, a task can roughly be completed by the models i-in time, which is kind of a weird way to measure it 'cause things will get faster.

20. 3:32

    But, um, it's going to keep going, and this reasoning model is the technique that was kind of unlocked in order to figure out how to push the limits. And when you look at things like this, it's not that just we're, like, on a path determined from AI and more gains are going to come.

21. 3:46

    It's really, like, we have to think about what the models need to be able to do in order to keep pushing out these frontiers. So there's a lot of human effort that goes into continuing the trends of AI progress.

22. 3:56

    So it's like gains aren't free, and I'm thinking that a lot of planning and kind of thinking about training in a bit of a different way beyond just reasoning skills is gonna be what helps push this and enable these, uh, language modeling applications and products that are kind of in their early stages to really shine.

23. 4:15

    So this is a core question that I'm thinking about is, like, what do I have to do to come up with a research plan to train reasoning models that can work autonomous- autonomously and really have meaningful ideas for what planning would be?

24. 4:28

    So I kind of came up with a taxonomy that has a few different what I call traits within it. Um, the first one is skills, which we've pretty much already done.

25. 4:36

    Skills are, like, getting really good at math and code. Inference time scaling was useful to getting there, but they kind of could become more research-y over time. I think for products, calibration is gonna be crucial, which is like these models overthink like crazy, so they need to be able to kind of have some calibration to how many

26. 4:54

    output tokens are used relative to the difficulty of the problem. And this will kind of become more important when we're spending more on each task that we're planning. And then the last two are subsets of planning that I'm thinking about and happy to take feedback on this taxonomy, but, like, strategy, which is just going in the right

27. 5:11

    direction and knowing different things that you can try. Because it's really hard for these language models to really change course, where they can backtrack a little bit, but restarting their plan is hard.

28. 5:21

    And then as tasks become very hard, we need to do abstraction, which is like the model has to choose on its own how to break down a problem into different things that it can do on its own.

29. 5:32

    I think right now humans would often do this, but if we want language models to do very hard things, they have to make a plan that has subtasks that are actually tractable, or it calls in a bigger model to do that for it.

30. 5:43

    But these are things that are-- the models aren't gonna do natively. Natively, they're trying to, like, doing math problem-solving. Like, that doesn't have clear abstraction on, like, this task it can do and with this additional tool and all these things.

31. 5:56

    So this is, this is a new thing that we're gonna have to add. So to kind of summarize, it's like we have skills, we have research or calibration, and I'll highlight some of it, but, like, planning is a new frontier where people are talking about it, and we really need to think about, like, how we will actually

32. 6:11

    put this into the models. So to just put this up on the slide, what we call reinforced learning with verifiable rewards looks very simple. I think a lot of RL and language models, especially before you get into this multi-turn setting, has been you take prompts, the agent creates a completion to the prompt, and then you score the

33. 6:31

    completions. And with those scored completions, you can update the weights to the model. It's been single turn. It's been very simple. We'll have- I'll have to update this diagram for multi-turn and tools, and it makes it a little bit more complex.

34. 6:42

    But the core of it is just a language model generates completions and gets feedback on it.

35. 6:49

    And it's good to just take time to look at these skills. These are a collection of evals, and we can look at, like, where GPT-4o was, and these were the hardest evals that have existed and look-- were t- called, like, the frontier of AI.

36. 7:02

    And if we look at the o1 improvements and the, like, o3 improvements in quick succession, these are really incredible eval gains that are mostly just from adding this new type of training in.

37. 7:13

    And the core of this argument is that we need to do something similar if we want planning to work. So I would say that a lot of the planning tasks look mostly like humanity's last exam in AMY, um, just after adding this reasoning skill, and we need to figure out what other types of things these models are

38. 7:30

    gonna be able to do. So it's like this list of reasoning abilities that these kind of s- like, low-level skills is gonna continue to go up. I think the most recent one, if you look at recent DeepSeek models or recent Qwen models, is really this tool use being added in.

39. 7:48

    And, uh, I-- that's gonna build more models like o3. So using o3 just feels very different because it is this kind of combination of tool use with reasoning, and it's obviously good at math and code.

40. 8:00

    But I think these kind of low-level skills that we expect from reasoning training are-- we're gonna keep getting more of them as we figure out what is useful. I think an abstraction for the kind of agentic-ness on top of tool use is going to be very nice, but it's hard to measure.

41. 8:14

    And people mostly say that Claude is the best at that, but it's not yet super established on how we measure it or communicate it across different models.

42. 8:23

    And then this is where we get into the fun, interesting things. I think it's hard for us because calibration is passed to the user, which is we have all sorts of things like model selectors if you're a ChatGPT user.

43. 8:35

    Um, Claude has reasoning on/off with this extended thinking, and Gemini has something similar, and there's these reasoning effort selectors in the API. And this is really rough on a user side of things, and making it so the model knows this will just really make it so it's easier to find the right model for the job and just

44. 8:52

    kind of, um, your kind of over-- spent tokens for no reason will go down a lot. It's kind of obvious to want it, and then it'll just-- it becomes a bigger problem the longer we don't have this.

45. 9:03

    Some examples from when overthinking was kind of identified as a problem, it's like the, um, left half of this is you can ask a language model, like, what is two plus three?

46. 9:12

    And you can see these reasoning models use hundreds to a thousand tokens for something that could realistically be, like, one token as an output. And then on the right is a kind of comparison of sequence lengths from a standard, like, non-RL trained instruction model versus the QWQ thinking model.

47. 9:30

    And you really can gain this, like, ten to a hundred x in token spend when you shift to a reasoning model. And if you do that in a way that is wasteful, it's just gonna really load your infrastructure and cost.

48. 9:40

    And as a user, I don't wanna wait minutes for an easy question, and I don't wanna have to switch models or providers to deal with that.

49. 9:49

    So I think one of the things that once we start to have this calibration is, like, um, is this kind of strategy idea. And on the right, I t- I went to the, um, I think it's Epoch AI website.

50. 9:59

    I took a quest- one of their example questions from Frontier Math, and I was like, "Does this new DeepSeek R1 0528 model, like, does it do any semblance of planning when it starts?"

51. 10:09

    And you ask it a math problem, it's just like, "Okay, the first thing I'm gonna do is I n- I need to construct a polynomial." It's like, it just goes right in, and it doesn't do anything like trying to sketch the problem before it thinks.

52. 10:21

    And this is gonna probably output ten to forty thousand tokens, and if it's gonna need to do another ten x there, it's just like, if that's all in the wrong direction, that's multiple dollars of spend and a lot of latency that's just totally useless.

53. 10:34

    And most of these applications are set up to expect a latency between one and thirty minutes. So it's like there, there is just a timeout they are fighting. So either going in the wrong direction or just thinking way too hard about a sub-problem is just gonna make it so the user leaves.

54. 10:47

    So, um, right now, these models, I said they do very little planning on their own. But as we look at these applications, they're very likely prompted to plan, which is like the beginning of Deep Research and Claude Code.

55. 10:59

    And we kinda have to make it so that is model native rather than something that we do manually.

56. 11:06

    And then once we look at this plan, there's all these implementation details across something like Deep Research or Codex, which is like, how do I manage a memory? So we have Claude Code compresses its memory when it fills up its context window.

57. 11:18

    We don't know if that's the optimal way for every application. We want to avoid repeating the same mistakes. We talked, uh, Greg was talking about the playing Pokémon ear-earlier, which is a great example of that.

58. 11:30

    We want to have trackable parts. We wanna offload thinking if we have a really challenging part. So I'll talk about parallel compute a little bit later as a way to kind of boost through harder things.

59. 11:41

    And really, we want mo- language models to call mer- multiple other models in parallel. So right now, people are spinning up Tmux and launching Claude Code in ten windows to do this themselves, but there's no reason a language model can't be able to do that.

60. 11:56

    It just needs to know the right way to approach it.

61. 12:00

    And as, as I've started with this idea of kind of we need effort for tr-- or like we need to make effort to add new, um, capabilities into language models, when you-- when I think about this kind of story of Q* that became Strawberry, that became o1, the reason that it was in the news for so long

62. 12:19

    and was such a big deal is, like, it was a major effort for OpenAI, spending, like, twelve to eighteen months building these initial reasoning traces that they could then train an initial model on that has some of these behaviors.

63. 12:33

    So it took a lot of human data to get things like backtracking and verification to be reliable in their models. And we need to go through a similar arc with planning, but with planning, the kind of outputs that we're gonna train on are, are much more intuitive than something like reasoning.

64. 12:48

    I think if I were to ask you to sit down and write a 10,000 token reasoning trace with backtracking, it's like you can't really do this. But a lot of expert people can write a five to 10-step plan that is very good or check the work of Gemini or OpenAI when asked to, um, write an initial plan.

65. 13:05

    So I'm a lot more optimistic on being able to hill climb on this, and then it goes through the same path where once you have initial data, you can do some SFT, and then the hard question is if the RL and even bigger tasks can reinforce these planning styles.

66. 13:22

    On the right, I added kind of a hypothetical, which is like we already have thinking tokens before answer tokens, and there's no reason we can't apply more structure to our models to just really make them plan out their answer before they think.

67. 13:39

    So, um, to give a bit more depth on this idea of skill versus planning, if we go back to this example, I would say that o3 is extremely skilled at search.

68. 13:52

    So being able to find a piece of niche in- niche information that researchers in a field know of but can't quite remember the exact search words, that is an incredible skill.

69. 14:02

    But when you try to put this into something like deep research, there's this lack of planning is making it so that sometimes you get a masterpiece and sometimes you get a dud.

70. 14:11

    And if, as these models get better at planning, it'll just be more thorough and reliable in getting the kind of coverage that you want. So it's like if-- It's crazy that we have models that can do this search, but if you ask it to recommend, um, some sort of electronics purchase or something, it, it's really hard to

71. 14:29

    trust 'cause it can't just know how to pull in the right information and how h-hard it should try to do all that coverage.

72. 14:39

    So to kind of summarize, these are the four things that I presented. I think you can obviously add more to these. You could call a mix of strategy and abstraction.

73. 14:47

    There's like con-- You could call what I was describing, uh, as like context management in many ways. But really, you just wanna have things like this so that you can break down the training problem and think about data acquisition or new algorithmic methods for kind of each of these tasks.

74. 15:05

    And I mentioned parallel compute because I think this is an interesting one because if you use o1 Pro, it's still been one of the best models and the most robust models for quite some time, and I'm v-been very excited for o3 Pro.

75. 15:18

    But it doesn't solve problems in the same way as like traditional inference-time scaling, where inference-time scaling just made a bunch of things that didn't work go from zero to one, where this parallel compute is really like it makes things more robust.

76. 15:31

    It just makes them nicer, and it seems like this kind of RL training is something that can encourage exploration. And then if you apply more compute in parallel, it feels something kind of exploiting and f-getting a really well-crafted answer.

77. 15:42

    So there's a time when you want that, but it doesn't solve every problem.

78. 15:47

    And to kind of transition into the end of this talk, it's like there's been a lot of talks today saying the things that you can do with RL, and there's obviously a lot of talk on the ground of, um, what is called continual learning and if we're just continually using very long-horizon RL tasks to update a model

79. 16:02

    and diminish the need of pre-training. And there are a lot of data points that were closer to that in many ways. I think continual learning has a big, um, algorithmic bottleneck where-- But just like scaling up RL further is very tractable and something that is happening.

80. 16:18

    So if people are to ask me what I'm working on at Ai2 and what I'm thinking about, this is my like rough summary of, uh, what I think a research plan looks like to train a reasoning model without, without all the in-between-the-line details.

81. 16:32

    So step one is you just get a lot of questions that have verified answers across a wide variety of domains. Um, most of these will be math and code because that's what out the- what is out there.

82. 16:43

    And then two, if you look at all these recipe papers, they're having a step where they filter the questions based on the, the difficulty with respect to your base model.

83. 16:53

    So if a question is solved zero out of a hundred times by your base model or a hundred out of a hundred, you don't want questions that look like that 'cause you're both not only wasting compute, but you're messing up the gradients in your RL updates to make them a bit noisier.

84. 17:07

    And once you do that, you just wanna make a stable RL run that'll go through all these questions and have the numbers keep going up. And that's the core of it, is really stable infrastructure and data.

85. 17:17

    And then you can tap into all these research papers that tell you to do methods like overlong filtering or different clipping or resetting the reference model, and that'll give you a few percentage points on the top, where really it's just data and stable infrastructure.

86. 17:32

    And this kind of leads to the provocation, which is like, what if we rename post-training as training? And if OpenAI o1 was like 1% of compute is post-training relative to pre-training, um, they've already said that o3 has increased it by ten perc- 10x.

87. 17:50

    So if the, if the numbers started at 1%, you're very quickly getting to, um, what you may see as like parity in compute in terms of GPU hours between pre-training and post-training, which if you were to take anybody back a year ago before o1, would seem pretty unfathomable.

88. 18:08

    And one of the fun data points for this is that, um, the DeepSeek-V3 paper, and you kind of watch DeepSeek's transition into becoming more serious about post-training. Like the original DeepSeek-V3 paper, they used 0.18% of compute on post-training in GPU hours, and they said their pre-training takes about two months, and there was a deleted tweet from one

89. 18:29

    of their RL researchers that said the R1 training took a few weeks. So if you make a few very strong, probably not completely accurate assumptions that RL was on the sa- whole cluster, that would already be ten to 20% of their compute, I think.

90. 18:44

    Like specific things for DeepSeek are like, oh, their pre-training efficiency is probably way better than their RL code and things like this. But scaling RL is a very real thing if you, if, if you look at this, if you look at Frontier Labs, and you look at the types of tasks that people wanna co-solve with these long-term

91. 19:01

    plans. So it's good to kind of embrace what you think these models will be able to do and kind of break down tasks on their own and solve some of them.

92. 19:11

    So thanks for having me, and let me know what you think. [audience applauding] [upbeat music]