AI Engineer World's Fair 2025
A Taxonomy for Next-Generation Reasoning Models
Read the talk
A Taxonomy for Next-Generation Reasoning Models
Reasoning models can retrieve obscure references yet struggle to plan reliable work. Skills, effort calibration, strategy, and abstraction suggest distinct training problems for more autonomous agents.
From a talk by Nathan Lambert
Finding the example you cannot name
Knowing how to solve a benchmark is different from finding a reference whose name you have forgotten. By June 2025, the wave of o1 and DeepSeek reasoning models had established a productive relationship between reinforcement learning during training and additional computation at inference. Much of the discussion centered on reinforcement learning with verifiable rewards (RLVR): rewarding answers that can be checked for correctness. Nathan Lambert wanted to know what came next beyond higher benchmark scores and answers consuming 10,000 tokens.
His everyday example was a familiar frustration for a researcher. He remembered an example of reward over-optimization but kept forgetting that the game was called CoastRunners. Searching for over-optimization meant repeatedly trying to recover the right reference. When he asked o3, he reports, it supplied the download link directly. The model recovered what he meant without requiring him to remember the name first.
That is an unexpected application of training initially associated with math and code. Lambert had made o3 his most-used model for finding information, and this retrieval success suggested that reasoning training could enable a broader range of useful behavior. It was an application anecdote, not a controlled comparison, but it made the research question concrete: which additional capabilities would turn isolated successes into dependable products?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning becomes a product capability
Deep Research extended that promise into a more steerable workflow. Lambert described directing it to inspect his website for typos or restricting its research to material published on that site. The useful capability was not simply producing a report; it was following a narrower definition of what to investigate.
Claude Code offered another kind of interaction. He used it to help build the website for his online book, treating it as a tool for enjoyable experiments. His enthusiasm came with a qualification: he was not evaluating it on demanding software-engineering work.
Codex represented a further step toward autonomous execution. Lambert thought the form factor was clearly promising, while finding its environment limiting for machine-learning tasks. At the time of the talk, he reported that it had no GPUs and had only recently gained internet access. Working with open models could require looking up Hugging Face configurations, so those environmental constraints mattered independently of the model’s reasoning ability.
His prediction was that these products would become everyday tools within the following six months. That was a forecast about the emerging applications of reasoning models, with useful behavior still dependent on access to the tools and resources a task required.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Longer tasks require deliberate capability work
A chart of task horizons placed these applications in a longer progression. Lambert read the period around GPT-4o as showing signs of saturation, followed by newer Sonnet models and o1 extending the frontier. He described the vertical axis loosely as how long a task could be completed by a model, and connected the subsequent gains to reasoning training.
Editor: In METR’s original March 2025 analysis, the task horizon measures task duration for a human expert at a specified probability of agent success. It does not measure the agent’s own running time.
Extending the curve requires identifying and training missing capabilities. A trend line does not make the next gain automatic. Lambert’s proposed next step was to move beyond individual reasoning skills toward planning: what would a research program need to teach a model so that it could work autonomously on a substantial task?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Skills, calibration, strategy, and abstraction
Lambert’s provisional taxonomy separates four traits that are easy to conflate when judging whether a model is good at reasoning:
| Trait | What the model needs to do | Distinction that matters |
|---|---|---|
| Skills | Solve problems in areas such as math and code. | Competence at an individual problem is the starting point. |
| Calibration | Match output-token expenditure to problem difficulty. | Here, calibration concerns effort, not the accuracy of reported probabilities. |
| Strategy | Choose a promising direction and recognize alternative approaches. | Backtracking within an approach is different from abandoning it and restarting with another plan. |
| Abstraction | Break a larger task into tractable subtasks. | The model must decide what it can solve, where tools help, and when to call a larger model. |
Skills were already the clearest success of reasoning training. Calibration became more consequential as each task consumed more computation. Strategy and abstraction were Lambert’s proposed subdivisions of planning: first choose a direction, then turn it into work that can actually be carried out.
Humans often supply that decomposition today. Training a model to solve math problems does not explicitly teach it how to divide an open-ended task among its own abilities, tools, and other models. In Lambert’s research agenda, those behaviors needed to become training targets in their own right.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the existing training loop teaches
The basic RLVR loop is straightforward:
- Give the model a prompt.
- Let it generate a completion.
- Score that completion using the available verification procedure.
- Use the scored completions to update the model’s weights.
The diagram Lambert described was a single-turn loop. Multi-turn interactions and tool calls make the formulation more complex, but the central mechanism remains generation followed by feedback and a weight update.
The progression from GPT-4o to o1 and then o3 showed how much this additional training could change performance on difficult evaluations. Lambert pointed to Humanity’s Last Exam and AIME while arguing that planning needed a comparable capability-building effort. Better scores on existing reasoning tasks did not establish that models could reliably organize longer work.
The repertoire of individual skills was also expanding. Lambert highlighted tool use in newer DeepSeek and Qwen models, and described o3’s usefulness as the combination of tools with reasoning, math, and code.
What sits above tool use was harder to describe precisely. Claude had a reputation for strong agentic behavior, but Lambert emphasized that there was no equally established way to measure or communicate that quality across models. Calling a tool successfully is observable; judging whether a model directs a whole task well requires a broader evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How much thinking does addition need?
At the time of the talk, products largely delegated effort calibration to the user. ChatGPT offered model selection; Claude exposed an extended-thinking toggle; Gemini had similar controls; APIs offered reasoning-effort settings. Each control asked the user to predict how much reasoning the next task deserved.
A model that could make that decision itself would reduce both the selection burden and unnecessary token expenditure. The problem becomes more expensive as the available reasoning budget grows.
Consider 2 + 3. Lambert described examples in which reasoning models used hundreds to roughly a thousand tokens for an addition whose answer could be a single token. A separate sequence-length comparison on the slide contrasted a standard instruction model with the QwQ thinking model. In the comparisons he discussed, switching to reasoning could increase token expenditure by 10–100×; this was not a universal multiplier for every task or model.
Extra tokens create infrastructure load, cost, and waiting time. For an easy question, the user should not have to wait minutes or switch providers to avoid unnecessary deliberation. Calibration means spending effort where it changes the result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a direction before spending the budget
Effort calibration leads to a second question: is the model spending that effort in a useful direction? Lambert took an example problem from Epoch AI’s FrontierMath material and tried it with DeepSeek-R1-0528. He wanted to see whether the model would plan before beginning the solution.
In his account, it immediately began constructing a polynomial. It did not first sketch the problem or compare possible approaches. This was a probe of one problem, but it illustrated the distinction between starting to reason and choosing a strategy.
Lambert expected an answer of 10,000–40,000 tokens. He then imagined increasing that budget by another factor of ten: following the wrong direction could consume multiple dollars and substantial latency without producing useful progress. Those were an expectation and a hypothetical extension, not measured results from the probe.
He described the applications under discussion as operating within roughly one-to-thirty-minute latency windows. A bad overall strategy and excessive work on a single subproblem can both exhaust that window. From the user’s perspective, either failure can mean abandoning the task before it finishes.
Products such as Deep Research and Claude Code appeared to Lambert to be prompted to plan at the start. His proposal was to make that behavior native to the model through training, so that choosing an approach would not depend entirely on instructions supplied by the application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the plan executable
A plan creates practical questions about memory and execution. Lambert described Claude Code compressing its memory when the context window filled, without assuming that this was the best memory policy for every application. An agent also needs to avoid repeating mistakes—a problem he connected to the earlier discussion of playing Pokémon. Remembering previous attempts matters only if that information changes what the agent tries next.
Abstraction also means producing manageable parts and allocating additional thinking to the difficult ones. That may involve offloading work to another model or calling several models in parallel.
Lambert’s memorable example was the manual version: people opening tmux and running Claude Code in ten windows. The desired model capability was not merely to launch more processes. It was to recognize when parallel work would help, decide how to divide the task, and choose an appropriate way to approach it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bootstrap planning with expert demonstrations
Lambert’s reconstruction of the path from Q* to Strawberry to o1 emphasized the effort required to establish new behaviors. In his account, OpenAI spent twelve to eighteen months developing initial reasoning traces, with human data helping make backtracking and verification reliable. He used that interpretation of reasoning-model development to motivate a comparable effort for planning, rather than presenting it as a disclosed training recipe.
Planning demonstrations may be easier for experts to produce. Writing a convincing 10,000-token reasoning trace with backtracking is awkward; writing a good five-to-ten-step plan, or checking a model-generated plan, is a more natural expert task.
The proposed training path is therefore concrete: collect initial planning demonstrations, apply supervised fine-tuning (SFT) to teach those behaviors, and then investigate whether reinforcement learning on larger tasks reinforces useful planning styles. The last step remains a research question. A model can learn to emit a plan without necessarily learning which plans lead to successful execution.
Lambert also suggested adding more structure to the output sequence. Reasoning models already distinguish thinking from answering; a hypothetical extension would be plan → think → answer. The purpose would be to choose an approach before committing a large reasoning budget. This was a proposed training structure, not an implemented API.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding one fact is different from reliable research
The CoastRunners example shows a strong individual skill: recovering niche information when the user cannot remember the right search terms. A research task asks for more than that one successful retrieval.
Lambert contrasted excellent Deep Research results with disappointing ones and attributed some of that inconsistency to missing planning ability. An electronics-purchase recommendation makes the gap tangible. A model may be good at finding facts yet still fail to gather the right information or investigate the alternatives thoroughly enough to justify a recommendation.
Reliable coverage requires decisions about both what to investigate and how much effort each part deserves. Search supplies information; planning determines whether the collected information is sufficient for the task.
The boundaries between strategy, abstraction, and context management need not be rigid. Lambert’s purpose was to decompose the training problem: identify the behavior that is missing, then ask what data or algorithmic method could teach it. The taxonomy is useful insofar as it makes those research choices more specific.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parallel compute for robustness
Lambert used o1 Pro as an example of robustness and was still anticipating o3 Pro. He distinguished two possible benefits of additional computation: making a previously unsolved task solvable, and making an already available capability more reliable or producing a better-crafted answer.
His interpretation was that reinforcement learning could encourage exploration, while additional parallel computation could help exploit the approaches available to the model. This was an intuition about how compute might improve results, not an account of the products’ internal architecture. Parallel work could be valuable when robustness mattered, but it would not solve every limitation or replace the need for new capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with suitable data and stable RL
Continual learning raises a larger possibility: repeatedly update a model through long-horizon tasks, potentially reducing its dependence on pretraining. Lambert saw an unresolved algorithmic bottleneck there. Scaling reinforcement learning further was, in his view, a more immediate and tractable direction.
For his work at the Allen Institute for AI, or Ai2, he reduced the practical reasoning-model research plan to a small number of priorities.
- Collect questions with verified answers. Seek coverage across domains, while recognizing that much of the available data is math and code.
- Filter by difficulty relative to the base model. Lambert’s illustrative extremes were questions solved zero times in 100 attempts or 100 times in 100 attempts. He argued that both waste computation and make RL updates noisier. The useful training set depends on what this particular starting model can sometimes, but not always, solve.
- Build a stable RL run. Train over the selected questions with infrastructure that allows sustained improvement. Suitable data and reliable training are the foundation of the recipe.
Only after those foundations are working does Lambert turn to refinements such as overlong filtering, changes to clipping, or resetting the reference model. He characterizes their contribution as a few additional percentage points on top of a working run, rather than a substitute for good data and stable infrastructure. The practical decision rule is to fix the data and training loop before expecting an algorithmic adjustment to rescue the run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When post-training becomes a major compute budget
The closing provocation was whether post-training should simply be called training. Lambert offered a conditional calculation: suppose o1’s post-training used 1% as much compute as its pretraining, then apply the 10× increase he reported for o3. On that assumed baseline, the result would be roughly 10%, not parity. Reaching comparable pretraining and post-training GPU-hours would require further growth.
The point of the scenario was the changing scale of the investment. A phase once treated as a small addition could become a substantial part of the training budget.
For DeepSeek-V3, Lambert cited a post-training share of 0.18% of GPU-hour compute and described pretraining as taking approximately two months.
Editor: The DeepSeek-V3 Technical Report, v2, reports 5,000 post-training H800 GPU-hours out of 2,788,000 total GPU-hours, or about 0.18%. This is the reported final training run, including context extension; research and ablation costs are excluded.
The basis for his DeepSeek-R1 comparison was different. Lambert recalled a deleted researcher tweet saying that R1 training took a few weeks. Assuming that RL used the same entire cluster, he tentatively estimated a 10–20% compute share. He explicitly regarded that assumption as strong and probably inaccurate, so the estimate should be read as his speculative comparison, not a DeepSeek-disclosed training cost.
He also noted that DeepSeek’s pretraining infrastructure might be substantially more efficient than its RL infrastructure. Elapsed duration alone cannot settle the compute comparison when cluster allocation and implementation efficiency differ.
The research direction is to spend that growing RL budget on models that can carry out longer plans: break a task into parts, decide how to approach those parts, and solve them with the resources available. That makes autonomous decomposition a training objective alongside the reasoning skills that made the first wave of applications possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original CoastRunners example behind Lambert's opening search anecdote: scoring rewards and finishing the race can diverge.
Explains the human-time task-horizon metric underlying the progress chart, including its reliability threshold and limits.
The training-cost table grounds the V3 post-training comparison at the end of the talk.
Further reading
Lambert's companion article develops the same four traits and links the contemporaneous calibration research.
- THOUGHTTERMINATORPaper
A paper linked from Lambert's taxonomy essay on matching reasoning-token expenditure to task difficulty.
Read the complete timestamped transcript
- 0:00
[upbeat music] I really came to this thinking about trying to reflect on six months into this, like, reinforcement learning with verifiable rewards, post o1, post DeepSeek.
- 0:23
And I think that a lot of this stuff is somewhat boring because everybody has a reasoning model. Um, we all know the basics of you can scale RL at training time, and the numbers will go up, and that's deeply correlated with being able to then do this inference-time scaling.
- 0:38
Um, but really in AI right now, everybody- there's a lot of people who are up to speed, but the crucial question is, like, where are things going to go, and how do you skate where the puck is going?
- 0:47
So a lot of this talk is really me trying to process is, like, where is this going besides getting high benchmark scores with using ten thousand tokens per answer?
- 0:57
And, like, what do we need to do to actually train these models? And what are the things that OpenAI, et cetera, are probably already doing, but it's increasingly hard to get that, uh, signal out of them.
- 1:08
So if we look at this, like, reasoning is really also unlocking really new language model applications. I think I-I... This is the same search query, which is, like, I-- as a RL researcher, I need to find this all the time.
- 1:20
I forget that it's called CoastRunners, and you Google, like, over-optimization twenty times to find it. But I tried asking o3, and it, like, literally gave me the download link directly, so I didn't even have to do anything.
- 1:31
And that's a very unusual use case to just pop out of this reasoning training where math and code was the real thing to start with. And o3 is great.
- 1:41
It's the model that I use the most for finding information, and this just really is the signal that I have that a lot of new, interesting things are coming down the pipe.
- 1:49
Um, I would say it's starting to unlock a lot of new language model applications that I use some of these. So this is a screenshot of Deep Research. It's great.
- 1:58
You can use it in really creative ways, like, uh, prompt it to look at your website and find typos or look at ev- only the material on your website and things like this.
- 2:07
It's actually more steerable than you, than you may expect. Um, Claude Code, which I describe as just the, the vibes are very good. It's fun. I'm not a serious software engineer, so I don't use it on hard things.
- 2:18
But I use it for fun things because I can. I can put the company API key in and just kind of mess around, like helping me build my-- the website for this book that I wrote online.
- 2:27
And then there's the really serious things, which are like Codex and these fully autonomous agents that are starting to come. If you play with it, it's obvious that the form factor is going to be able to work.
- 2:37
I'm sure there are people that are getting a lot of value out of it right now. I think for ML tasks, it's like there's no GPUs in it right now, and if you are dealing with open models, it's like they just added internet.
- 2:48
So, like, it wasn't gonna be able to go back and forth and look at, like, Hugging Face configs or something, and all these headaches that you don't wanna deal with.
- 2:54
But in the six months, like, all of these things are gonna be stuff you should be using on a day-to-day basis, and this is all downstream of this kind of step change in performance from reasoning models.
- 3:05
And then this is kind of like another plot that's been talked about. And when I look at this, it's like through twenty twenty-four, if we look at like GPT-4o, it-- things a lot-- it really were saturating then, and then there's these new Sonnet models in o1, which really helped push out the frontier and time horizon.
- 3:22
So this is the y-axis is how long a, a task can roughly be completed by the models i-in time, which is kind of a weird way to measure it 'cause things will get faster.
- 3:32
But, um, it's going to keep going, and this reasoning model is the technique that was kind of unlocked in order to figure out how to push the limits. And when you look at things like this, it's not that just we're, like, on a path determined from AI and more gains are going to come.
- 3:46
It's really, like, we have to think about what the models need to be able to do in order to keep pushing out these frontiers. So there's a lot of human effort that goes into continuing the trends of AI progress.
- 3:56
So it's like gains aren't free, and I'm thinking that a lot of planning and kind of thinking about training in a bit of a different way beyond just reasoning skills is gonna be what helps push this and enable these, uh, language modeling applications and products that are kind of in their early stages to really shine.
- 4:15
So this is a core question that I'm thinking about is, like, what do I have to do to come up with a research plan to train reasoning models that can work autonomous- autonomously and really have meaningful ideas for what planning would be?
- 4:28
So I kind of came up with a taxonomy that has a few different what I call traits within it. Um, the first one is skills, which we've pretty much already done.
- 4:36
Skills are, like, getting really good at math and code. Inference time scaling was useful to getting there, but they kind of could become more research-y over time. I think for products, calibration is gonna be crucial, which is like these models overthink like crazy, so they need to be able to kind of have some calibration to how many
- 4:54
output tokens are used relative to the difficulty of the problem. And this will kind of become more important when we're spending more on each task that we're planning. And then the last two are subsets of planning that I'm thinking about and happy to take feedback on this taxonomy, but, like, strategy, which is just going in the right
- 5:11
direction and knowing different things that you can try. Because it's really hard for these language models to really change course, where they can backtrack a little bit, but restarting their plan is hard.
- 5:21
And then as tasks become very hard, we need to do abstraction, which is like the model has to choose on its own how to break down a problem into different things that it can do on its own.
- 5:32
I think right now humans would often do this, but if we want language models to do very hard things, they have to make a plan that has subtasks that are actually tractable, or it calls in a bigger model to do that for it.
- 5:43
But these are things that are-- the models aren't gonna do natively. Natively, they're trying to, like, doing math problem-solving. Like, that doesn't have clear abstraction on, like, this task it can do and with this additional tool and all these things.
- 5:56
So this is, this is a new thing that we're gonna have to add. So to kind of summarize, it's like we have skills, we have research or calibration, and I'll highlight some of it, but, like, planning is a new frontier where people are talking about it, and we really need to think about, like, how we will actually
- 6:11
put this into the models. So to just put this up on the slide, what we call reinforced learning with verifiable rewards looks very simple. I think a lot of RL and language models, especially before you get into this multi-turn setting, has been you take prompts, the agent creates a completion to the prompt, and then you score the
- 6:31
completions. And with those scored completions, you can update the weights to the model. It's been single turn. It's been very simple. We'll have- I'll have to update this diagram for multi-turn and tools, and it makes it a little bit more complex.
- 6:42
But the core of it is just a language model generates completions and gets feedback on it.
- 6:49
And it's good to just take time to look at these skills. These are a collection of evals, and we can look at, like, where GPT-4o was, and these were the hardest evals that have existed and look-- were t- called, like, the frontier of AI.
- 7:02
And if we look at the o1 improvements and the, like, o3 improvements in quick succession, these are really incredible eval gains that are mostly just from adding this new type of training in.
- 7:13
And the core of this argument is that we need to do something similar if we want planning to work. So I would say that a lot of the planning tasks look mostly like humanity's last exam in AMY, um, just after adding this reasoning skill, and we need to figure out what other types of things these models are
- 7:30
gonna be able to do. So it's like this list of reasoning abilities that these kind of s- like, low-level skills is gonna continue to go up. I think the most recent one, if you look at recent DeepSeek models or recent Qwen models, is really this tool use being added in.
- 7:48
And, uh, I-- that's gonna build more models like o3. So using o3 just feels very different because it is this kind of combination of tool use with reasoning, and it's obviously good at math and code.
- 8:00
But I think these kind of low-level skills that we expect from reasoning training are-- we're gonna keep getting more of them as we figure out what is useful. I think an abstraction for the kind of agentic-ness on top of tool use is going to be very nice, but it's hard to measure.
- 8:14
And people mostly say that Claude is the best at that, but it's not yet super established on how we measure it or communicate it across different models.
- 8:23
And then this is where we get into the fun, interesting things. I think it's hard for us because calibration is passed to the user, which is we have all sorts of things like model selectors if you're a ChatGPT user.
- 8:35
Um, Claude has reasoning on/off with this extended thinking, and Gemini has something similar, and there's these reasoning effort selectors in the API. And this is really rough on a user side of things, and making it so the model knows this will just really make it so it's easier to find the right model for the job and just
- 8:52
kind of, um, your kind of over-- spent tokens for no reason will go down a lot. It's kind of obvious to want it, and then it'll just-- it becomes a bigger problem the longer we don't have this.
- 9:03
Some examples from when overthinking was kind of identified as a problem, it's like the, um, left half of this is you can ask a language model, like, what is two plus three?
- 9:12
And you can see these reasoning models use hundreds to a thousand tokens for something that could realistically be, like, one token as an output. And then on the right is a kind of comparison of sequence lengths from a standard, like, non-RL trained instruction model versus the QWQ thinking model.
- 9:30
And you really can gain this, like, ten to a hundred x in token spend when you shift to a reasoning model. And if you do that in a way that is wasteful, it's just gonna really load your infrastructure and cost.
- 9:40
And as a user, I don't wanna wait minutes for an easy question, and I don't wanna have to switch models or providers to deal with that.
- 9:49
So I think one of the things that once we start to have this calibration is, like, um, is this kind of strategy idea. And on the right, I t- I went to the, um, I think it's Epoch AI website.
- 9:59
I took a quest- one of their example questions from Frontier Math, and I was like, "Does this new DeepSeek R1 0528 model, like, does it do any semblance of planning when it starts?"
- 10:09
And you ask it a math problem, it's just like, "Okay, the first thing I'm gonna do is I n- I need to construct a polynomial." It's like, it just goes right in, and it doesn't do anything like trying to sketch the problem before it thinks.
- 10:21
And this is gonna probably output ten to forty thousand tokens, and if it's gonna need to do another ten x there, it's just like, if that's all in the wrong direction, that's multiple dollars of spend and a lot of latency that's just totally useless.
- 10:34
And most of these applications are set up to expect a latency between one and thirty minutes. So it's like there, there is just a timeout they are fighting. So either going in the wrong direction or just thinking way too hard about a sub-problem is just gonna make it so the user leaves.
- 10:47
So, um, right now, these models, I said they do very little planning on their own. But as we look at these applications, they're very likely prompted to plan, which is like the beginning of Deep Research and Claude Code.
- 10:59
And we kinda have to make it so that is model native rather than something that we do manually.
- 11:06
And then once we look at this plan, there's all these implementation details across something like Deep Research or Codex, which is like, how do I manage a memory? So we have Claude Code compresses its memory when it fills up its context window.
- 11:18
We don't know if that's the optimal way for every application. We want to avoid repeating the same mistakes. We talked, uh, Greg was talking about the playing Pokémon ear-earlier, which is a great example of that.
- 11:30
We want to have trackable parts. We wanna offload thinking if we have a really challenging part. So I'll talk about parallel compute a little bit later as a way to kind of boost through harder things.
- 11:41
And really, we want mo- language models to call mer- multiple other models in parallel. So right now, people are spinning up Tmux and launching Claude Code in ten windows to do this themselves, but there's no reason a language model can't be able to do that.
- 11:56
It just needs to know the right way to approach it.
- 12:00
And as, as I've started with this idea of kind of we need effort for tr-- or like we need to make effort to add new, um, capabilities into language models, when you-- when I think about this kind of story of Q* that became Strawberry, that became o1, the reason that it was in the news for so long
- 12:19
and was such a big deal is, like, it was a major effort for OpenAI, spending, like, twelve to eighteen months building these initial reasoning traces that they could then train an initial model on that has some of these behaviors.
- 12:33
So it took a lot of human data to get things like backtracking and verification to be reliable in their models. And we need to go through a similar arc with planning, but with planning, the kind of outputs that we're gonna train on are, are much more intuitive than something like reasoning.
- 12:48
I think if I were to ask you to sit down and write a 10,000 token reasoning trace with backtracking, it's like you can't really do this. But a lot of expert people can write a five to 10-step plan that is very good or check the work of Gemini or OpenAI when asked to, um, write an initial plan.
- 13:05
So I'm a lot more optimistic on being able to hill climb on this, and then it goes through the same path where once you have initial data, you can do some SFT, and then the hard question is if the RL and even bigger tasks can reinforce these planning styles.
- 13:22
On the right, I added kind of a hypothetical, which is like we already have thinking tokens before answer tokens, and there's no reason we can't apply more structure to our models to just really make them plan out their answer before they think.
- 13:39
So, um, to give a bit more depth on this idea of skill versus planning, if we go back to this example, I would say that o3 is extremely skilled at search.
- 13:52
So being able to find a piece of niche in- niche information that researchers in a field know of but can't quite remember the exact search words, that is an incredible skill.
- 14:02
But when you try to put this into something like deep research, there's this lack of planning is making it so that sometimes you get a masterpiece and sometimes you get a dud.
- 14:11
And if, as these models get better at planning, it'll just be more thorough and reliable in getting the kind of coverage that you want. So it's like if-- It's crazy that we have models that can do this search, but if you ask it to recommend, um, some sort of electronics purchase or something, it, it's really hard to
- 14:29
trust 'cause it can't just know how to pull in the right information and how h-hard it should try to do all that coverage.
- 14:39
So to kind of summarize, these are the four things that I presented. I think you can obviously add more to these. You could call a mix of strategy and abstraction.
- 14:47
There's like con-- You could call what I was describing, uh, as like context management in many ways. But really, you just wanna have things like this so that you can break down the training problem and think about data acquisition or new algorithmic methods for kind of each of these tasks.
- 15:05
And I mentioned parallel compute because I think this is an interesting one because if you use o1 Pro, it's still been one of the best models and the most robust models for quite some time, and I'm v-been very excited for o3 Pro.
- 15:18
But it doesn't solve problems in the same way as like traditional inference-time scaling, where inference-time scaling just made a bunch of things that didn't work go from zero to one, where this parallel compute is really like it makes things more robust.
- 15:31
It just makes them nicer, and it seems like this kind of RL training is something that can encourage exploration. And then if you apply more compute in parallel, it feels something kind of exploiting and f-getting a really well-crafted answer.
- 15:42
So there's a time when you want that, but it doesn't solve every problem.
- 15:47
And to kind of transition into the end of this talk, it's like there's been a lot of talks today saying the things that you can do with RL, and there's obviously a lot of talk on the ground of, um, what is called continual learning and if we're just continually using very long-horizon RL tasks to update a model
- 16:02
and diminish the need of pre-training. And there are a lot of data points that were closer to that in many ways. I think continual learning has a big, um, algorithmic bottleneck where-- But just like scaling up RL further is very tractable and something that is happening.
- 16:18
So if people are to ask me what I'm working on at Ai2 and what I'm thinking about, this is my like rough summary of, uh, what I think a research plan looks like to train a reasoning model without, without all the in-between-the-line details.
- 16:32
So step one is you just get a lot of questions that have verified answers across a wide variety of domains. Um, most of these will be math and code because that's what out the- what is out there.
- 16:43
And then two, if you look at all these recipe papers, they're having a step where they filter the questions based on the, the difficulty with respect to your base model.
- 16:53
So if a question is solved zero out of a hundred times by your base model or a hundred out of a hundred, you don't want questions that look like that 'cause you're both not only wasting compute, but you're messing up the gradients in your RL updates to make them a bit noisier.
- 17:07
And once you do that, you just wanna make a stable RL run that'll go through all these questions and have the numbers keep going up. And that's the core of it, is really stable infrastructure and data.
- 17:17
And then you can tap into all these research papers that tell you to do methods like overlong filtering or different clipping or resetting the reference model, and that'll give you a few percentage points on the top, where really it's just data and stable infrastructure.
- 17:32
And this kind of leads to the provocation, which is like, what if we rename post-training as training? And if OpenAI o1 was like 1% of compute is post-training relative to pre-training, um, they've already said that o3 has increased it by ten perc- 10x.
- 17:50
So if the, if the numbers started at 1%, you're very quickly getting to, um, what you may see as like parity in compute in terms of GPU hours between pre-training and post-training, which if you were to take anybody back a year ago before o1, would seem pretty unfathomable.
- 18:08
And one of the fun data points for this is that, um, the DeepSeek-V3 paper, and you kind of watch DeepSeek's transition into becoming more serious about post-training. Like the original DeepSeek-V3 paper, they used 0.18% of compute on post-training in GPU hours, and they said their pre-training takes about two months, and there was a deleted tweet from one
- 18:29
of their RL researchers that said the R1 training took a few weeks. So if you make a few very strong, probably not completely accurate assumptions that RL was on the sa- whole cluster, that would already be ten to 20% of their compute, I think.
- 18:44
Like specific things for DeepSeek are like, oh, their pre-training efficiency is probably way better than their RL code and things like this. But scaling RL is a very real thing if you, if, if you look at this, if you look at Frontier Labs, and you look at the types of tasks that people wanna co-solve with these long-term
- 19:01
plans. So it's good to kind of embrace what you think these models will be able to do and kind of break down tasks on their own and solve some of them.
- 19:11
So thanks for having me, and let me know what you think. [audience applauding] [upbeat music]