AI Engineer World's Fair 2026
The Base Model is Dead
Read the talk
The Base Model Is Becoming a Prior for Reinforcement Learning
As reinforcement learning takes on reasoning and software tasks, pre-training recipes are changing to teach the representations and behaviors those tasks need.
From a talk by Varun Singh
Before you start: Familiarity with next-token prediction, supervised fine-tuning, and reinforcement learning will help; mixture-of-experts routing is explained where it matters.
What should a base model contain?
What should a base model learn before anyone teaches it to chat, reason, or use tools? The traditional answer was a broad representation of human knowledge, acquired by predicting text from a large portion of the internet. Varun Singh, introducing himself as Arcee AI’s pre-training lead, opens The Base Model Is Dead by immediately qualifying its title: the model is still needed; the familiar conception of its job is changing.
The training pipelines for Trinity Large Thinking, GLM-4.5, and GLM-5 all retain a pre-training phase. That shared starting point matters: later stages depend on representations learned before the model becomes an assistant or an agent.
In a decoder-only transformer, next-token prediction supplies the learning objective. Given a prefix, the model learns to predict its continuation. Repeating that process across text builds representations of language and world knowledge. The GPT-3 paper illustrates a further consequence: a model trained this way can learn to use examples in its context to perform a task. This training is often called self-supervised learning because the text supplies its own targets; Singh also notes the unsupervised and supervised terminology used for it.
GPT-3’s recipe makes the original conception concrete. Common Crawl supplied a large web scrape, WebText2 supplied another, and books and Wikipedia contributed additional material. Web sources account for roughly 85% of GPT-3’s training mixture, using the paper’s sampling weights rather than raw corpus sizes. Singh then points to Llama 3, describing 50% of its training tokens as general knowledge. The categories are different, but both examples establish the prominence of broad knowledge in the starting model.
In that earlier division of labor, post-training largely taught the model how to surface what it already knew: follow a chat template, answer questions, and behave usefully in conversation. Reinforcement learning shaped the interaction more than it supplied new capabilities. Pre-training therefore consumed the bulk of the compute budget and largely determined the ceiling of the resulting assistant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
RL becomes a source of capability
The arrival of OpenAI o1 in 2024 and DeepSeek-R1 in January 2025 changed that allocation of responsibility. o1 demonstrated the potential of reasoning models; R1 made a route to building them more openly accessible. Reinforcement learning could now substantially improve task performance, rather than merely adjust conversational behavior. Singh uses o1’s AIME competitive-mathematics graphs to illustrate the shift.
Claude Code extended the practical setting to a terminal, where developers could ask a model to build applications. As function calling improved, the interaction became more than a question followed by an answer: a model could act within a software environment. Training that interaction end to end with RL lets the model learn how to work in the environment and construct software.
If RL is responsible for learning increasingly consequential behavior, pre-training should prepare the model to learn that behavior well. The question is no longer just how much knowledge a base model contains. It is whether a conventional web-trained model provides the best starting point for large-scale reasoning and agent training.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two routes away from a web-heavy recipe
The emerging recipes do not agree on whether synthetic data is essential. Singh favors it, but presents the direction of the research as unsettled. His first example is MAI-Thinking-1, whose recipe he describes as excluding synthetic data and outputs from other language models, including filtering web scrapes for such material. This retains the idea of using human-produced knowledge to bootstrap representations.
Yet retaining human-produced data does not mean retaining the old mixture. Singh puts MAI-Thinking-1’s web-text share at 15%, compared with roughly 85% for GPT-3. His comparison illustrates a change in emphasis, rather than an independently established equivalence between the two recipes’ categories. Web text still matters, but code and STEM receive more attention as downstream uses demand those capabilities.
The alternative is to move data associated with post-training into pre-training itself. In the Nemotron 3 Ultra report, Singh highlights three categories prefixed with SFT in the recipe chart. These are question-and-answer or conversational examples of the kind normally associated with supervised fine-tuning. Putting them into the earlier mix exposes the model to the shape of conversations and expected downstream tasks from the beginning.
The two approaches disagree about data provenance while converging on a more task-oriented recipe:
| Recipe direction | Data choice | Intended preparation |
|---|---|---|
| MAI-Thinking-1, as described by Singh | Human-produced data; less web text | Greater code and STEM emphasis |
| Nemotron 3 Ultra, as interpreted by Singh | Synthetic and SFT-style examples earlier | Familiarity with downstream conversations and tasks |
Books and general web text occupy less of the mix. Singh contrasts GPT-3’s lack of a dedicated code dataset with the prominence of code in newer recipes. The shift in what a base model is for does not depend on every lab choosing synthetic data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rephrasing changes the exposure, not just the token count
Synthetic data raises a legitimate concern: indiscriminately feeding generated text back into training can degrade performance or contribute to model collapse. Singh’s concrete counterexample is Trinity Large, where he reports using synthetic data at web scale, largely through rephrasing. Start with a seed item and generate multiple expressions of the same information. Adding those variants to the mix increases exposure to the information without requiring the model to encounter exactly the same wording each time.
At the record level, the operation can be represented by keeping a seed identifier attached to every variant. In this illustrative Python example, the candidate additions express the same fact in different forms:
python
seed = {
"id": "dict-keys",
"text": "Python dictionary keys must be hashable.",
}
rephrases = [
"A Python dictionary requires hashable keys.",
"To use an object as a dictionary key in Python, it must be hashable.",
]
candidate_records = [
{"seed_id": seed["id"], "text": text}
for text in [seed["text"], *rephrases]
]
This constructs candidate training records; the operation itself does not establish that generated variants preserve the seed’s meaning. The mechanism Singh describes is repeated exposure to the same information through different expressions.
Singh cites Kimi K2 as a larger-scale example of rephrasing applied broadly across a pre-training dataset. He also points to SwallowCode and SwallowMath, introduced in Rewriting Pre-Training Data Boosts LLM Performance in Math and Code, as earlier examples of the approach.
The intended benefit is therefore broader than acquiring more tokens. Rewriting can clean up material, improve its usefulness, and present it in instruction or agent-oriented forms. Those forms let the model begin learning downstream task representations during pre-training, rather than first encountering them after the base model is finished.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Earlier exposure can reduce later distribution shifts
For mixture-of-experts models, moving downstream data earlier also has a routing motivation. Experts specialize over training, while load-balancing objectives encourage broadly equal utilization across a batch or a sequence, depending on the objective. A model trained on one distribution can route a very different distribution unevenly when post-training begins.
Singh cites MAI-Thinking-1 as an example of a distribution shift producing large expert imbalances, and describes the response as increasing the balancing coefficient during SFT. The report is more specific: it gives a load-balancing coefficient of 1e-2 during self-distillation and 1e-5 during RL. Singh’s preferred direction is to expose the model to relevant data early enough that it learns stable representations, reducing the need to force a major adjustment to expert balance late in training.
Mid-training addresses a related mismatch. It exposes the model to distributions it will encounter during post-training and RL, often at longer context lengths. Longer contexts admit agentic traces into the training mix, preparing the model for interactions that extend beyond a short question and answer. As pre-training itself adopts longer contexts, Singh argues that these datasets can also move earlier, allowing the corresponding representations to develop from the start.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the learning objectives from the stage names
Once conversational data, agentic traces, and longer contexts can appear at multiple stages, the labels pre-training, mid-training, and post-training become less informative. Singh proposes a simpler distinction between two learning paradigms:
- Supervised next-token learning: learn from supplied sequences and their continuations.
- Reinforcement learning: improve behavior through the training environment and its feedback.
The important relationship is how the first prepares the model for the second, wherever a dataset happens to enter the schedule.
Compute allocation makes that relationship consequential. Citing an interview with the head of Xiaomi’s MiMo Labs, Singh reports roughly equal pre-training and post-training compute allocations for the final model. He presents Composer 2.5 as a more extreme case, claiming that its RL compute exceeded its supervised-learning compute. Cursor’s public description groups additional training and RL together, so it does not isolate the RL-only comparison.
As RL takes a larger role in the budget, supervised learning acquires a different optimization target: build representations that make subsequent RL effective. The base model still matters, but its value increasingly includes what it enables the next learning process to discover.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give RL skills it can compose
What must the supervised phase supply? Singh highlights research suggesting that the base model needs exposure to atomic skills that RL can later compose. With an environment of sufficient difficulty, RL can learn to extrapolate from those ingredients. This makes the training distribution important at two levels: the model needs useful component skills, and the RL environment needs to demand something beyond merely reproducing them.
The familiar AlphaGo comparison illustrates the possibility of reinforcement learning eventually overtaking supervised learning. But Singh leaves its applicability to language models unresolved. Human language is an exceptionally broad distribution to acquire through RL alone. Less supervised learning and more RL remain a possible direction, not a demonstrated endpoint for LLM training. Even without reaching that endpoint, thinking of a base model as a collection of skills available for RL becomes increasingly useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare the model for the behavior you want next
Some of the useful ingredients may have forms that are uncommon in ordinary human writing. Reasoning traces are one example: their structure differs from much of the text that a broad web scrape would supply. Supervised learning can introduce these unfamiliar forms before RL asks the model to use them effectively.
The same idea extends to test-time compute schemes. Singh describes warming a model up to such schemes during SFT, or even during pre-training, using excerpts he attributes to a Zyphra report. The underlying purpose is to choose data that prepares the model to explore effectively during RL. Supervised data can teach not only information, but also forms of behavior that later training will develop.
That changes the meaning of the base model: from a general prior over human knowledge toward a prior for reasoning and agentic behavior. Singh acknowledges that this framing reflects the current prominence of reasoners and agents. If a different way of interacting with models becomes important, its demands should shape the prior instead. The durable design question is what the model’s eventual use requires it to be ready to learn—not how closely its training corpus resembles a massive scrape of the web.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
GPT-3's original paper explains its training mixture, sampling weights and evaluation of in-context learning.
Microsoft's technical report describes model development, evaluation, reinforcement learning and expert load balancing.
NVIDIA's report documents the architecture and training pipeline of Nemotron 3 Ultra.
Introduces SwallowCode and SwallowMath and evaluates rewritten data for continual pre-training in code and mathematics.
Further reading
Zyphra's report on a reasoning-focused mixture-of-experts model with 700M active and 8B total parameters.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi everyone. My name is Varun. I'm the pre-training lead at Arcee AI, and the talk I'm gonna be giving today is, uh, called The Base Model is Dead.
- 0:21
Uh, but not really. Um, the idea of the base model that we have, um, kinda is like built on, uh, this idea of like training on super large-scale web text, and the base model kind of being a reflection of like the whole knowledge of like the, uh, human internet.
- 0:42
Um, you can see in like, uh, these... I've, I've taken these, uh, from a bunch of, uh, different papers on like the entire LLM training process. Um, our own model, Arcee lo- uh, Trinity Large Thinking, um, the process looked kinda like the simplified diagram on the left.
- 1:00
I've taken the top one from, uh, GLM 4.5, the bottom one from GLM 5.
- 1:05
Um, all these have a pre-training phase, and, uh,
- 1:11
pre-training is like the stage where the model loo- uh, accumulates world knowledge, builds useful representations, uh, all through next-token prediction on, um, web text. Um, I've got a simplified, uh, transformer diagram, um, decoder-only transformer, and, um, a screenshot from the GPT-3 paper that talks about
- 1:35
how, um, language models can, uh, learn how to do in-context learning through, um, unsupervised or self-supervised or, um, some- some even just call it supervised, um, learning on, uh, through next-token prediction.
- 1:56
Um, the way that, uh, older base models were trained was, like I said, mostly on, uh, things that reflected the entirety of human knowledge. Uh, so Common Crawl, uh, which is like a commonly available web scrape, uh, made up most of the training dataset for GPT-3.
- 2:16
Um, WebText2, another web scrape dataset. Um, some sources from like, uh, books as well, um, and Wikipedia as like a high-quality, uh, representation of human knowledge. Um, you can see that, uh, WebText alone here, including like Wikipedia, makes up like roughly eighty-five percent of the whole training mix.
- 2:38
Um, looking at, uh, the bottom with Llama 3, um, WebText still kind of makes up a majority of, of the, um, model's training data with like fifty percent of the tokens corresponding to general knowledge.
- 2:56
Back then, post-training was mostly shaping the model to use, um, the parts, it to like surface, uh, the knowledge that it accumulates in-- accumulated in pre-training, uh, in like a chat interface.
- 3:11
So mostly allowing the model to adapt to a chat template, to the question answer format, um, and be us- be useful in an interaction that way. Um, RL was mostly just a cherry on top, um, shaping the, you know, flavor of the interactions more than conferring extra, um, knowledge or quality onto the base model itself.
- 3:36
Now, in, in like this realm of, of, of how language models used to be, pre-training, um, and the base model kind of defined how good you were able to get a m- a model.
- 3:50
Um, it was like the bulk of the compute, uh, budget, and it was,
- 3:56
um, the, the like core of the, uh, uh, training process. Um, however, uh, this kind of changed l- a lot last year when, um, Open- OpenAI-- I guess 2024 actually, OpenAI released, uh, o1, um, pioneering reasoning models, and DeepSeek, uh, also released R1 in January
- 4:20
2025, um, allowing the whole world to know how to build these types of language models. And
- 4:30
now we have this new, uh, new, um, use for reinforcement learning, which is no longer a cherry on top, but it can dramatically improve the performance of, of the model on various different tasks.
- 4:45
Um, the, the famous graphs from o1 there talking about, uh, AIME performance, um, competitive math contest.
- 4:54
Um, and then even later in the year, we, we saw CloudCode, um, come into being as a way for developers to easily, um, kinda use a language models in a, in a terminal to build out applications as models got stronger and stronger on things like function calling.
- 5:13
Um, and then people realized you could, uh, you know, RL this end to end. Um, and now models could learn how to interact with software environments and build software and, uh, perform really useful work.
- 5:28
And so, um, the question then becomes like,
- 5:33
is your standard base model still, uh, what the best, uh, what would be the best, um,
- 5:41
prior for the, for the, this large-scale reinforcement learning phase that, uh, reasoners and, um, agentic models now use? And we can kinda see like in the, in, uh, a few open research papers what the trend is, where the trend is going.
- 5:59
Um, and interestingly enough, the, um, it seems like-
- 6:05
Not super clear yet. Um, I have my opinions on, like, synthetic data being the way forward, but I'm-- I've got, like, two contrasting, uh, perspectives here kind of in the slide.
- 6:18
The top image is from the Mei- uh- Thinking 1 paper, where they make a, make a really large point to not use any synthetic data or any, uh, data from any other language model.
- 6:29
Um, and they really try to, you know, filter their web scrapes for this as well, um, in order f- uh, to kind of adhere to, like, the previous,
- 6:38
um, paradigm of, like, uh, using human knowledge as a way to bootstrap, uh, model representations and, like,
- 6:46
uh, capabilities. Um, but I would say that this is also... Like, even, even though they stuck with no synthetic data, the data mix that they've chosen here is still, um, totally different from what you'd expect in, like, a, um, in a, in a classical, uh, language model.
- 7:05
And I mean, the, the main reason for that is that web text, which used to make, like, up to eighty-five percent of the trained data in GPT, uh, three, is now all the way down at fifteen percent.
- 7:17
And that-- I mean, that just shows that, uh, the value of, like, web text contributing to, like, the downstream, um, performance of, like, the models on RL and stuff is kind of, uh...
- 7:33
It-it's still important, but taking a back seat to things like code and STEM abilities as the models kind of gain more real-world use cases related to, to those.
- 7:43
Um, the other approach, uh, is to bring, um, post-training data and large-scale synthetic data back, uh, through-- pull it back through the process into the pre-training phase. Um, the, the bottom chart I've taken from Nemotron 3 Ultra, um, where they reveal their, um, the data recipe, and I'm not sure how readable it is, but these top
- 8:09
three, um, on the left b- uh, pie chart, the top three on the kind of right, right side of it, uh, they're all labeled SFT, uh, with, with SFT as a prefix.
- 8:20
And that's the type of question announcer kind of chat data set that you'd s- you'd expect to see only in post-training. But by pulling it back into the process, they're able to, like, get the model to learn, um, the shape of these conversations and what kind of tasks they might be expected to do downstream, um, from the
- 8:41
very beginning of the pre-training process. Um, and, uh, this follows, like, um, a similar, um, trend in, like, diminishing, uh, the amount of web text used in the model.
- 8:55
Um, yeah. Um, it's really interesting to see the, the Nemotron series lean so heavily into synthetic data, but, uh, Mei Thinking 1 kind of lean in the opposite direction.
- 9:11
Um, I've, I've, I've just got this slide here as, like, a con- uh, easy contrast of-- uh, people can see on the, like, amount of web text and the amount of books and stuff, um, being less of a percentage here.
- 9:26
And GPT-3 didn't even use to have any specific code data sets, but now code is, like, the dominating, um,
- 9:35
data, data subset that we have in, uh, pre-training recipes.
- 9:41
Um, so I mentioned synthetic data, but what is actually, uh... Like, how is synthetic data used? There's a lot of, uh, talk around synthetic data that, you know, blindly tossing it into a model can cause the model to collapse and, uh,
- 9:56
and performance to tank. But there's been a lot of work and, uh, even, like, a large-scale, you know, example of this, uh, turning out really well. Um, so in our own, in our, in our own, uh, model, Trinity Large, we, uh, had a large amount of, uh, web, uh, web scale synthetic data, um, mostly through rephrasing, where
- 10:20
you take a seed data item and you sort of upsample it in the mix by, uh, generating synthetic rephrases of the same information. So the model sees the same information in, like, multiple ways.
- 10:32
Um, the bottom two, uh, screenshots are from Kimi K two, um, an even larger scale model that, uh, broadly use this, um, across the whole pre-training dataset. Um, the top, uh, top right, um, screenshot is from a paper that, uh, resulted in the dataset Swallowcode and Swallowmath, which are early examples of
- 10:57
this. But the trend seems to be that, um, synthetic data not only allows you to get more and more tokens, uh, but also, you know, clean up tokens, get higher quality tokens, and have, um, tokens that are shaped more like instruct or agentic tasks all the way back in pre-training and, uh, allowing the model to, like, learn
- 11:20
those task representations from the very beginning. Um, another reason that it's, uh, beneficial to add post-training data early in pre-training is now with MoEs. Um, one of the biggest, uh, pain points in training an MoE is dealing with load balancing, um, where experts can specialize over the course of training and, um, they'll-- uh, and load balancing
- 11:45
objectives aim to, uh, achieve broadly equal utilization of the experts, um, in a given batch or a sequence, depending on the objective. Um, without, uh, post-training data in, uh, early in pre-training, uh, with an MoE, one really easy pitfall, uh, that we can fall into is this-
- 12:05
Is kind of illustrated in the MEI Thinking 1 report, which is that the data distribution that the model sees, uh, in post-training is really, really different, uh, compared to what it sees in pre-training.
- 12:19
And this can cause massive imbalances, and, um, MEI overcame it by, uh, really cranking up the load balancing coefficient during the SFT stages.
- 12:31
Um, but I mean, ideally you don't want to mess with the balance that far into training, and the model should learn stable representations from really early on.
- 12:44
Another, uh, interesting thing that, um, is changing in base models now is that, yeah, there's this whole advent in mid-training, uh, which is exposing the model to the distribution that it would see during post-training and RL and at a longer context, so for things like agentic traces to be allowed into the mix and, uh, to kind of
- 13:05
help prepare the model that way. Um, a lot of models though are training with much longer context in pre-training, and there's no reason that these datasets can't be pulled back into the mix to allow for more stable representations from the very beginning.
- 13:20
Um, I think, uh, a better, a better way to understand the current f- uh, phase of LLM training, uh, isn't so much like pre-training, mid-training, post-training, RL. Uh, it all gets a bit muddy that way.
- 13:33
But the- there's two broad paradigms that are like-- that really help build a LLM today, and that's supervised learning, uh, through next-token prediction, and RL. Um, and RL is becoming more and more important.
- 13:47
Um, the bottom, uh, thing is a screenshot from a interview, um, with, uh, the head of, uh, Xiaomi's Mimo Labs, uh, where she talks about how they allocate compute, uh, between research, pre-training, and post-training.
- 14:04
And pre-training and post-training in the final model have a roughly equal compute allocation. Um, Composer 2.5 takes this to the extreme, where, um, Koso really, uh, sank much, much more RL compute into the model than the model had ever seen in supervised learning.
- 14:21
But with RL dominating such a massive amount of the compute budget, uh, it makes sense to view supervised learning as a way specifically to prepare the model for, uh, to build useful representations for R- for RL instead of it being the bulk of like, um,
- 14:40
what the model would be used for, like, uh, previously.
- 14:44
Um, there's been some, some work on, uh, how supervised learning affects RL. Um, I really like this one paper where the main takeaways are basically that, uh, the base model needs to have some exposure to like, uh, like the atomic skills that it would need to compose during RL, and, um, the model can learn to extrapolate from
- 15:06
there during RL given like the environment has a sufficient level of difficulty.
- 15:13
Um, I had to put in the classic r- AlphaGo graph there, where RL eventually overtakes supervised learning. Uh, it's unclear if we'll see something like that for language models because, of course, you know, uh, human language is such an insane distribution to have to like learn through reinforcement learning alone.
- 15:31
Um, but it's, it's definitely possible that we might see diminished supervised learning and more and more RL, uh, which makes this kind of thinking of a base model as, um, atomic skills for RL more and more valuable.
- 15:46
Another thing that some labs are doing, um, is kind of introducing novel data, uh, during supervised learning. And by novel I mean something that the model really wouldn't have seen the shape of before.
- 15:57
Uh, easy example is, you know, reasoning traces. They don't really look like a, like a ton of what, um, humans output. And, um, another interesting thing is like training for test time compute, um, schemes, um, by kind of warming the model up to them during SFT, um, or even pre-training itself.
- 16:19
Um, this, these screenshots were taken from, uh, Zyphra's ZyA 1.0 paper.
- 16:24
Um, and I think that they're, uh, very interesting ways of thinking about how data can affect, um,
- 16:32
like, uh, the skills needed to explore well in RL.
- 16:37
Um, in conclusion, um, base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors. Um, of course, that's kind of, uh, reductive in the, in a, in a way that reasoning, reasoners and agents are like the main way we see, uh, bots, uh, the main way we see these
- 16:59
chatbots used now. But if a new paradigm were to take off, or like a new way of interacting with the models, um, it makes sense to like think of a base model as building a prior for that instead of, um,
- 17:13
just building off like a, a massive, uh, scrape of web text. And yeah. Thanks for, thanks for listening. Thanks for your time.
- 17:24
And yeah. [clapping] [outro jingle]