# The Base Model is Dead

Varun Singh

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/xbPriQWXtWM-base-model-is-dead) · [Video](https://www.youtube.com/watch?v=xbPriQWXtWM)

## The Base Model is Dead

As reinforcement learning takes on reasoning and software tasks, pre-training recipes are changing to teach the representations and behaviors those tasks need.

Familiarity with next-token prediction, supervised fine-tuning, and reinforcement learning will help; mixture-of-experts routing is explained where it matters.

### What should a base model contain?

[0:00](https://www.youtube.com/watch?v=xbPriQWXtWM&t=0s)

What should a base model learn before anyone teaches it to chat, reason, or use tools? The traditional answer was a broad representation of human knowledge, acquired by predicting text from a large portion of the internet. Varun Singh, introducing himself as Arcee AI’s pre-training lead, opens *The Base Model Is Dead* by immediately qualifying its title: the model is still needed; the familiar conception of its job is changing.

The training pipelines for **Trinity Large Thinking**, **GLM-4.5**, and **GLM-5** all retain a pre-training phase. That shared starting point matters: later stages depend on representations learned before the model becomes an assistant or an agent.

![Slide titled “What is a base model?” with an Arcee training flowchart on the left and GLM training diagrams on the right.](https://ai.engineer/api/talk-essay-media/cd374d07e54e240704e711f48723f20dabc087cc882543c049386a872937747b)

Training pipelines for Arcee Trinity Large Thinking, GLM-4.5, and GLM-5.

In a decoder-only transformer, **next-token prediction** supplies the learning objective. Given a prefix, the model learns to predict its continuation. Repeating that process across text builds representations of language and world knowledge. The [**GPT-3** paper](https://arxiv.org/html/2005.14165v4) illustrates a further consequence: a model trained this way can learn to use examples in its context to perform a task. This training is often called self-supervised learning because the text supplies its own targets; Singh also notes the unsupervised and supervised terminology used for it.

GPT-3’s recipe makes the original conception concrete. Common Crawl supplied a large web scrape, WebText2 supplied another, and books and Wikipedia contributed additional material. Web sources account for roughly 85% of GPT-3’s training mixture, using the paper’s sampling weights rather than raw corpus sizes. Singh then points to **Llama 3**, describing 50% of its training tokens as general knowledge. The categories are different, but both examples establish the prominence of broad knowledge in the starting model.

In that earlier division of labor, post-training largely taught the model how to surface what it already knew: follow a chat template, answer questions, and behave usefully in conversation. Reinforcement learning shaped the interaction more than it supplied new capabilities. Pre-training therefore consumed the bulk of the compute budget and largely determined the ceiling of the resulting assistant.

### RL becomes a source of capability

[3:56](https://www.youtube.com/watch?v=xbPriQWXtWM&t=236s)

The arrival of **OpenAI o1** in 2024 and **DeepSeek-R1** in January 2025 changed that allocation of responsibility. o1 demonstrated the potential of reasoning models; R1 made a route to building them more openly accessible. Reinforcement learning could now substantially improve task performance, rather than merely adjust conversational behavior. Singh uses o1’s AIME competitive-mathematics graphs to illustrate the shift.

Claude Code extended the practical setting to a terminal, where developers could ask a model to build applications. As function calling improved, the interaction became more than a question followed by an answer: a model could act within a software environment. Training that interaction end to end with RL lets the model learn how to work in the environment and construct software.

![Slide juxtaposing two performance graphs with a DeepSeek-R1 paper heading and a Claude Code excerpt and partial terminal image.](https://ai.engineer/api/talk-essay-media/29b24dbaa3904b06204b32b0dc8bb6c334b763cea4ffd396e7501a1a50ea620f)

“Everything changed”: reasoning-performance graphs, DeepSeek-R1, and Claude Code.

**If RL is responsible for learning increasingly consequential behavior, pre-training should prepare the model to learn that behavior well.** The question is no longer just how much knowledge a base model contains. It is whether a conventional web-trained model provides the best starting point for large-scale reasoning and agent training.

### Two routes away from a web-heavy recipe

[5:59](https://www.youtube.com/watch?v=xbPriQWXtWM&t=359s)

The emerging recipes do not agree on whether synthetic data is essential. Singh favors it, but presents the direction of the research as unsettled. His first example is [**MAI-Thinking-1**](https://microsoft.ai/pdf/mai-thinking-1.pdf), whose recipe he describes as excluding synthetic data and outputs from other language models, including filtering web scrapes for such material. This retains the idea of using human-produced knowledge to bootstrap representations.

Yet retaining human-produced data does not mean retaining the old mixture. Singh puts MAI-Thinking-1’s web-text share at 15%, compared with roughly 85% for GPT-3. His comparison illustrates a change in emphasis, rather than an independently established equivalence between the two recipes’ categories. Web text still matters, but code and STEM receive more attention as downstream uses demand those capabilities.

The alternative is to move data associated with post-training into pre-training itself. In the [**Nemotron 3 Ultra** report](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf), Singh highlights three categories prefixed with `SFT` in the recipe chart. These are question-and-answer or conversational examples of the kind normally associated with **supervised fine-tuning**. Putting them into the earlier mix exposes the model to the shape of conversations and expected downstream tasks from the beginning.

The two approaches disagree about data provenance while converging on a more task-oriented recipe:

| Recipe direction | Data choice | Intended preparation |
| --- | --- | --- |
| MAI-Thinking-1, as described by Singh | Human-produced data; less web text | Greater code and STEM emphasis |
| Nemotron 3 Ultra, as interpreted by Singh | Synthetic and SFT-style examples earlier | Familiarity with downstream conversations and tasks |

Books and general web text occupy less of the mix. Singh contrasts GPT-3’s lack of a dedicated code dataset with the prominence of code in newer recipes. **The shift in what a base model is for does not depend on every lab choosing synthetic data.**

### Rephrasing changes the exposure, not just the token count

[9:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=581s)

Synthetic data raises a legitimate concern: indiscriminately feeding generated text back into training can degrade performance or contribute to model collapse. Singh’s concrete counterexample is **Trinity Large**, where he reports using synthetic data at web scale, largely through **rephrasing**. Start with a seed item and generate multiple expressions of the same information. Adding those variants to the mix increases exposure to the information without requiring the model to encounter exactly the same wording each time.

At the record level, the operation can be represented by keeping a seed identifier attached to every variant. In this illustrative Python example, the candidate additions express the same fact in different forms:

```python
seed = {
    "id": "dict-keys",
    "text": "Python dictionary keys must be hashable.",
}
rephrases = [
    "A Python dictionary requires hashable keys.",
    "To use an object as a dictionary key in Python, it must be hashable.",
]

candidate_records = [
    {"seed_id": seed["id"], "text": text}
    for text in [seed["text"], *rephrases]
]
```

This constructs candidate training records; the operation itself does not establish that generated variants preserve the seed’s meaning. The mechanism Singh describes is repeated exposure to the same information through different expressions.

Singh cites **Kimi K2** as a larger-scale example of rephrasing applied broadly across a pre-training dataset. He also points to SwallowCode and SwallowMath, introduced in [*Rewriting Pre-Training Data Boosts LLM Performance in Math and Code*](https://arxiv.org/abs/2505.02881), as earlier examples of the approach.

![Slide titled “The advent of synthetic data for pretraining,” containing highlighted Trinity Large text, research excerpts, and a colored rephrasing flowchart.](https://ai.engineer/api/talk-essay-media/f1d5edd3a16a34bd2ebf4b2ac5835bab481be82c71f13a9047504f6c35c6e29d)

Synthetic pretraining data: research excerpts and a chunk-rephrasing diagram.

The intended benefit is therefore broader than acquiring more tokens. Rewriting can clean up material, improve its usefulness, and present it in instruction or agent-oriented forms. Those forms let the model begin learning downstream task representations during pre-training, rather than first encountering them after the base model is finished.

### Earlier exposure can reduce later distribution shifts

[11:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=680s)

For **mixture-of-experts models**, moving downstream data earlier also has a routing motivation. Experts specialize over training, while load-balancing objectives encourage broadly equal utilization across a batch or a sequence, depending on the objective. A model trained on one distribution can route a very different distribution unevenly when post-training begins.

Singh cites MAI-Thinking-1 as an example of a distribution shift producing large expert imbalances, and describes the response as increasing the balancing coefficient during SFT. The report is more specific: it gives a load-balancing coefficient of `1e-2` during self-distillation and `1e-5` during RL. Singh’s preferred direction is to expose the model to relevant data early enough that it learns stable representations, reducing the need to force a major adjustment to expert balance late in training.

**Mid-training** addresses a related mismatch. It exposes the model to distributions it will encounter during post-training and RL, often at longer context lengths. Longer contexts admit agentic traces into the training mix, preparing the model for interactions that extend beyond a short question and answer. As pre-training itself adopts longer contexts, Singh argues that these datasets can also move earlier, allowing the corresponding representations to develop from the start.

### Separate the learning objectives from the stage names

[13:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=800s)

Once conversational data, agentic traces, and longer contexts can appear at multiple stages, the labels pre-training, mid-training, and post-training become less informative. Singh proposes a simpler distinction between two learning paradigms:

- **Supervised next-token learning:** learn from supplied sequences and their continuations.
- **Reinforcement learning:** improve behavior through the training environment and its feedback.

The important relationship is how the first prepares the model for the second, wherever a dataset happens to enter the schedule.

Compute allocation makes that relationship consequential. Citing an interview with the head of Xiaomi’s MiMo Labs, Singh reports roughly equal pre-training and post-training compute allocations for the final model. He presents [**Composer 2.5**](https://cursor.com/blog/composer-2-5) as a more extreme case, claiming that its RL compute exceeded its supervised-learning compute. [Cursor’s public description](https://prod.cursor.com/en-US/composer) groups additional training and RL together, so it does not isolate the RL-only comparison.

As RL takes a larger role in the budget, supervised learning acquires a different optimization target: **build representations that make subsequent RL effective**. The base model still matters, but its value increasingly includes what it enables the next learning process to discover.

### Give RL skills it can compose

[14:44](https://www.youtube.com/watch?v=xbPriQWXtWM&t=884s)

What must the supervised phase supply? Singh highlights research suggesting that the base model needs exposure to **atomic skills** that RL can later compose. With an environment of sufficient difficulty, RL can learn to extrapolate from those ingredients. This makes the training distribution important at two levels: the model needs useful component skills, and the RL environment needs to demand something beyond merely reproducing them.

The familiar **AlphaGo** comparison illustrates the possibility of reinforcement learning eventually overtaking supervised learning. But Singh leaves its applicability to language models unresolved. Human language is an exceptionally broad distribution to acquire through RL alone. Less supervised learning and more RL remain a possible direction, not a demonstrated endpoint for LLM training. Even without reaching that endpoint, thinking of a base model as a collection of skills available for RL becomes increasingly useful.

### Prepare the model for the behavior you want next

[15:46](https://www.youtube.com/watch?v=xbPriQWXtWM&t=946s)

Some of the useful ingredients may have forms that are uncommon in ordinary human writing. Reasoning traces are one example: their structure differs from much of the text that a broad web scrape would supply. Supervised learning can introduce these unfamiliar forms before RL asks the model to use them effectively.

The same idea extends to **test-time compute** schemes. Singh describes warming a model up to such schemes during SFT, or even during pre-training, using excerpts he attributes to a Zyphra report. The underlying purpose is to choose data that prepares the model to explore effectively during RL. Supervised data can teach not only information, but also forms of behavior that later training will develop.

![Slide titled ““Novel” data during supervised learning” with two paper excerpts and blue highlighting in the test-time compute passage.](https://ai.engineer/api/talk-essay-media/0b6204cb0d28b8d6153121587e9ec5bfc866186c89538f476f76eaac098ba39a)

Reasoning-aware training and test-time compute in supervised-learning data.

That changes the meaning of the base model: from a general prior over human knowledge toward a prior for reasoning and agentic behavior. Singh acknowledges that this framing reflects the current prominence of reasoners and agents. If a different way of interacting with models becomes important, its demands should shape the prior instead. The durable design question is what the model’s eventual use requires it to be ready to learn—not how closely its training corpus resembles a massive scrape of the web.

## Resources

- [Language Models are Few-Shot Learners](https://arxiv.org/html/2005.14165v4) — GPT-3's original paper explains its training mixture, sampling weights and evaluation of in-context learning.
- [MAI-Thinking-1: Building a Hill-Climbing Machine](https://microsoft.ai/pdf/mai-thinking-1.pdf) — Microsoft's technical report describes model development, evaluation, reinforcement learning and expert load balancing.
- [Nemotron 3 Ultra Technical Report](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf) — NVIDIA's report documents the architecture and training pipeline of Nemotron 3 Ultra.
- [Rewriting Pre-Training Data Boosts LLM Performance in Math and Code](https://arxiv.org/abs/2505.02881) — Introduces SwallowCode and SwallowMath and evaluates rewritten data for continual pre-training in code and mathematics.
- [ZAYA1-8B Technical Report](https://arxiv.org/html/2605.05365v1) — Zyphra's report on a reasoning-focused mixture-of-experts model with 700M active and 8B total parameters.

## Slides

### The Base Model is Dead

Varun Singh introduces the idea of a base model as a reflection of the human internet, built through large-scale web-text training.

[0:12](https://www.youtube.com/watch?v=xbPriQWXtWM&t=12s) [Title slide for Varun Singh’s talk, “The Base Model is Dead.”](https://assets.ai.engineer/sites/aiecode2025/sha256/b3fd6261fe83fe0de180d2576e5bf8930c6d883e69b746db65418977b27bedd3.webp)

The Base Model is Dead
(long live the base model)
Varun Singh, Pre-training Lead
Arcee AI

### What is a base model?

Arcee’s simplified pipeline moves through pre-training, context extension and mid-training before SFT and RL. The GLM-5 diagram groups pre-training and mid-training under the base model, followed by post-training.

[0:40](https://www.youtube.com/watch?v=xbPriQWXtWM&t=40s) [A simplified Arcee training pipeline alongside GLM pre-training, mid-training and post-training diagrams.](https://assets.ai.engineer/sites/aiecode2025/sha256/719ba4403d09914e5e372c898995fed322b8a0a11d5a28b26f13f41629f92616.webp)

What is a base model?

Training pipeline for Arcee Trinity Large Thinking:
pre-training → pre-training phase 2 → pre-training phase 3 → context extension → mid-training → SFT → RL

Pre-training
General Pre-training Corpus (15T) → Code & Reasoning Continual Pre-training Corpus (7T)
4K → 4K

Mid-training
Repo-Level Code Data (500B) → Synthetic Reasoning Data (500B) → Long Context & Agent Data (100B)
32K → 32K → 128K

[illegible]ure 3: Pre-training and mid-training stages for GLM-4.5. We adapt a multi-stage training recipe
extend the sequence length from 4K to 128K.

Base Model
Pre-training
General Pre-training Corpus (18T) → Code & Reasoning Corpus (9T)
4K → 4K

Mid-training
Long Code & Reasoning Data (1T) → Long Context & Agent Data (500B / 50B)
32K → 128K / 200K

Sparse Attention Adaption (20B, 200K)

Post-Training
Base Model
Overall SFT
Reasoning RL
Agentic RL
General RL
On-Policy Cross-Stage Distillation
logits
logits
logits weights
GLM-5

Figure 5: Overall training pipeline of GLM-5.

### Next-token prediction for useful priors

Pre-training uses next-token prediction to build patterns that support in-context learning. The diagram contrasts learning across sequences via SGD with adapting to repeated examples within a sequence.

[1:18](https://www.youtube.com/watch?v=xbPriQWXtWM&t=78s) [A decoder transformer diagram and an illustration of learning across sequences during pre-training.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1ab6dbdb82c346b2c2a0ae37619e944cae5e011e33e9742ffb6fa3fa945dbaa.webp)

Next-token prediction for useful priors

Cross-Entropy Loss
Labels
Predictions
Output Probabilities
Softmax
Linear
RMS Norm
8× decoder layer
SwiGLU Feed Forward
RMS Norm
GQA self-attention
V K Q
M
RMS Norm
Embedding
Input Sequence
RoPE
Causal mask

outer loop
Learning via SGD during unsupervised pre-training
inner loop

sequence #1
In-context learning
5 + 8 = 13
7 + 2 = 9
1 + 0 = 1
3 + 4 = 7
5 + 9 = 14
9 + 8 = 17

sequence #2
In-context learning
gaot => goat
sakne => snake
brid => bird
fsih => fish
dcuk => duck
cmihp => chimp

sequence #3
In-context learning
thanks => merci
hello => bonjour
mint => menthe
wall => mur
otter => loutre
bread => pain

Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded within a single sequence.

### What were older base models trained on?

Filtered Common Crawl made up 60% of GPT-3’s training mix. The Llama 3 excerpt describes roughly 50% general knowledge, 25% math and reasoning, 17% code, and 8% multilingual tokens.

[1:55](https://www.youtube.com/watch?v=xbPriQWXtWM&t=115s) [A GPT-3 dataset table and a Llama 3 data mix summary.](https://assets.ai.engineer/sites/aiecode2025/sha256/77e421a089264a28ee2b23aab49fa5fd36e08ae676b0fd87918dfa77a48624ba.webp)

What were older base models trained on?

Dataset | Quantity (tokens) | Weight in training mix | Epochs elapsed when training for 300B tokens
Common Crawl (filtered) | 410 billion | 60% | 0.44
WebText2 | 19 billion | 22% | 2.9
Books1 | 12 billion | 8% | 1.9
Books2 | 55 billion | 8% | 0.43
Wikipedia | 3 billion | 3% | 3.4

Table 2.2: Datasets used to train GPT-3. “Weight in training mix” refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets are seen less than once.

that model on several key benchmarks. (Llama 3)

Data mix summary. Our final data mix contains roughly 50% of tokens corresponding to general knowledge, 25% of mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens.

### Everything changed

o1’s AIME accuracy rises with both train-time and test-time compute. The slide pairs those plots with DeepSeek-R1 and Claude Code as examples of the shift toward reasoning and agentic coding.

[4:00](https://www.youtube.com/watch?v=xbPriQWXtWM&t=240s) [Plots of o1 accuracy alongside excerpts about DeepSeek-R1 and Claude Code.](https://assets.ai.engineer/sites/aiecode2025/sha256/6dd582f2b564af2c0b279b6c2c2448ad93facc613bea1f613e195c018593b79a.webp)

Everything changed

Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process. We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). The constraints on scaling this approach differ substantially from those of LLM pretraining, and we are continuing to investigate them.

o1 AIME accuracy during training
Y-axis: pass@1 accuracy
Y-axis ticks: 0, 20, 40, 60, 80, 100
X-axis: train-time compute (log scale)

o1 AIME accuracy at test time
Y-axis: pass@1 accuracy
Y-axis ticks: 0, 20, 40, 60, 80, 100
X-axis: test-time compute (log scale)

o1 performance smoothly improves with both train-time and test-time compute

deepseek

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI
research@deepseek.com

Abstract

General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) (Brown et al., 2020; OpenAI, 2023) and chain-of-thought prompting (Wei et al., 2022b), have achieved con-

Claude 3.7 Sonnet shows particularly strong improvements in coding and front-end web development. Along with the model, we’re also introducing a command line tool for agentic coding, Claude Code. Claude Code is available as a limited research preview, and enables developers to delegate substantial engineering tasks to Claude directly from their terminal.

Welcome to the Claude Code research preview!

### Modern base model data recipes

MAI-Base-1 allocates 54.6% of training tokens to code and 14.9% to web text. Nemotron includes SFT and synthetic data in pretraining, shifting its mixture from diversity toward quality.

[5:26](https://www.youtube.com/watch?v=xbPriQWXtWM&t=326s) [MAI-Base-1 data composition table and Nemotron pretraining mixture charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/5edf136c9f7bcfc3368f5cfa6d3230ddc5d9e7ba1398b097e4a1dbc31bf801f1.webp)

Modern base model data recipes

Source family | Unique tokens (T) | Training tokens (T) | Mix Percentage (%) | Avg. Epochs
Code | 7.4 | 16.4 | 54.6 | 2.22×
STEM | 2.2 | 4.7 | 15.8 | 2.17×
Math | 0.3 | 1.6 | 5.4 | 5.28×
Books and journals | 0.6 | 0.9 | 3.1 | 1.65×
PDFs | 2.7 | 1.4 | 4.7 | 0.53×
Web text | 8.1 | 4.5 | 14.9 | 0.55×
Multilingual (other) | 8.1 | 0.5 | 1.6 | 0.06×
Total | 29.2 | 30.0 | 100.0 | 1.03×

Table 5. MAI-Base-1 pre-training data composition. Unique tokens is the deduplicated token count per source family. Training tokens is the number of tokens consumed from that family over the full run. Avg. Epochs is the ratio of training tokens to unique tokens; values above 1× indicate repeated sampling.

Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

(a) Data mixture of phase 1
multilingual 5.0%
math 6.4%
nemotron-cc-code 2.1%
code 14.0%
academic 1.6%
wiki 0.6%
crawl++ 1.4%
finepdfs-unfiltered 4.9%
syn-crawl-high 22.4%
sft-general 1.6%
sft-stem 9.2%
sft-code 3.9%
crawl-medium 4.3%
crawl-medium-high 5.7%
crawl-high 6.5%
syn-crawl-medium 10.3%

(b) Data mixture of phase 2
multilingual 5.0%
math 6.4%
nemotron-cc-code 2.1%
code 14.0%
legal 0.1%
academic 1.6%
wiki 0.6%
crawl++ 1.4%
finepdfs-high 10.7%
finepdfs-medium 2.5%
sft-general 4.0%
sft-stem 9.2%
sft-code 3.9%
crawl-medium-high 5.7%
crawl-high 6.5%
syn-crawl-medium 3.8%
syn-crawl-high 22.4%

Figure 4 | The data mixtures for both pretraining phases. We design the phase 1 data mixture to have a bias for diversity and the phase 2 data mixture to have a bias for quality.

### GPT-3 and MAI-Base-1 data mixtures

GPT-3’s mix allocated 60% to filtered Common Crawl, while MAI-Base-1 allocated 54.6% to code. The GPT-3 table lists no separate code dataset.

[9:13](https://www.youtube.com/watch?v=xbPriQWXtWM&t=553s) [Tables compare GPT-3 datasets with MAI-Base-1 source families and training weights.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c2fc712c0e8d3a894a48b1c2f6989ab5096f818451426f670156d7791789487.webp)

GPT-3

Dataset | Quantity (tokens) | Weight in training mix | Epochs elapsed when training for 300B tokens
Common Crawl (filtered) | 410 billion | 60% | 0.44
WebText2 | 19 billion | 22% | 2.9
Books1 | 12 billion | 8% | 1.9
Books2 | 55 billion | 8% | 0.43
Wikipedia | 3 billion | 3% | 3.4

Table 2.2: Datasets used to train GPT-3. “Weight in training mix” refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets are seen less than once.

MAI-Base-1

Source family | Unique tokens (T) | Training tokens (T) | Mix Percentage (%) | Avg. Epochs
Code | 7.4 | 16.4 | 54.6 | 2.22×
STEM | 2.2 | 4.7 | 15.8 | 2.17×
Math | 0.3 | 1.6 | 5.4 | 5.28×
Books and journals | 0.6 | 0.9 | 3.1 | 1.65×
PDFs | 2.7 | 1.4 | 4.7 | 0.53×
Web text | 8.1 | 4.5 | 14.9 | 0.55×
Multilingual (other) | 8.1 | 0.5 | 1.6 | 0.06×
Total | 29.2 | 30.0 | 100.0 | 1.03×

Table 5. MAI-Base-1 pre-training data composition. Unique tokens is the deduplicated token count per source family. Training tokens is the number of tokens consumed from that family over the full run. Avg. Epochs is the ratio of training tokens to unique tokens; values above 1× indicate repeated sampling.

### Adding post-training data early in pretraining

Domain-specific RL data is narrower than pre- and mid-training mixtures, risking imbalanced MoE expert selection. The excerpt uses stronger load balancing in self-distillation than in RL; the speaker favors earlier exposure.

[11:22](https://www.youtube.com/watch?v=xbPriQWXtWM&t=682s) [Two Nemotron pretraining mixture charts appear above an excerpt on self-distillation and reinforcement learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0ab9a8673bda57361f8800379a0d4d87ba2bf3c74d7dec410061cab6f2ba938.webp)

It’s beneficial to add post-training data early in pretraining

Nemotron 3 Ultra : Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

(a) Data mixture of phase 1
multilingual 5.0%
math 6.4%
nemotron-cc-code 2.1%
code 14.0%
academic 1.6%
wiki 0.6%
crawl++ 1.4%
finepdfs-unfiltered 4.9%
syn-crawl-high 22.4%
sft-general 1.6%
sft-stem 9.2%
sft-code 3.9%
crawl-medium 4.3%
crawl-medium-high 5.7%
crawl-high 6.5%
syn-crawl-medium 10.3%

(b) Data mixture of phase 2
multilingual 5.0%
math 6.4%
nemotron-cc-code 2.1%
code 14.0%
legal 0.1%
academic 1.6%
wiki 0.6%
crawl++ 1.4%
finepdfs-high 10.7%
finepdfs-medium 2.5%
sft-general 4.0%
sft-stem 9.2%
sft-code 3.9%
crawl-medium-high 5.7%
crawl-high 6.5%
syn-crawl-medium 3.5%
syn-crawl-high 22.4%

Figure 4 | The data mixtures for both pretraining phases. We design the phase 1 data mixture to have a bias for diversity and the phase 2 data mixture to have a bias for quality.

We carefully tune two hyperparameters during self-distillation: dropout (Srivastava et al., 2014) and the MoE load balancing loss coefficient. We use a relatively high dropout rate of 0.15, which increases entropy and helps prevent model collapse, improving performance in subsequent RL stages. As the data distribution in domain-specific RL is substantially narrower than the mixtures used during pre- and mid-training, expert selection during RL can become highly imbalanced. However, using a large load balancing coefficient during RL also harms stable performance improvement. To address this, we use a comparatively large load balancing coefficient 1 · 10⁻² during self-distillation and a much smaller coefficient 1 · 10⁻⁵ during RL. Since the self-distillation contexts are generated from the RL run itself and therefore follow a similar distribution, the expert balancing effect induced during self-distillation is preserved during RL.

### Mid-training and long-context pretraining

Mid-training exposes models to longer contexts and data resembling post-training. GLM-5’s pipeline adds long code and agent data, while MAI-Base-1 progressively extends context length.

[12:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=761s) [A GLM-5 training pipeline appears alongside a table of MAI-Base-1 training phases and context lengths.](https://assets.ai.engineer/sites/aiecode2025/sha256/002b0aa47dff915461e6668e5b5fb9f2d1467f2e40be3c60a17adee52658177c.webp)

Mid-training, long context pre-training - slowly unifying?

Base Model

Pre-training
General Pre-training Corpus (18T) → Code & Reasoning Corpus (9T)
4K → 4K

Mid-training
Long Code & Reasoning Data (1T) → Long Context & Agent Data (500B / 50B)
32K → 128K / 200K

Sparse Attention Adaption (20B, 200K)

Post-Training
Base Model
Overall SFT
Reasoning RL
Agentic RL
General RL
On-Policy Cross-Stage Distillation
logits
logits
logits + weights
GLM-5

Figure 5: Overall training pipeline of GLM-5.

Phase | Tokens | Context length | GB200 GPUs
Pre-training | 30 T | 16,384 | 8,192
Mid-training 1 | 3.4 T | 65,536 | 8,192
Mid-training 2 | 150 B | 262,144 | 4,096

Table 6. Training specifications across phases for MAI-Base-1.

5.3. Enabling long context

Instead of training with 128K sequences from scratch, we pre-train our models with 32K se-

### Two stages of LLM training

Supervised learning builds representations for RL. Composer 2.5 illustrates the shift in compute: 85% goes to additional training and RL.

[13:18](https://www.youtube.com/watch?v=xbPriQWXtWM&t=798s) [A supervised learning box points to an RL box, with a compute chart and a compute split quotation.](https://assets.ai.engineer/sites/aiecode2025/sha256/828fb8549e9d12177256301abd9ad19f5d221cc63f31f4d5cd8fab6c78cfb959.webp)

2 stages of LLM training - supervised learning builds priors

supervised learning → RL

85% of the compute for Composer 2.5 comes from additional training and RL
Kimi k2: 7.5%
Kimi k2.5: 7.5%
Composer training and RL: 85%

1) Compute split 3:1:1

“A very reasonable GPU ratio, in my mind, is 3:1:1. Research should be at least as large as the GPUs you formally allocate to training — you want more GPUs leaking outward into research.”

### Novel data during supervised learning

Zyphra trains on long reasoning traces and reshuffles expert rollouts into aggregation examples, preparing the model for Markovian RSA at test time.

[15:44](https://www.youtube.com/watch?v=xbPriQWXtWM&t=944s) [Two paper excerpts discuss reasoning-aware training and integrating Markovian RSA into training.](https://assets.ai.engineer/sites/aiecode2025/sha256/d426d5f4db25cd00965cd2fccf91106a52b9359320bab7a557ef06f38540afd6.webp)

“Novel” data during supervised learning

Reasoning-aware training across stages: We designed ZAYA1-8B from scratch for reasoning. Motivated by evidence that including reasoning data during pretraining can produce gains that post-training alone does not recover (Akter et al., 2025), we include long chain-of-thought (CoT) data in all pretraining phases and during midtraining. To train on reasoning traces that exceed the pretraining context length, we introduce a novel answer-preserving trimming methodology, which truncates the tail of the reasoning trace while preserving the final answer, or drops the example if the answer alone does not fit. Unlike prior length-control methods that operate during inference or RL rollout generation (Khatri et al., 2025; Yang et al., 2025), AP-trimming is applied during training-data construction.

Test-time compute methods: We introduce Markovian RSA, a novel test-time compute method that combines the recursive candidate-aggregation structure of RSA (Venkatraman et al., 2025) with the bounded-workspace principle of Markovian Thinking (Aghajohari et al., 2025). Markovian RSA turns long reasoning into staged batched inference: each stage generates N candidates in parallel, each candidate has bounded decode length β, and aggregation prefill depends only on C carried-forward tails of length τ, not on the full reasoning history. Crucially, we also integrate Markovian RSA into training: SFT data is constructed by reshuffling expert-model rollouts into aggregation examples, and RL stages train both expert-model and policy-self-aggregation variants. The resulting model is trained for the Markovian RSA workflow

### Base model priors

Base models have shifted from general world priors toward reasoning and agentic behavior priors. The speaker calls this framing reductive: future ways of interacting with models could call for different priors.

[16:35](https://www.youtube.com/watch?v=xbPriQWXtWM&t=995s) [Base models have moved from general world priors to reasoning and agentic behavior priors.](https://assets.ai.engineer/sites/aiecode2025/sha256/d5c16febb74ff435598a5880ae7e55bf1e4c4e8303e35b444a38b02a5ed7b493.webp)

Base models have moved from general world priors to reasoning and agentic behavior priors

### AI Engineer World's Fair

[17:36](https://www.youtube.com/watch?v=xbPriQWXtWM&t=1056s) [AI Engineer World's Fair title on a black background, with the AI Engineer website address.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff8e7caea1c5ecc1a13f9198098f20f2163dc6a7a9f46dbe322f0432d29d340a.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=xbPriQWXtWM&t=0s) [upbeat music] Hi everyone. My name is Varun. I'm the pre-training lead at Arcee AI, and the talk I'm gonna be giving today is, uh, called The Base Model is Dead.

[0:21](https://www.youtube.com/watch?v=xbPriQWXtWM&t=21s) Uh, but not really. Um, the idea of the base model that we have, um, kinda is like built on, uh, this idea of like training on super large-scale web text, and the base model kind of being a reflection of like the whole knowledge of like the, uh, human internet.

[0:42](https://www.youtube.com/watch?v=xbPriQWXtWM&t=42s) Um, you can see in like, uh, these... I've, I've taken these, uh, from a bunch of, uh, different papers on like the entire LLM training process. Um, our own model, Arcee lo- uh, Trinity Large Thinking, um, the process looked kinda like the simplified diagram on the left.

[1:00](https://www.youtube.com/watch?v=xbPriQWXtWM&t=60s) I've taken the top one from, uh, GLM 4.5, the bottom one from GLM 5.

[1:05](https://www.youtube.com/watch?v=xbPriQWXtWM&t=65s) Um, all these have a pre-training phase, and, uh,

[1:11](https://www.youtube.com/watch?v=xbPriQWXtWM&t=71s) pre-training is like the stage where the model loo- uh, accumulates world knowledge, builds useful representations, uh, all through next-token prediction on, um, web text. Um, I've got a simplified, uh, transformer diagram, um, decoder-only transformer, and, um, a screenshot from the GPT-3 paper that talks about

[1:35](https://www.youtube.com/watch?v=xbPriQWXtWM&t=95s) how, um, language models can, uh, learn how to do in-context learning through, um, unsupervised or self-supervised or, um, some- some even just call it supervised, um, learning on, uh, through next-token prediction.

[1:56](https://www.youtube.com/watch?v=xbPriQWXtWM&t=116s) Um, the way that, uh, older base models were trained was, like I said, mostly on, uh, things that reflected the entirety of human knowledge. Uh, so Common Crawl, uh, which is like a commonly available web scrape, uh, made up most of the training dataset for GPT-3.

[2:16](https://www.youtube.com/watch?v=xbPriQWXtWM&t=136s) Um, WebText2, another web scrape dataset. Um, some sources from like, uh, books as well, um, and Wikipedia as like a high-quality, uh, representation of human knowledge. Um, you can see that, uh, web text alone here, including like Wikipedia, makes up like roughly eighty-five percent of the whole training mix.

[2:38](https://www.youtube.com/watch?v=xbPriQWXtWM&t=158s) Um, looking at, uh, the bottom with Llama 3, um, web text still kind of makes up a majority of, of the, um, model's training data with like fifty percent of the tokens corresponding to general knowledge.

[2:56](https://www.youtube.com/watch?v=xbPriQWXtWM&t=176s) Back then, post-training was mostly shaping the model to use, um, the parts, it to like surface, uh, the knowledge that it accumulates in-- accumulated in pre-training, uh, in like a chat interface.

[3:11](https://www.youtube.com/watch?v=xbPriQWXtWM&t=191s) So mostly allowing the model to adapt to a chat template, to the question answer format, um, and be us- be useful in an interaction that way. Um, RL was mostly just a cherry on top, um, shaping the, you know, flavor of the interactions more than conferring extra, um, knowledge or quality onto the base model itself.

[3:36](https://www.youtube.com/watch?v=xbPriQWXtWM&t=216s) Now, in, in like this realm of, of, of how language models used to be, pre-training, um, and the base model kind of defined how good you were able to get a m- a model.

[3:50](https://www.youtube.com/watch?v=xbPriQWXtWM&t=230s) Um, it was like the bulk of the compute, uh, budget, and it was,

[3:56](https://www.youtube.com/watch?v=xbPriQWXtWM&t=236s) um, the, the like core of the, uh, uh, training process. Um, however, uh, this kind of changed l- a lot last year when, um, Open- OpenAI-- I guess 2024 actually, OpenAI released, uh, o1, um, pioneering reasoning models, and DeepSeek, uh, also released R1 in January

[4:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=260s) 2025, um, allowing the whole world to know how to build these types of language models. And

[4:30](https://www.youtube.com/watch?v=xbPriQWXtWM&t=270s) now we have this new, uh, new, um, use for reinforcement learning, which is no longer a cherry on top, but it can dramatically improve the performance of, of the model on various different tasks.

[4:45](https://www.youtube.com/watch?v=xbPriQWXtWM&t=285s) Um, the, the famous graphs from o1 there talking about, uh, AIME performance, um, competitive math contest.

[4:54](https://www.youtube.com/watch?v=xbPriQWXtWM&t=294s) Um, and then even later in the year, we, we saw Claude Code, um, come into being as a way for developers to easily, um, kinda use a language models in a, in a terminal to build out applications as models got stronger and stronger on things like function calling.

[5:13](https://www.youtube.com/watch?v=xbPriQWXtWM&t=313s) Um, and then people realized you could, uh, you know, RL this end to end. Um, and now models could learn how to interact with software environments and build software and, uh, perform really useful work.

[5:28](https://www.youtube.com/watch?v=xbPriQWXtWM&t=328s) And so, um, the question then becomes like,

[5:33](https://www.youtube.com/watch?v=xbPriQWXtWM&t=333s) is your standard base model still, uh, what the best, uh, what would be the best, um,

[5:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=341s) prior for the, for the, this large-scale reinforcement learning phase that, uh, reasoners and, um, agentic models now use? And we can kinda see like in the, in, uh, a few open research papers what the trend is, where the trend is going.

[5:59](https://www.youtube.com/watch?v=xbPriQWXtWM&t=359s) Um, and interestingly enough, the, um, it seems like-

[6:05](https://www.youtube.com/watch?v=xbPriQWXtWM&t=365s) Not super clear yet. Um, I have my opinions on, like, synthetic data being the way forward, but I'm-- I've got, like, two contrasting, uh, perspectives here kind of in the slide.

[6:18](https://www.youtube.com/watch?v=xbPriQWXtWM&t=378s) The top image is from the Mei- uh- Thinking 1 paper, where they make a, make a really large point to not use any synthetic data or any, uh, data from any other language model.

[6:29](https://www.youtube.com/watch?v=xbPriQWXtWM&t=389s) Um, and they really try to, you know, filter their web scrapes for this as well, um, in order f- uh, to kind of adhere to, like, the previous,

[6:38](https://www.youtube.com/watch?v=xbPriQWXtWM&t=398s) um, paradigm of, like, uh, using human knowledge as a way to bootstrap, uh, model representations and, like,

[6:46](https://www.youtube.com/watch?v=xbPriQWXtWM&t=406s) uh, capabilities. Um, but I would say that this is also... Like, even, even though they stuck with no synthetic data, the data mix that they've chosen here is still, um, totally different from what you'd expect in, like, a, um, in a, in a classical, uh, language model.

[7:05](https://www.youtube.com/watch?v=xbPriQWXtWM&t=425s) And I mean, the, the main reason for that is that web text, which used to make, like, up to eighty-five percent of the trained data in GPT, uh, three, is now all the way down at fifteen percent.

[7:17](https://www.youtube.com/watch?v=xbPriQWXtWM&t=437s) And that-- I mean, that just shows that, uh, the value of, like, web text contributing to, like, the downstream, um, performance of, like, the models on RL and stuff is kind of, uh...

[7:33](https://www.youtube.com/watch?v=xbPriQWXtWM&t=453s) It-it's still important, but taking a back seat to things like code and STEM abilities as the models kind of gain more real-world use cases related to, to those.

[7:43](https://www.youtube.com/watch?v=xbPriQWXtWM&t=463s) Um, the other approach, uh, is to bring, um, post-training data and large-scale synthetic data back, uh, through-- pull it back through the process into the pre-training phase. Um, the, the bottom chart I've taken from Nemotron 3 Ultra, um, where they reveal their, um, the data recipe, and I'm not sure how readable it is, but these top

[8:09](https://www.youtube.com/watch?v=xbPriQWXtWM&t=489s) three, um, on the left b- uh, pie chart, the top three on the kind of right, right side of it, uh, they're all labeled SFT, uh, with, with SFT as a prefix.

[8:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=500s) And that's the type of question-and-answer kind of chat data set that you'd s- you'd expect to see only in post-training. But by pulling it back into the process, they're able to, like, get the model to learn, um, the shape of these conversations and what kind of tasks they might be expected to do downstream, um, from the

[8:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=521s) very beginning of the pre-training process. Um, and, uh, this follows, like, um, a similar, um, trend in, like, diminishing, uh, the amount of web text used in the model.

[8:55](https://www.youtube.com/watch?v=xbPriQWXtWM&t=535s) Um, yeah. Um, it's really interesting to see the, the Nemotron series lean so heavily into synthetic data, but, uh, Mei Thinking 1 kind of lean in the opposite direction.

[9:11](https://www.youtube.com/watch?v=xbPriQWXtWM&t=551s) Um, I've, I've, I've just got this slide here as, like, a con- uh, easy contrast of-- uh, people can see on the, like, amount of web text and the amount of books and stuff, um, being less of a percentage here.

[9:26](https://www.youtube.com/watch?v=xbPriQWXtWM&t=566s) And GPT-3 didn't even use to have any specific code data sets, but now code is, like, the dominating, um,

[9:35](https://www.youtube.com/watch?v=xbPriQWXtWM&t=575s) data, data subset that we have in, uh, pre-training recipes.

[9:41](https://www.youtube.com/watch?v=xbPriQWXtWM&t=581s) Um, so I mentioned synthetic data, but what is actually, uh... Like, how is synthetic data used? There's a lot of, uh, talk around synthetic data that, you know, blindly tossing it into a model can cause the model to collapse and, uh,

[9:56](https://www.youtube.com/watch?v=xbPriQWXtWM&t=596s) and performance to tank. But there's been a lot of work and, uh, even, like, a large-scale, you know, example of this, uh, turning out really well. Um, so in our own, in our, in our own, uh, model, Trinity Large, we, uh, had a large amount of, uh, web, uh, web scale synthetic data, um, mostly through rephrasing, where

[10:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=620s) you take a seed data item and you sort of upsample it in the mix by, uh, generating synthetic rephrases of the same information. So the model sees the same information in, like, multiple ways.

[10:32](https://www.youtube.com/watch?v=xbPriQWXtWM&t=632s) Um, the bottom two, uh, screenshots are from Kimi K2, um, an even larger scale model that, uh, broadly use this, um, across the whole pre-training dataset. Um, the top, uh, top right, um, screenshot is from a paper that, uh, resulted in the dataset Swallowcode and Swallowmath, which are early examples of

[10:57](https://www.youtube.com/watch?v=xbPriQWXtWM&t=657s) this. But the trend seems to be that, um, synthetic data not only allows you to get more and more tokens, uh, but also, you know, clean up tokens, get higher quality tokens, and have, um, tokens that are shaped more like instruct or agentic tasks all the way back in pre-training and, uh, allowing the model to, like, learn

[11:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=680s) those task representations from the very beginning. Um, another reason that it's, uh, beneficial to add post-training data early in pre-training is now with MoEs. Um, one of the biggest, uh, pain points in training an MoE is dealing with load balancing, um, where experts can specialize over the course of training and, um, they'll-- uh, and load balancing

[11:45](https://www.youtube.com/watch?v=xbPriQWXtWM&t=705s) objectives aim to, uh, achieve broadly equal utilization of the experts, um, in a given batch or a sequence, depending on the objective. Um, without, uh, post-training data in, uh, early in pre-training, uh, with an MoE, one really easy pitfall, uh, that we can fall into is this-

[12:05](https://www.youtube.com/watch?v=xbPriQWXtWM&t=725s) Is kind of illustrated in the MEI Thinking 1 report, which is that the data distribution that the model sees, uh, in post-training is really, really different, uh, compared to what it sees in pre-training.

[12:19](https://www.youtube.com/watch?v=xbPriQWXtWM&t=739s) And this can cause massive imbalances, and, um, MEI overcame it by, uh, really cranking up the load balancing coefficient during the SFT stages.

[12:31](https://www.youtube.com/watch?v=xbPriQWXtWM&t=751s) Um, but I mean, ideally you don't want to mess with the balance that far into training, and the model should learn stable representations from really early on.

[12:44](https://www.youtube.com/watch?v=xbPriQWXtWM&t=764s) Another, uh, interesting thing that, um, is changing in base models now is that, yeah, there's this whole advent in mid-training, uh, which is exposing the model to the distribution that it would see during post-training and RL and at a longer context, so for things like agentic traces to be allowed into the mix and, uh, to kind of

[13:05](https://www.youtube.com/watch?v=xbPriQWXtWM&t=785s) help prepare the model that way. Um, a lot of models though are training with much longer context in pre-training, and there's no reason that these datasets can't be pulled back into the mix to allow for more stable representations from the very beginning.

[13:20](https://www.youtube.com/watch?v=xbPriQWXtWM&t=800s) Um, I think, uh, a better, a better way to understand the current f- uh, phase of LLM training, uh, isn't so much like pre-training, mid-training, post-training, RL. Uh, it all gets a bit muddy that way.

[13:33](https://www.youtube.com/watch?v=xbPriQWXtWM&t=813s) But the- there's two broad paradigms that are like-- that really help build a LLM today, and that's supervised learning, uh, through next-token prediction, and RL. Um, and RL is becoming more and more important.

[13:47](https://www.youtube.com/watch?v=xbPriQWXtWM&t=827s) Um, the bottom, uh, thing is a screenshot from a interview, um, with, uh, the head of, uh, Xiaomi's Mimo Labs, uh, where she talks about how they allocate compute, uh, between research, pre-training, and post-training.

[14:04](https://www.youtube.com/watch?v=xbPriQWXtWM&t=844s) And pre-training and post-training in the final model have a roughly equal compute allocation. Um, Composer 2.5 takes this to the extreme, where, um, Cursor really, uh, sank much, much more RL compute into the model than the model had ever seen in supervised learning.

[14:21](https://www.youtube.com/watch?v=xbPriQWXtWM&t=861s) But with RL dominating such a massive amount of the compute budget, uh, it makes sense to view supervised learning as a way specifically to prepare the model for, uh, to build useful representations for R- for RL instead of it being the bulk of like, um,

[14:40](https://www.youtube.com/watch?v=xbPriQWXtWM&t=880s) what the model would be used for, like, uh, previously.

[14:44](https://www.youtube.com/watch?v=xbPriQWXtWM&t=884s) Um, there's been some, some work on, uh, how supervised learning affects RL. Um, I really like this one paper where the main takeaways are basically that, uh, the base model needs to have some exposure to like, uh, like the atomic skills that it would need to compose during RL, and, um, the model can learn to extrapolate from

[15:06](https://www.youtube.com/watch?v=xbPriQWXtWM&t=906s) there during RL given like the environment has a sufficient level of difficulty.

[15:13](https://www.youtube.com/watch?v=xbPriQWXtWM&t=913s) Um, I had to put in the classic r- AlphaGo graph there, where RL eventually overtakes supervised learning. Uh, it's unclear if we'll see something like that for language models because, of course, you know, uh, human language is such an insane distribution to have to like learn through reinforcement learning alone.

[15:31](https://www.youtube.com/watch?v=xbPriQWXtWM&t=931s) Um, but it's, it's definitely possible that we might see diminished supervised learning and more and more RL, uh, which makes this kind of thinking of a base model as, um, atomic skills for RL more and more valuable.

[15:46](https://www.youtube.com/watch?v=xbPriQWXtWM&t=946s) Another thing that some labs are doing, um, is kind of introducing novel data, uh, during supervised learning. And by novel I mean something that the model really wouldn't have seen the shape of before.

[15:57](https://www.youtube.com/watch?v=xbPriQWXtWM&t=957s) Uh, easy example is, you know, reasoning traces. They don't really look like a, like a ton of what, um, humans output. And, um, another interesting thing is like training for test time compute, um, schemes, um, by kind of warming the model up to them during SFT, um, or even pre-training itself.

[16:19](https://www.youtube.com/watch?v=xbPriQWXtWM&t=979s) Um, this, these screenshots were taken from, uh, Zyphra's ZyA 1.0 paper.

[16:24](https://www.youtube.com/watch?v=xbPriQWXtWM&t=984s) Um, and I think that they're, uh, very interesting ways of thinking about how data can affect, um,

[16:32](https://www.youtube.com/watch?v=xbPriQWXtWM&t=992s) like, uh, the skills needed to explore well in RL.

[16:37](https://www.youtube.com/watch?v=xbPriQWXtWM&t=997s) Um, in conclusion, um, base models have kind of moved from general, uh, human knowledge and world priors to reasoning and agentic behavior priors. Um, of course, that's kind of, uh, reductive in the, in a, in a way that reasoning, reasoners and agents are like the main way we see, uh, bots, uh, the main way we see these

[16:59](https://www.youtube.com/watch?v=xbPriQWXtWM&t=1019s) chatbots used now. But if a new paradigm were to take off, or like a new way of interacting with the models, um, it makes sense to like think of a base model as building a prior for that instead of, um,

[17:13](https://www.youtube.com/watch?v=xbPriQWXtWM&t=1033s) just building off like a, a massive, uh, scrape of web text. And yeah. Thanks for, thanks for listening. Thanks for your time.

[17:24](https://www.youtube.com/watch?v=xbPriQWXtWM&t=1044s) And yeah. [clapping] [outro jingle]
