# Scaling up Continual Learning

Ronak Malde

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/zL1kLftVTlo-scaling-up-continual-learning) · [Video](https://www.youtube.com/watch?v=zL1kLftVTlo)

## Scaling up Continual Learning

Learning from production requires more than collecting traces. On-policy self-distillation supplies dense feedback, but long tool-calling trajectories expose divergence, hint leakage, and deployment challenges.

Familiarity with language-model token probabilities, fine-tuning, and tool-calling agents will help; the article introduces the training methods as they appear.

### Why build harder benchmarks while production keeps generating experience?

[0:16](https://www.youtube.com/watch?v=zL1kLftVTlo&t=16s)

How can models learn from the work people already ask them to do? Following a talk about measuring continual learning, Ronak Malde turns to scaling it. His starting point is experience growing Windsurf’s research team and training **SWE-1**, before founding Trajectory. He describes giving up acquisition-related compensation to make that move; the Google transaction he references was a hiring and licensing deal, rather than a purchase of Windsurf itself.

The existing path to progress has been to build benchmarks, train against them, and replace them as they saturate. Malde describes saturation accelerating from years to months, while the remaining tasks become increasingly expensive: some take four, six, or twenty-four hours, and others take days. More investment in reinforcement-learning environments and data can sustain that process, but it does not ensure that the tasks resemble what users actually need.

![Slide titled “What are we left with?” lists three benchmark limitations beside four charts.](https://ai.engineer/api/talk-essay-media/c1019d3bf3008a0173778d0915392772645637e5aab9c0e12a3b9de9bb79f2ff)

Benchmarks are becoming more time-consuming, expensive, and disconnected from real-world use cases.

Meanwhile, production inference generates examples of models succeeding, failing, and responding to the consequences of their actions. Malde estimates that inference consumes hundreds of trillions of tokens daily, without specifying a measurement source or accounting scope. The central opportunity does not depend on that exact total: **real interactions contain training signal that is currently being left unused**.

Continual learning would make those interactions part of the improvement process. Malde places this after internet-scale pretraining and benchmark-driven training, and points to growing interest from Ilya Sutskever, Andrej Karpathy, Sholto Douglas, Demis Hassabis, and Satya Nadella, alongside a recent Dwarkesh video. The technical question is how to turn that interest into a learning algorithm that fits actual usage.

### Four requirements that existing methods trade against each other

[2:43](https://www.youtube.com/watch?v=zL1kLftVTlo&t=163s)

The first distinction is between **where tasks come from** and **which policy generates the training trajectories**. Production tasks can match real usage while their recorded answers still come from an older policy. Conversely, a current model can generate fresh, on-policy answers to benchmark tasks that poorly represent production. These are separate axes.

Two further requirements concern infrastructure and feedback. Generating several alternative trajectories requires environments that can reproduce the same starting conditions and support parallel actions. Building faithful copies is expensive, and imperfections introduce training bias. Then, after collecting a rich interaction, reducing its outcome to one scalar discards much of the information that could explain which individual decisions were useful.

Malde’s historical comparison starts with supervised fine-tuning, associated here with instruction tuning and **GPT-3.5** and **ChatGPT**, then moves through preference training to GRPO. The table captures the training setups he contrasts; it is not a claim that every implementation of each method has these properties.

| Training setup | Task source | Sampling | Examples per task | Feedback |
|---|---|---|---|---|
| SFT | Curated dataset | Off-policy | One example | Per-token targets |
| DPO / RLHF, as presented | Online tasks | Model-generated, still off-policy | Preference pairs | Sequence-level preference |
| GRPO | Offline benchmark tasks | On-policy | Parallel rollout group | Sequence-level reward |

SFT makes the infrastructure simple and supplies token-level supervision, but learns from a fixed collection of demonstrations. Preference training brings real user tasks into the picture while introducing pairs and sequence-level judgments. GRPO prioritizes fresh on-policy rollouts; Malde credits that with powerful capability gains and fewer of the forgetting problems associated with earlier methods. Its costs are grouped rollouts, substantial environment infrastructure, and a return to whole-sequence rewards.

![Table compares SFT, DPO, and GRPO across task distribution, sampling, parallelism, and reward, using colored cells.](https://ai.engineer/api/talk-essay-media/a0a4469544bb71932f51f909ed17226828ab76c6753ccfffc9fca6d8848c5d9a)

SFT, DPO, and GRPO compared across four training criteria.

The desired combination is therefore specific: **online tasks, on-policy samples, one rollout, and dense token-level feedback**. Getting all four would let a system learn from a real interaction without first reconstructing the world and replaying the task several times.

### What a group-relative reward actually tells the model

[5:54](https://www.youtube.com/watch?v=zL1kLftVTlo&t=354s)

GRPO starts with one task and produces several parallel outputs, labeled `O1` through `O4` in the explanation. Each output receives an end-state reward. The algorithm compares those rewards with the group mean and shifts the policy toward outputs that did better and away from outputs that did worse. The group supplies a relative measure of which behavior to reinforce.

That still leaves a narrow feedback channel. Imagine writing an essay and receiving only **87/100** from a teacher. The score gives a judgment, but it does not identify a weak transition, an unsupported claim, or a good paragraph worth preserving. Many attempts may be necessary to infer what the teacher wants. GRPO combines this sparse-feedback problem with the earlier task-distribution mismatch and the need to run multiple copies of an environment.

### Make the same model a better teacher by giving it a hint

[7:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=434s)

Distillation provides a different kind of supervision. After a brief reference to **Mythos** and restrictions on distillation into open-source models, Malde starts with the conventional setup: a student and a smarter, usually larger teacher receive the same fixed dataset. Training fits the student’s token probabilities to the teacher’s. Instead of learning only whether a complete answer was good, the student learns how the teacher distributes probability at each position.

**On-policy distillation** changes the source of the trajectory. Rather than scoring a fixed demonstration, the teacher scores the student’s own rollout. The student therefore receives guidance at the prefixes its current policy actually produces. But frontier training creates a problem: if the student is already the strongest available model, where does a smarter teacher come from?

**On-policy self-distillation**, or OPSD, changes the teacher’s information rather than its underlying model. Give the same model privileged information—a hint about the task or environment—and it can make better predictions. The student receives the original prompt; the teacher receives that prompt with the hint prepended. Both evaluate the student’s trajectory, and training moves the unhinted student toward the hinted teacher’s probabilities.

The demonstration asks the student to find a function’s derivative. Guidance about how to solve it, or even a golden solution, makes the teacher’s job easier. The learning procedure is:

1. Generate a trajectory from the student using the original derivative question.
2. Add the guidance to the teacher’s prompt.
3. Evaluate the student’s trajectory under that better-informed context.
4. Train the student without the hint to match the teacher’s predictions.

The teacher does not need to generate a separate successful trajectory for the student to copy. Its advantage comes from knowing more while evaluating what the student already did.

### One rollout can carry vocabulary-wide supervision

[9:36](https://www.youtube.com/watch?v=zL1kLftVTlo&t=576s)

This setup can use an actual online task, preserve sampling from the student’s current policy, and learn from a single trajectory. Malde contrasts that with an illustrative group of eight rollouts: the extra information comes from the teacher’s predictions, so it does not require eight independent attempts in cloned environments. At each generated position, the teacher supplies a distribution rather than one end-of-sequence score.

The feedback also extends beyond the token the student happened to choose. Consider the Python `for` loop and `range` example on the slide. At a position where the student favors one token, the teacher may favor a different token from the vocabulary. Malde uses a vocabulary of roughly 65,000 tokens to illustrate the breadth of that signal. The blue regions identify alternatives that were not the student’s top sampled token but that the teacher wants to make more likely.

![Slide shows a red GRPO token row above a red-and-blue SDPO vocabulary grid, beside an OPSD student-to-student diagram labeled “Fit logprobs.”](https://ai.engineer/api/talk-essay-media/4fcb5def81e20770895dcb81d03fc72430fab41138767596654f02eeae202bac)

Token and vocabulary feedback illustrated alongside on-policy self-distillation.

A full-distribution matching loss can be expressed directly in Python. Here, each row represents the next-token distribution at the same student-generated prefix; the teacher rows were computed with the extra hint. `response_mask` selects the generated positions to train on.

```python
import torch
import torch.nn.functional as F


def distillation_loss(
    student_logits: torch.Tensor,
    teacher_logits: torch.Tensor,
    response_mask: torch.Tensor,
) -> torch.Tensor:
    # Logits: [batch, response_positions, vocabulary]
    student_log_probs = F.log_softmax(student_logits.float(), dim=-1)
    teacher_probs = F.softmax(teacher_logits.detach().float(), dim=-1)

    per_position_loss = -(teacher_probs * student_log_probs).sum(dim=-1)
    mask = response_mask.to(per_position_loss.dtype)
    return (per_position_loss * mask).sum() / mask.sum().clamp_min(1)
```

This makes the distributional idea explicit: every vocabulary entry contributes to the target at a trained position. It does not require every implementation to materialize the full vocabulary distribution. The slide labels its vocabulary grid SDPO, while Malde calls the mechanism OPSD; those labels alone do not establish identical implementation details. His central comparison is that distillation can move probability toward teacher-preferred alternatives, beyond reinforcing the successful tokens already sampled by the student.

### Promising short-task results are the starting point

[11:05](https://www.youtube.com/watch?v=zL1kLftVTlo&t=665s)

For short-horizon tasks, Malde reports that GRPO saturated around **Sonnet**-level performance on [LiveCodeBench](https://github.com/LiveCodeBench/LiveCodeBench), while OPSD reached higher performance in their experiments. He does not specify the Sonnet version, exact scores, or evaluation window. He attributes the improvement to the ability to shift the output distribution into behavior that the original policy did not readily produce.

Malde also reports sharply reduced token counts for solving difficult tasks with OPSD, without giving a numerical efficiency ratio. He contrasts this with the tendency of RL-trained models to improve by spending more tokens reasoning. The attraction is therefore not just better answers, but a route to better answers without continually lengthening the reasoning trace.

For accessible implementations, he points to [OpenClaw-RL](https://github.com/Gen-Verse/OpenClaw-RL), describing chatbots that learn user behavior from unstructured interactions. These examples establish a useful starting point: small models and short tasks can benefit from this learning signal. They do not settle what happens when an agent must maintain coherent behavior across a long sequence of tool calls.

### Long trajectories turn correction into hesitation

[12:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=752s)

Trajectory has experimented with scaling the approach to 120-billion-, 500-billion-, and one-trillion-parameter models, according to Malde. He reports instability emerging around 120 billion parameters on trajectories with 50–100 tool calls. Evaluation accuracy becomes erratic, run-to-run variance rises, and tool-call errors appear. Models also start departing from the formatting behavior established during instruction tuning.

The first failure mode is the **“but wait” problem**. On a short task, the student’s trajectory and the hinted teacher’s preferred behavior may remain close. On a longer task, the student keeps following its own policy and can drift far from the route the teacher would take. Whenever the teacher evaluates another position, it tries to redirect the trajectory.

Repeated correction creates pressure toward tokens such as `wait`, `maybe`, `then`, and `but`. Malde describes a word cloud in which these words become more prominent as training proceeds. Eventually, the policy can settle into an unhelpful local solution dominated by hesitation: the student and teacher distributions have diverged, and fitting between them produces a model that keeps reconsidering rather than making coherent progress.

### Weight each step by how far the teacher and student disagree

[14:27](https://www.youtube.com/watch?v=zL1kLftVTlo&t=867s)

One response is to measure **step-level divergence**. In an illustrative trajectory with 100 tool calls, compare the student’s and teacher’s distributions as the trajectory develops. Use their KL divergence to determine a multiplicative weight on the token loss within each step. This differs from adding a conventional KL penalty to an objective: the divergence controls how much that step contributes to learning.

Independent step weights allow three different treatments:

- **A trajectory that stays in distribution:** Keep the weights at one and learn throughout.
- **A trajectory that diverges heavily:** In Malde’s example, modify `W1` first, concentrating the correction on getting the initial step right before proceeding.
- **A trajectory that departs and later recovers:** Downweight the off-track portion, then retain a mild update where the trajectory becomes useful again.

The recovery case is particularly valuable. A poor middle section need not make every later step unusable, and a good beginning need not justify equally strong updates everywhere afterward. Malde presents this as one way to stabilize long-horizon tool calling, without specifying an exact divergence-to-weight formula.

### A teacher can know too much

[15:57](https://www.youtube.com/watch?v=zL1kLftVTlo&t=957s)

RL has reward hacking; self-distillation has an analogous problem in **hint leakage**. The teacher receives information that the student will not have at deployment. If matching the teacher teaches behavior that depends on already knowing that information, the student may learn to skip the very reasoning or investigation required to obtain it.

The numerical example asks for the last three digits of an expression. A hint includes the solution steps and reveals that the answer is `000`. The resulting reasoning starts from knowing `000` and works backward to fit that answer into the trace. This is a shortcut that would not be available when facing a new problem without the privileged hint. Hint design must therefore distinguish guidance that helps the model reason from information that lets it bypass reasoning.

A simple remedy is to have an LLM filter the hint. Suppose a user cannot log in. The raw hint says the user’s SSO token has expired. A filtered version instead directs the model to inspect the logs. The useful action is preserved, while the answer the agent should discover is removed from the teacher’s guidance. Malde says this works decently well, though it is only one approach to the problem.

![Raw and filtered hint boxes connected by an LLM filter arrow: the raw hint names an expired SSO token, while the filtered hint suggests searching the logs.](https://ai.engineer/api/talk-essay-media/7a387832f7f467bb6b8fec49e3342b32817e6e65800748d00fbaac2bba823fb6)

An LLM filter rewrites an answer-revealing hint into a suggestion to search the logs.

### Use partial hints to moderate the learning signal

[17:47](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1067s)

**Residual guidance** tackles the same problem through the teacher signal. A hint often reveals its full meaning only after the model has seen all of it. Cutting it in half creates a partial teacher: the same model has some useful information, but less than the full teacher. Both contexts can then be used to compute token probabilities.

Malde describes taking a linear combination of the partial- and full-teacher signals to assess the strength of the hint and moderate the resulting update. In his illustrated case, the partial-hint distribution remains close to the student, while the full-hint distribution lies much farther away. Combining their signals offers a way to use the extra information without pulling the student entirely into behavior supported only by privileged knowledge. The talk gives the mechanism, but not the combination coefficients or a precise implementation formula.

### Recovering the original promise at agent scale

[18:59](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1139s)

With these remedies combined, Malde reports that a 120-billion-parameter model surpassed RL on [Mercor’s APEX-Agents](https://www.mercor.com/blog/introducing-apex-agents/) tasks, which he says often require more than 100 tool calls. The talk does not provide the exact score, RL baseline configuration, or effect size for that comparison. The result matters here because it extends the approach beyond short answers into the long trajectories where naive self-distillation became unstable.

The intended properties remain intact: tasks can come from online usage, the student can generate its own on-policy trajectory, one rollout can supply a training example, and the teacher can provide dense per-token supervision. Malde presents this as a substantial step forward, not as an algorithm that has solved continual learning end to end.

### The algorithm sits inside a larger production loop

[19:46](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1186s)

A weight update is only one part of a continually improving product. Malde’s broader forecast is software that becomes more capable each time it is used, which he sees as an important opportunity for 2026–2027. At Trajectory, the improvement targets include the model, its harness, and the behavior of the complete agent system.

The platform he describes follows a production loop:

1. Ingest agent traces from real usage.
2. Optimize the system through a self-serve learning process.
3. Deploy the resulting improvements.
4. Oversee the process through a control plane.

He describes a team with backgrounds at DeepMind, Meta Superintelligence, and OpenAI, alongside product builders, and names Harvey, Decagon, and Rogo as early-access companies. The platform ambition is to connect the learning mechanism to collection, deployment, and operational control.

### Continual collection does not yet mean immediate learning

[21:13](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1273s)

The first audience question makes the deployment issue concrete: how long does training take, and can learning happen in real time? Malde describes the current research stage as **pseudo-continual learning**. Data arrives through usage, but updates still happen in offline batches, followed by re-uploading the model. He gives no training-duration measurement.

Moving from that cycle to immediate learning is both an infrastructure and an algorithmic problem. Malde asks the audience to imagine 10,000 rollouts running through a product and the work required to bring them together. Online collection alone does not guarantee that each trace remains on-policy by the time training consumes it. Trajectory is working on this coordination problem, but he explicitly says the end-to-end solution is still far away.

The final question asks whether different harnesses change the effectiveness of learning. Malde points to emerging work on harness improvement, then identifies a harder question: **what happens when the model and the harness both change?** A model update changes the behavior available to the harness; a harness update changes the contexts and action sequences the model encounters. Trajectory is exploring that interaction online with customers, but Malde offers no comparative harness results. The remaining research problem is to make the whole agent improve coherently while both sides of that interaction are moving.

## Resources

- [Scaling SDPO](https://www.trajectory.ai/field-notes/scaling-sdpo) — Trajectory's experiments on single-rollout learning, stale production data, training stability, and APEX-Agents.
- [Self-Distilled Reasoner](https://arxiv.org/abs/2601.18734) — Introduces OPSD with a privileged-context self-teacher and evaluates mathematical reasoning and token efficiency.
- [Reinforcement Learning via Self-Distillation](https://arxiv.org/abs/2601.20802) — Original SDPO research cited by Trajectory's companion article.
- [OpenClaw-RL](https://github.com/Gen-Verse/OpenClaw-RL) — Training framework that converts agent interactions into binary rewards, distillation signals, or combined updates.
- [LiveCodeBench](https://github.com/LiveCodeBench/LiveCodeBench) — Official evaluation tools for coding tasks drawn from programming competitions.
- [Introducing APEX-Agents](https://www.mercor.com/blog/introducing-apex-agents/) — Mercor's introduction to long-horizon agent evaluation across professional workflows.

## Slides

### Scaling Up Continual Learning

The talk turns from measuring continual learning to scaling it up.

[0:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=32s) [Title slide for Scaling Up Continual Learning by Trajectory.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a25bee3d06128a62e3702a7a999040b7182a60e9870f080e8c9122f765885fb.webp)

Scaling Up
Continual Learning

Trajectory

### Rapidly Saturating AI Benchmarks

Benchmark performance rises toward and sometimes above human level. The speaker says benchmarks have saturated in years and then months.

[0:43](https://www.youtube.com/watch?v=zL1kLftVTlo&t=43s) [Chart showing AI benchmark performance approaching or exceeding human-level performance over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/17f9fefc78ddceb39a679a28c566efd96e197c320467171f7825a0dd9289d749.webp)

Trajectory
MOTIVATION
01

AI benchmarks have rapidly saturated over time
EPOCH AI
Performance (normalized)

Y-axis ticks: 20%, 0%, -20%, -40%, -60%, -80%, -100%
0%: Human-level performance
Timeline: 2000, 2004, 2008, 2012, 2016, 2020, 2024
X-axis: Year

Benchmarks: MNIST, Switchboard, ImageNet, SQuAD 1.1, SQuAD 2.0, GLUE, MMLU, Big Bench, MATH, GPQA

Source: International AI Safety Report, Figure 1.4.
CC-BY
epoch.ai

AI progress has been fueled by rapidly saturating benchmarks

### Benchmark Progress Continues

AI benchmark scores continued to rise through 2025. Newer tests, including computer use and software engineering, climbed sharply toward the human baseline, while several older tests reached or exceeded it.

[0:59](https://www.youtube.com/watch?v=zL1kLftVTlo&t=59s) [Two charts compare AI benchmark performance over time, including recent progress against human baselines.](https://assets.ai.engineer/sites/aiecode2025/sha256/f548977275cdc1d48e0d46821f757f41a8c8414e0bb3937785c2e0a793159d96.webp)

Trajectory
MOTIVATION
01

AI benchmarks have rapidly saturated over time
EPOCH AI
Performance (normalized)
Y-axis ticks: 20%, 0%, -20%, -40%, -60%, -80%, -100%
Human-level performance
Timeline: 2000, 2004, 2008, 2012, 2016, 2020, 2024
Year
MNIST
Switchboard
ImageNet
SQuAD 1.1
SQuAD 2.0
GLUE
Big Bench
MMLU
MATH
GPQA
Source: International AI Safety Report, Figure 1.4.
CC-BY
epoch.ai

Select AI Index technical performance benchmarks vs. human performance
Source: AI Index, 2026 | Chart: 2026 AI Index report
Performance relative to the human baseline (%)
Y-axis ticks: 0%, 20%, 40%, 60%, 80%, 100%, 120%
Human baseline
Timeline: 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025
Legend:
Image classification (ImageNet Top-5)
English language understanding (SuperGLUE)
Multitask language understanding (MMLU)
PhD-level science questions (GPQA Diamond)
Agent multimodal computer use (OSWorld)
Autonomous software engineering (SWE-bench Verified)
Visual reasoning (VQA)
Medium-level reading comprehension (SQuAD 2.0)
Competition-level mathematics (MATH)
Multimodal understanding and reasoning (MMMU)
Mathematical reasoning (AIME)

That hasn’t stopped in the last 2 years

### The opportunity

The slide puts production AI at 100 trillion tokens per day and argues that real-world successes and failures can provide training signal. The OpenRouter chart shows weekly model usage rising sharply.

[1:42](https://www.youtube.com/watch?v=zL1kLftVTlo&t=102s) [Slide says production AI generates 100 trillion tokens per day and proposes learning from real-world interactions, alongside an OpenRouter usage chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/2994c599006a84fccfdd349bdb9141ccb02a7aee8c17bcb8017348f3437458b8.webp)

Trajectory
MOTIVATION
02

THE OPPORTUNITY

100,000,000,000,000
tokens / day

Every day, production AI generates a stream of real-world signal, where models help, where they fail. Today it goes to waste.

Learning from real world interactions continuously is more akin to how humans get smarter

Top Models
Weekly usage of models across OpenRouter
Y-axis ticks: 15T, 30T, 45T, 60T
Timeline: Jul 7, 2025; Sep 22; Dec 8; Feb 23; May 11

Interface text:
Linear
Log

### The whole field is saying the same thing

Researchers describe continual learning as a missing capability, while Satya Nadella envisions AI systems improving with use and Dwarkesh Patel predicts models will no longer be frozen.

[2:19](https://www.youtube.com/watch?v=zL1kLftVTlo&t=139s) [Researcher and industry leader quotations discuss continual learning, with a video thumbnail about future training paradigms.](https://assets.ai.engineer/sites/aiecode2025/sha256/437cde3ddcd31eaa0f097fa544ec460a27d91185c036795de1bbedb54ea6c7e1.webp)

The whole field is saying the same thing.

AI RESEARCHERS

“Real intelligence involves continual learning.”
— Ilya Sutskever

“Modern AI models lack a sleep-like ‘distillation’ phase to consolidate and turn awake experiences into habitual brain weights.”
— Andrej Karpathy

“Continual learning will get solved in a satisfying way in 2026.”
— Sholto Douglas

“Another thing that’s missing is continual learning — the ability to online-teach the system something new, or adjust its behaviour.”
— Demis Hassabis

AND INDUSTRY LEADERS

“LLMs are frozen. They won’t be in the future.”
— Dwarkesh Patel

“Companies need to turn their workflows, domain knowledge, and accumulated judgment into AI systems that improve with each use.”
— Satya Nadella

Video thumbnail:
LLMs are frozen. They won't be in the future.
What does the next training paradigm look like?
19:53
121K views • 4 days ago

Interface text:
Trajectory
EXPLORATION
03 · F

### Continual learning

A reaction meme builds from more data to more benchmarks to continual learning, which the speaker describes as the next unlock.

[2:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=152s) [A reaction meme over the quotations slide contrasts more data and more benchmarks with continual learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2c7a38332190d3a3cbdd631d74788d4f8d0540bc106ca3360abad21f5b8831d.webp)

The whole fi[illegible]

AI LEADERS ARE SAYING IT

“Modern AI models lack [illegible]
consolidate and turn awa[illegible]
weights.”
— Andrej Karpathy

“Real intelligence involve[illegible]
— Ilya Sutskever

“Continual learning will g[illegible]
— Sholto Douglas

“Another thing that’s mis[illegible]
to online-teach the syste[illegible]
behaviour.”
— Demis Hassabis

[illegible] the future.”
[illegible]kflows, domain
[illegible]ment into AI systems that

MORE DATA
MORE BENCHMARKS
CONTINUAL LEARNING

Interface text:
Trajectory
03 · F
imgflip.com
19:53
[illegible]radigm look like?

### So what’s missing?

Current methods train on curated data and off-policy behavior; grouped rollouts per prompt require perfectly replayable environments, and one scalar grades a messy trajectory.

[3:08](https://www.youtube.com/watch?v=zL1kLftVTlo&t=188s) [Slide lists task distribution mismatch, off-policy sampling, rollout parallelism, and sequence-level reward as limitations of current methods.](https://assets.ai.engineer/sites/aiecode2025/sha256/1652492b0b7c3b98d9af522666e90eaa6de03ea1f0471cbd49aacf02031fda8d.webp)

So what’s missing?

Current methods introduce offline bias

I  Task distribution mismatch
We train on curated data, not the states the model actually hits in production.

II  Off-policy sampling
We learn from behavior the current model didn’t generate, not how it really acts.

III  Rollout parallelism
Sampling a group of rollouts per prompt demands a perfectly replayable environment.

IV  Sequence-level reward
One scalar grades a whole trajectory, but reality is semantic, dense, and messy.

### The offline gap

SFT, DPO and GRPO each fall short on at least one axis. The proposed combination uses an online task distribution, on-policy sampling, single parallelism and dense reward.

[5:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=332s) [Table compares SFT, DPO, GRPO, and a proposed approach across task distribution, sampling, parallelism, and reward.](https://assets.ai.engineer/sites/aiecode2025/sha256/8859fef5f0f8028e6b0d29b4a64454b62b2d2e66838c078fa9a50907f7192a37.webp)

Trajectory
THE OFFLINE GAP

Every algorithm is still offline on some axis.

Task dist. | Sampling | Parallelism | Reward

SFT (2020–22) | Offline | None | Single | Per-token
DPO (2023) | Online | Off-policy | Pairs | Sequence-level
GRPO (2024-2025) | Offline | On-policy | Group × N | Sequence-level
??? | Online | On-policy | Single | Dense

### How GRPO works

GRPO samples parallel rollouts for one prompt, scores them, and compares each score with the group mean. It shifts the model toward above-mean rollouts and away from below-mean ones.

[5:52](https://www.youtube.com/watch?v=zL1kLftVTlo&t=352s) [Four rollouts are scored against their mean, with positive and negative signals used to update the model.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f2078fe8eb926993e682d495e88b3deaa23b0ce45c1da7733ee9be1a254d089.webp)

SLIDE 6 · ALT 1 · ONE DIAGRAM, NO CARDS

How GRPO works.
Sample a group, score within it, push toward the winners.

Prompt
q

Rollout o₁  0.9  +
Rollout o₂  0.2  −
Rollout o₃  0.7  +
Rollout o₄  0.4  −

μ = mean(r)

Update model toward winners

### Why GRPO still isn’t enough

Benchmarks differ from production, and GRPO requires parallel rollouts. Its single reward per trajectory is like an 87/100 essay grade: it ranks work without showing how to improve it.

[6:37](https://www.youtube.com/watch?v=zL1kLftVTlo&t=397s) [Lists task distribution mismatch, parallelism, and scalar rewards, alongside an essay grading comparison.](https://assets.ai.engineer/sites/aiecode2025/sha256/747c13a2747fc0e23ad495f83049a2da6967791f3a099aaf301bf52dc0e8cd60.webp)

Why GRPO still isn’t enough.

Trajectory | THE OFFLINE GAP | 07

1. Tasks are not aligned to production
Benchmarks are from a different distribution

2. Requires parallelism
Parallel rollouts, reproducible environments, and judge criteria for every task.

3. One scalar per trajectory
The whole rollout collapses to a single number, about one bit of feedback.

GRADING AN ESSAY
RL reward: 87/100
Dense feedback
≈ O(1) · one number
≈ O(N) · every token
A single score can rank essays. It can’t teach anyone how to write.

GRPO
Offline
On-policy
Group × N
Sequence-level

### Deriving on-policy self-distillation

On-policy distillation uses a student-generated trajectory. Self-distillation then fits the student’s log probabilities to those of the same model given a privileged hint.

[8:28](https://www.youtube.com/watch?v=zL1kLftVTlo&t=508s) [Three diagrams compare fixed-data distillation, on-policy distillation, and self-distillation using a privileged hint.](https://assets.ai.engineer/sites/aiecode2025/sha256/4fa0f0e97840e1c1b78b77209e3034fdceee2f8f6159943e23d1eb8b2a4efcab.webp)

Deriving On-Policy Self Distillation

Distillation
OFF-POLICY
fixed data
Student
Fit logprobs
Teacher
Student and teacher read the same fixed trajectory. Fit the student’s logprobs to the teacher’s.

On-policy distillation
ON-POLICY
student generates τ
Student
Fit logprobs
Teacher
Now the student generates the trajectory. Same teache, but on-policy.

OPSD
SELF-DISTILLATION
What if we don’t have a smarter teacher?
Student
Fit logprobs
hint
same model + hint
Student
No stronger teacher. The student, given a privileged hint, becomes its own teacher.

squares =
off-policy trajectory
on-policy · student-generated
privileged hint

### Student and teacher prompts

The student receives a derivative problem; the teacher receives the same problem plus a worked solution. Self-distillation matches the student’s log probabilities to those of the teacher with that hint.

[9:01](https://www.youtube.com/watch?v=zL1kLftVTlo&t=541s) [A derivative problem appears in both prompts; the teacher prompt also includes a worked solution.](https://assets.ai.engineer/sites/aiecode2025/sha256/84b7a3c1d7470d462d701c796b052836d73e444dd6de3f52f5e9abe0a5624882.webp)

Deriving On-Policy Self Distillation

Student Prompt
Problem: Find the derivative of f(x) = 3x² + 2x − 5 at x = 2
Answer:

Teacher Prompt
Problem: Find the derivative of f(x) = 3x² + 2x − 5 at x = 2
Here is a reference solution:
First find f′(x) = 6x + 2, then evaluate at x = 2: f′(2) = 6(2) + 2 = 14
After understanding the reference solution, please try to solve this problem using your own approach below:
Answer:

On-Policy Self-Distillation for Large Language Models (Zhao et al., 2026)

### Comparing GRPO and OPSD

OPSD fits the student’s log probabilities to those of the same model given a privileged hint. The slide contrasts its online tasks and per-token feedback with GRPO; both use on-policy sampling.

[9:34](https://www.youtube.com/watch?v=zL1kLftVTlo&t=574s) [A table compares four training features alongside a self-distillation diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d7a3a179152bd02d97faaed35ede6824765d90f0de00d7e566b99b94d2955bd.webp)

On-Policy Self Distillation

GRPO | OPSD
Online Task Distribution: ✗ | ✓
On-policy sampling: ✓ | ✓
No parallel rollouts: ✗ | ✓
Dense, per-token reward: ✗ | ✓

OPSD
SELF-DISTILLATION
What if we don’t have a smarter teacher?
Student
Fit logprobs
hint
same model + hint
Student

No stronger teacher. The student, given a privileged hint, becomes its own teacher.

### Token-level self-distillation

With a privileged hint, the student becomes its own teacher. Self-distillation can shift the distribution across the vocabulary at each token, including toward tokens the student did not sample.

[10:13](https://www.youtube.com/watch?v=zL1kLftVTlo&t=613s) [A GRPO token sequence and an SDPO vocabulary grid appear beside a self-distillation diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/df8cdc7bb71b6d7148a0eb5d5fbfeb295f7973c64b897f929066a3444e5823c9.webp)

On-Policy Self Distillation

GRPO
Generated tokens →
… (range | ( | 1 | , | n | + | 1 | ))\n

SDPO
Generated tokens →
Vocabulary ↓
(range | ( | 1 | , | n | + | 1 | ))\n
range | (n | 2 |  | n | + | 0 | ))
range | (- | 0 | ,\n | .n | ))\n | 2 | ))\n\n
.range | [ |  | . | (n |  | 3 | ))}\n
⋮

OPSD
SELF-DISTILLATION
What if we don’t have a smarter teacher?

Student
Fit logprobs
hint
same model + hint
Student

No stronger teacher. The student, given a privileged hint, becomes its own teacher.

### Short-horizon task results

On LiveCodeBench v6, SDPO rises above GRPO and the Claude Sonnet 4 reference. The training charts show a 6× speedup and an 11× reduction in response length versus GRPO.

[11:06](https://www.youtube.com/watch?v=zL1kLftVTlo&t=666s) [Charts compare coding accuracy, training accuracy and response length for SDPO and GRPO.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe805ce7ea223fb859bb77c2e03c8972ca6bf475be1dedd778d31a7ada929ac0.webp)

On-Policy Self Distillation

For short horizon tasks, OPSD works well!

LiveCodeBench v6
Y-axis: Accuracy
Y-axis ticks: 0.30, 0.35, 0.40, 0.45, 0.50
X-axis: Number of Generations
X-axis ticks: 0, 5000, 10000, 15000, 20000
Legend: SDPO, GRPO, Claude Sonnet 4

Legend: SDPO (on-policy), GRPO (on-policy), GRPO

Test accuracy during training
Y-axis: Accuracy
Y-axis ticks: 0.2, 0.4, 0.6, 0.8
X-axis: Training wall-clock time (hours)
X-axis ticks: 0, 1, 2, 3, 4, 5
6× speedup

Response length during training
Y-axis: Response length
Y-axis ticks: 0, 1000, 2000, 3000, 4000
X-axis: Training wall-clock time (hours)
X-axis ticks: 0, 1, 2, 3, 4, 5
11× reduction in response length

### Step-Level Divergence

The proposed step-level divergence averages the absolute gap between student and teacher token log probabilities within a step. The speaker proposes using it to weight the step’s tokens.

[14:28](https://www.youtube.com/watch?v=zL1kLftVTlo&t=868s) [A formula defines divergence from student and teacher token log probabilities within a step.](https://assets.ai.engineer/sites/aiecode2025/sha256/352bdefe04db46c8465e006d37e9c388aef7bd5f2138b29f6cd1bc5b2f8d6265.webp)

OPSD Challenge #1 - “but wait!”

On long horizon tasks, the teacher attempts to diverge on each token, collapsing to no answer

Potential solution: Define a step-level divergence term

 d_k = 1/|𝓘_k| ∑_{t∈𝓘_k} |log π_θ(y_t | y_<t) − log π_teacher(y_t | y_<t)|.

SOD: Step-wise On-policy Distillation for Small Language Model Agents (Zhong et al., 2026)

### Filtering hints with an LLM

An LLM filter turns a hint revealing an expired SSO token into a suggestion to search the logs, guiding the student toward evidence while preserving intermediate reasoning.

[17:13](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1033s) [A raw troubleshooting hint is rewritten to point toward logs without revealing the diagnosis.](https://assets.ai.engineer/sites/aiecode2025/sha256/6403d641d173c5542f946048907cfc221eefcc45c41eece65aecc10864294803.webp)

OPSD Challenge #2 - Hint Leakage

The teacher’s hint might contain information that the student shouldn’t have known, leading to skipped intermediate reasoning

One solution: LLM-based filtering

RAW HINT
User can’t log in. If you search the logs, you’ll find their SSO token expired.
✗ hands over the answer

LLM filter →
rewrites the answer as a nudge

FILTERED HINT
User can’t log in. You should try searching the logs.
✓ points to the evidence, not the answer

### Anchored residual guidance

Residual guidance truncates the privileged hint to half its length by default. It anchors the target to the partial teacher, then adds a scaled full-minus-partial residual.

[18:18](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1098s) [Equations define partial and full teachers and an anchored residual target.](https://assets.ai.engineer/sites/aiecode2025/sha256/79bb1d1eeb6edad0529df1ac44b2152e105bf2ee9b57a0e94f3d251dcf18c35c.webp)

OPSD Challenge #2 - Hint Leakage

The teacher’s hint might contain information that the student shouldn’t have known, leading to skipped intermediate reasoning

Another Solution: Residual Guidance

Let the privileged information be z = (z₁, ..., zL). We construct a partial privileged context by deterministic truncation:

zpart = z₁:⌊ρL⌋, ρ ∈ (0, 1). (5)

The default setting is ρ = 0.5. The partial teacher is

qpart(· | st) = πθ̄(· | x, zpart, y<t). (6)

The full teacher is

qfull(· | st) = πθ(· | x, z, y<t). (7)

Anchored residual target.

log qλ(v | st) = log qpart(v | st) + λ(log qfull(v | st) − log qpart(v | st)) − Ct. (9)

partial anchor
full-minus-partial residual

Anchored Residual Guidance for Privileged On-Policy (Zhang et al., 2026)

### Residual guidance comparison

Standard OPD pulls the student toward the full-view teacher, potentially shifting its distribution far from the student’s. Residual guidance anchors the target to a partial hint and controls the shift toward the full view.

[18:28](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1108s) [Diagrams compare a full-view teacher target with an anchored residual guidance target.](https://assets.ai.engineer/sites/aiecode2025/sha256/5dfd656c289c0b14ec81dea18ce98a0b357f7aff2dac228744e0c0e83971f83b.webp)

OPSD Challenge #2 - Hint Leakage

The teacher’s hint might contain information that the student shouldn’t have known, leading to skipped intermediate reasoning

Another Solution: Residual Guidance

Standard Privilege Info OPD
Student
Imitate q_full
Monolithic target
Full view q_full
Excess full view shift

Anchored Residual Guidance
Δ_t = Controllable Residual Guidance
Student
Partial anchor view q_part
Target q*
q* = Softmax(log q_part + λΔ_t)
controlled transport
Full view q_full

Anchored Residual Guidance for Privileged On-Policy (Zhang et al., 2026)

## Full transcript

[0:00](https://www.youtube.com/watch?v=zL1kLftVTlo&t=0s) [upbeat music] All right. Hey, everyone. Uh, thanks for coming by over here.

[0:16](https://www.youtube.com/watch?v=zL1kLftVTlo&t=16s) Hope you enjoyed the chat by Parth on, uh, how to measure continual learning. Now I'm here to talk about how we scale it up. So a bit of background about me.

[0:25](https://www.youtube.com/watch?v=zL1kLftVTlo&t=25s) I, uh, went to this company called Windsurf, where I was growing the research team over there. Uh, we trained this model called SWE-1 that ended up leading to the two billion acquisition at DeepMind, and then I ended up giving up all the acquisition money to start Trajectory, where we're building the platform for continual learning.

[0:42](https://www.youtube.com/watch?v=zL1kLftVTlo&t=42s) So, uh, right to dive in, uh, I think it's useful to talk a little bit about what AI progress, uh, has looked like for the past couple of years.

[0:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=51s) So we've been in this mode where we've been rapidly scaling up benchmarks that have saturated within first years and then months. And that has continued on for the past couple of years and, and will continue to grow as well.

[1:05](https://www.youtube.com/watch?v=zL1kLftVTlo&t=65s) The problem is it's becoming more time-consuming and more expensive. Uh, we're seeing domains where it takes four hours, six hours, twenty-four hours, or even several days in order to scale up benchmarks.

[1:18](https://www.youtube.com/watch?v=zL1kLftVTlo&t=78s) And obviously, you guys have seen the massive amount of money that the labs are pouring into scaling up RL environments and data. Now, the thing is, uh, we are left with a bunch of benchmarks that are getting more time-consuming, uh, more and more expensive, and, and perhaps more concerningly, uh, they're not tied to real-world use cases where

[1:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=100s) people are using AI. Now, the, uh, trillion, uh, token problem here is that we are actually spending hundreds of trillions of tokens every single day on inference. Uh, and we're generating great amounts of data on how models in the real world are, are failing, how they're doing well, uh, and that should be signal that we should be

[2:00](https://www.youtube.com/watch?v=zL1kLftVTlo&t=120s) capturing and training on. Uh, and this is actually how humans learn, right? We're continuously updating in the real world and getting smarter every single day. And, uh,

[2:11](https://www.youtube.com/watch?v=zL1kLftVTlo&t=131s) we're seeing now the entire field talking about continual learning from AI experts like Ilya, Karpathy, Sholto, uh, and Demis, uh, to also many industry leaders as well. Uh, Satya, uh, talking about how it helps companies.

[2:25](https://www.youtube.com/watch?v=zL1kLftVTlo&t=145s) And, uh, there was actually a great video by Dwarkesh a couple of days ago talking a lot about the intricacies of scaling up continual learning. And obviously, the goal is we started with more data training on internet-scale pre-training, scaling up to benchmarks, but the real unlock moving forward will be continual learning.

[2:43](https://www.youtube.com/watch?v=zL1kLftVTlo&t=163s) So what's the problem? Why aren't we here yet? The main thing is, uh, th-there's several problems right now with our current algorithms. So first is we have a task distribution mismatch.

[2:55](https://www.youtube.com/watch?v=zL1kLftVTlo&t=175s) We're scaling up these benchmarks which might not even be tied to reality of what people are training on. And second, uh, oftentimes some of these methods aren't even sampling truly on policy with what, what's happening online.

[3:09](https://www.youtube.com/watch?v=zL1kLftVTlo&t=189s) Third is, uh, we have a paradigm right now where we're sending out multiple rollouts, and this requires huge amounts of infrastructure to make sure our environments are one-to-one copies of the real world, uh, and it ends up being a bias that we're adding to our training paradigm.

[3:26](https://www.youtube.com/watch?v=zL1kLftVTlo&t=206s) And then finally, third is we are shoving every single reward into one scalar in order to train on when the real world is messy, it's noisy, and it has rich amounts of data, uh, that should be per token signal.

[3:38](https://www.youtube.com/watch?v=zL1kLftVTlo&t=218s) So we have all of these kind of broken criteria, uh, and we're training on it. And I wanna take a little bit of a step through memory lane to, to see how we've been dealing with algorithms in the past.

[3:50](https://www.youtube.com/watch?v=zL1kLftVTlo&t=230s) Um, so the first part, uh, you know, we started with SFT. These were the, the days of instruction fine-tuning, GPT-3.5, ChatGPT. And, uh, if you take a look at these four criteria-- I mean, we had parallelism, uh, solved back then.

[4:03](https://www.youtube.com/watch?v=zL1kLftVTlo&t=243s) It was just one roll-- or one, uh, u-use case or example that we needed to train on. So that was a solved problem. Uh, and then the reward for SFT is per token, which is great.

[4:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=254s) But we are not sampling on policy, and the task distribution is just some sort of benchmark that we've curated or some sort of dataset. Then we moved on to the DPO RLHF age.

[4:25](https://www.youtube.com/watch?v=zL1kLftVTlo&t=265s) Uh, this is when ChatGPT really started taking off. Uh, and we finally got online task distributions that we were able to train on. Uh, but sampling, it was a little bit better because we were actually sampling from a model, but it was still off policy.

[4:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=280s) But then we ended up losing some of the, uh, the key infrastructure, easy infrastructure that we had with SFT. Now we suddenly had pairs that we were training on, uh, and then the reward went back to sequence level.

[4:53](https://www.youtube.com/watch?v=zL1kLftVTlo&t=293s) Then we moved on to GRPO, which is kind of the mode that we're in right now. Uh, we basically took a Faustian bargain and wanted to max on on-policy rollouts, which is extremely powerful, right?

[5:04](https://www.youtube.com/watch?v=zL1kLftVTlo&t=304s) We now have models that are capable of amazing things, uh, because they are on policy and able to grow and, and not have all of the catastrophic forgetting problems that we've had with previous methods.

[5:15](https://www.youtube.com/watch?v=zL1kLftVTlo&t=315s) But on the other hand, we're working with off-policy task distributions. Uh, our parallelism has now exploded, meaning that we need really robust environment infrastructure. And then finally, for rewards, uh, we're back to this paradigm of training on the entire sequence.

[5:31](https://www.youtube.com/watch?v=zL1kLftVTlo&t=331s) So can we get to a world where all of these are true, where we have an online task distribution, we're sampling on policy, we have one parallelism, so we don't need any of this crazy infrastructure, and then finally, uh, our reward is actually token level?

[5:49](https://www.youtube.com/watch?v=zL1kLftVTlo&t=349s) Well, this is what I'm excited to share and, and see how we can scale this up.

[5:54](https://www.youtube.com/watch?v=zL1kLftVTlo&t=354s) So first, le-let's just go through what our best algorithm of post-training is today. Hopefully, you guys are all familiar with how GRPO works high level. Uh, you start off with some sort of task as an input.

[6:05](https://www.youtube.com/watch?v=zL1kLftVTlo&t=365s) You do a bunch of parallel rollouts. So we'll call these O one through O four. And then you have some sort of end state reward. So you classify these, let, let's say as, uh, a couple numbers, and then GRPO works on advantage, the idea that you have some sort of mean that you're calculating over.

[6:21](https://www.youtube.com/watch?v=zL1kLftVTlo&t=381s) And you're trying to shift the distribution of the model to be, uh, better on the ones that you're better on, uh, and then obviously move away from the ones that were worse than the mean.

[6:29](https://www.youtube.com/watch?v=zL1kLftVTlo&t=389s) So this is how GRPO works. Now let's take a look at a new continual learning algorithm called self-dist- or, uh, uh, uh, self-distillation policy optimization. Um, no, uh, sorry.

[6:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=400s) First of all, we'll talk through why GRPO actually is, uh, still not enough. So, um, the first part is, uh, if you remember the tasks. So these benchmarks are obviously very different.

[6:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=411s) Uh, it requires a bunch of parallelism. And then it's almost like you're drinking through a straw in order to get the reward. The way to think about it is imagine you were trying to write an essay, and your teacher just gave you a score of eighty-seven out of a hundred.

[7:04](https://www.youtube.com/watch?v=zL1kLftVTlo&t=424s) Uh, you would have to run through so many different examples to get to the idea of what a good essay is, and it's, it-- very sample-inefficient. Um, and so these are the fundamental problems with RL.

[7:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=434s) Now, uh, we're, we're gonna go through a new algorithm, uh, con- called on-policy self-distillation. So, uh, you guys are probably familiar with distillation in general. Uh, there's been obviously a lot of talk about Mythos and, and blocking that from happening for a lot of open source models.

[7:29](https://www.youtube.com/watch?v=zL1kLftVTlo&t=449s) Uh, the high level here is that you start with some sort of dataset. Uh, it's usually, uh, off policy or not actually tied to reality, but some sort of fixed dataset.

[7:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=460s) And what you have is a student model and then a smarter teacher model, so usually a bigger model. Uh, and you're trying to basically fit the log probs of the student to the teacher, and both of these are passed in with the same data.

[7:53](https://www.youtube.com/watch?v=zL1kLftVTlo&t=473s) All right, so this is normal distillation. Now there's a new innovation called on-policy distillation. The only thing we're doing is actually swapping out the fixed data for instead a rollout of what the student would've rolled out.

[8:07](https://www.youtube.com/watch?v=zL1kLftVTlo&t=487s) All right? So we take a, a now rollout of the student. That is our trajectory, and then we're trying to fit the log probs again of a smarter teacher model to the student.

[8:16](https://www.youtube.com/watch?v=zL1kLftVTlo&t=496s) Great. Now, the problem is when we're trying to push the frontier, we don't magically have some smarter model, right? Uh, uh, like a teacher. And so now what do we do if we're, we're already at the smartest model?

[8:28](https://www.youtube.com/watch?v=zL1kLftVTlo&t=508s) Well, here's the final algorithm. Uh, and this is where the self-distillation part comes in. If you basically take the, the student model and give it some sort of what we call privileged information, a hint about the world, and put that into the prompt, well, suddenly that student is a little bit smarter, and that's essentially the key idea

[8:49](https://www.youtube.com/watch?v=zL1kLftVTlo&t=529s) of on-policy self-distillation. You take what's called this hint, put it into the beginning of the prompt, and now you match the log probs of the student without that hint to the teacher with that hint.

[8:59](https://www.youtube.com/watch?v=zL1kLftVTlo&t=539s) Uh, and this is an extremely powerful algorithm. To visualize it a little bit more, uh, uh, let's say you have this student prompt, right? Uh, find the derivative of this function.

[9:09](https://www.youtube.com/watch?v=zL1kLftVTlo&t=549s) What a hint would be is if you had some sort of environment information or, or some sort of guidance on how you should actually solve the problem. The simplest form of this is, is literally just a example of a golden solution.

[9:20](https://www.youtube.com/watch?v=zL1kLftVTlo&t=560s) You put that into the teacher prompt and then say, "Here is guidance on how you would solve the problem." And you can imagine now that solving that problem from the teacher's perspective becomes a little bit easier, and we're, we're trying to shift to those log probs had it known the answer in the first place.

[9:36](https://www.youtube.com/watch?v=zL1kLftVTlo&t=576s) Um, and so by doing this, we've solved several key problems that existed with RL. Uh, so if you remember, uh, the task distribution, now we can suddenly take something that is truly from online without needing to have sort, sort of benchmarks that are created.

[9:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=591s) Uh, the second part is we're still on policy sampling, which is great. Uh, but now there's no parallel rollouts. We don't need a group of eight in order to roll out, but just from a single example, we're able to get information.

[10:02](https://www.youtube.com/watch?v=zL1kLftVTlo&t=602s) So that takes away the environment bottleneck and all of these other infrastructures. And then finally, the most exciting part is we're matching every single log prob of every token.

[10:10](https://www.youtube.com/watch?v=zL1kLftVTlo&t=610s) So there's massively rich feedback about what this algorithm is doing.

[10:15](https://www.youtube.com/watch?v=zL1kLftVTlo&t=615s) Uh, to see it a little more in action, a- and this is a really exciting part of self-distillation, is it's actually not just the top token that you sampled, uh, that you're actually making better, but instead the entire vocabulary.

[10:28](https://www.youtube.com/watch?v=zL1kLftVTlo&t=628s) So for every single token, there's a vocabulary of, let's say, sixty-five K tokens that you're optimizing over. And let's say the model generated some sort of, uh, for loop or, uh, a range in Python.

[10:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=640s) And essentially what we're doing is saying, "Hey, the teacher," uh, for some of these areas where it's blue, it wasn't the top sampled token of the student, but instead we're pushing that distribution to sample a brand-new token.

[10:52](https://www.youtube.com/watch?v=zL1kLftVTlo&t=652s) And this is really exciting because we're not just taking now a distribution like RL and slightly sharpening it, but we're instead actually shifting entire distributions. So this is a really exciting part about OPSD.

[11:05](https://www.youtube.com/watch?v=zL1kLftVTlo&t=665s) Now, we might be saying, uh, you know, OPSD is, is awesome and, and we've solved this problem. So I mean, for a short horizon task, it works incredibly well.

[11:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=674s) If you take a look at LiveCodeBench, uh, actually we found that GRPO saturates a- around Sonnet level performance and, and doesn't really push the frontier. But because we're actually shifting distributions, we're able to get to brand-new territory, uh, of results with a lot of datasets.

[11:30](https://www.youtube.com/watch?v=zL1kLftVTlo&t=690s) And then the really cool part is, um, with RL, a fundamental limitation a- and if you guys have ever trained RL models, is the models like to think a lot, right?

[11:39](https://www.youtube.com/watch?v=zL1kLftVTlo&t=699s) The more tokens that you expend, it's, it's just going to do better. But with OPSD, you don't have that problem. And so the actual tokens to solve some of these really difficult challenges actually collapses, which is really exciting for token efficiency.

[11:54](https://www.youtube.com/watch?v=zL1kLftVTlo&t=714s) And, uh, for some of these short horizon tasks, you can actually plug and play OPSD right now with several open source projects. So, uh, OpenClaw is a really good example of this, OpenClaw RL.

[12:03](https://www.youtube.com/watch?v=zL1kLftVTlo&t=723s) Uh, and you can actually use it to do simple chatbots, learn from your behaviors with unstructured data, uh, which is super, super awesome. All right. So we might be thinking, "Okay, continual learning is solved.

[12:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=734s) We can all go home and, and be super happy." Uh, the thing is, this works really well for small models, short-horizon tasks, like something like a chatbot. Uh, but this is where academic papers kind of end, uh, and where you really need to scale things up to start to see, to see the limitations.

[12:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=752s) Uh, so at Trajectory we've been scaling up this algorithm, um, to 120Bs to 500Bs to one trillion parameter models. A- and as soon as you get to the 120B range with not just one or two tool calls but 50 or 100, well, things start to break apart a little bit.

[12:50](https://www.youtube.com/watch?v=zL1kLftVTlo&t=770s) So, uh, first of all, uh, eval accuracy is all over the place, right? The range is going really high. Uh, run-to-run variance is extremely high. And then also we start to see a lot of tool call errors.

[13:02](https://www.youtube.com/watch?v=zL1kLftVTlo&t=782s) Uh, the model is not behaving accordingly to the format that it was trained on in the first place with the instruction fine-tuning. All right. So what are the problems that we started to see with this algorithm as we were scaling it up?

[13:15](https://www.youtube.com/watch?v=zL1kLftVTlo&t=795s) So the first part i- is actually really funny and what I call the, uh, but wait problem. So on shorter tasks, you're fitting to a distribution, everything's great. But when you move on to longer tasks, what you get is, is basically this student model.

[13:30](https://www.youtube.com/watch?v=zL1kLftVTlo&t=810s) It's going off and doing whatever it thinks is on policy. And then at some point, because you're so divergent in a long task, the teacher is going to try to course-correct every single time it gets a chance.

[13:42](https://www.youtube.com/watch?v=zL1kLftVTlo&t=822s) And so what you end up with is the teacher model just, uh, continuously trying to improve this token of wait or, uh, maybe, or some of these kind of hedging words.

[13:52](https://www.youtube.com/watch?v=zL1kLftVTlo&t=832s) And there's this really interesting Wordle on the, the, uh, bottom left there you can see... Or word cloud, uh, that as steps go on with OPSD and you really scale it up, uh, you start to see some of these words like wait and then, but, uh, start to appear.

[14:07](https://www.youtube.com/watch?v=zL1kLftVTlo&t=847s) And then you actually end up in this really interesting local suboptimal position where everything just turns into maybe. And on the right here, you can kinda see some visualization of this.

[14:17](https://www.youtube.com/watch?v=zL1kLftVTlo&t=857s) As the task goes on, you get two different distributions that are really divergent, and what you end up with is the model trying to be in the middle of both of those, uh, which is obviously really suboptimal.

[14:27](https://www.youtube.com/watch?v=zL1kLftVTlo&t=867s) All right, so there's a couple ways to solve this. Uh, one potential solution is to define step-level divergence. So, uh, in a tool calling trajectory, let's say you have 100 tool calls going on.

[14:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=880s) Well, what we can do is start to look at the KL divergence of the student model and the teacher model as time goes on. And now, uh, we can use this as a weighting factor.

[14:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=891s) So not just like normal KL where we use that as a KL penalty, but instead we're actually multiplying the token weight, uh, of every single step based on this divergence property.

[15:04](https://www.youtube.com/watch?v=zL1kLftVTlo&t=904s) And now the cool part of this, uh, is that... So on the left side, we have a normal trajectory that is like pretty in distribution, and everything's fine, right?

[15:12](https://www.youtube.com/watch?v=zL1kLftVTlo&t=912s) It's everything is a weight of one. On the second, uh, part we have a trajectory that diverges pretty heavily. And so in here we're only going to modify W1 as the first step.

[15:24](https://www.youtube.com/watch?v=zL1kLftVTlo&t=924s) Uh, and, and so that we can train that, get that right and then move on, which is awesome. And then finally on the third part, and this is really cool by having independent weighting for every step, is we can actually have a scenario where, uh, the model might go off tracks.

[15:37](https://www.youtube.com/watch?v=zL1kLftVTlo&t=937s) We don't want to heavily weight that in. Uh, but then later on in the trajectory it might get back on track again and, and we're fine with that and will mildly shift the distribution.

[15:48](https://www.youtube.com/watch?v=zL1kLftVTlo&t=948s) So this is one way that we've been able to overcome this for a long-horizon tool calling. The second part... A- and this is actually really nefarious with, uh, OPSD.

[15:57](https://www.youtube.com/watch?v=zL1kLftVTlo&t=957s) So with RL, the number one problem that people face is reward hacking, as you guys are, are all aware of. Uh, and that's a game that continuously RL researchers have had to play.

[16:06](https://www.youtube.com/watch?v=zL1kLftVTlo&t=966s) Well, there is an equivalent for OPSD as well, and that is hint leakage. The way you can think about it is we're, we're taking this hint, right? Putting it in the, the beginning of a prompt.

[16:16](https://www.youtube.com/watch?v=zL1kLftVTlo&t=976s) Well, if the student had no way of knowing what that hint would be, uh, then you're gonna end up with some weird scenarios and, and kind of skipping some steps along the way.

[16:25](https://www.youtube.com/watch?v=zL1kLftVTlo&t=985s) So the way this manifests, uh, is... Uh, let's take this example here where you're trying to find the last three digits of, of this formula. Uh, and a normal hint might be, uh, actually giving the correct steps and then maybe a final answer and saying, "Hey, actually, hint, the last three digits are all zeros."

[16:43](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1003s) Well, you can see on the right here that when you actually roll out the model, uh, you end up with something that's says, "Oh, actually, I know what the solution is.

[16:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1011s) It's zero, zero, zero. So let me go back and put that into my reasoning trace and then figure out what's going on." Well, you can imagine that this is not gonna occur whatsoever in the real world.

[17:00](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1020s) And so we've ended up in this really strange position. Uh, and so there's a lot of care that you have to put into how you design these hints and making sure there's not leakage of information.

[17:10](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1030s) So, uh, there's one kind of trivial solution to this that you can imagine, and this is just literally using an LLM to filter out these hints. So an example of this is, uh, you know, l- let's say a user can't log in.

[17:22](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1042s) You have some sort of problem like this. Uh, and there might be some sort of information like, uh, e- exactly the solution, right? You'll find their SSO token that is expired, and you can have an LLM basically translate that into what is something reasonable that they should have known.

[17:36](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1056s) And that's the process of looking through the logs but not actually giving it the solution that it would shortcut some of its vital reasoning. So this is one trivial, uh, solution, and it works decently well.

[17:47](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1067s) But there's some more satisfying algorithmic approaches too. One of these is called residual guidance. The general idea is in a hint, uh, most of the time you need to actually get through the entire hint in order to get the full information.

[18:01](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1081s) So what if you, let's say, cut it in half? Then you have a partial hint. And then you basically say, "Okay, this is the partial teacher." We're able to get the log probs of a slightly smart teacher.

[18:14](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1094s) And then we have the full teacher as well, and that's with the full hint. Now what we can do is actually take the linear combination of both of these, and this gives us a good idea of how strong the hint is and how out of distribution it is for the original model.

[18:30](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1110s) And what you end up with here is-- So on the left, you have normal OPD, uh, where you might be entirely shifting distributions, and there's almost no overlap between your model and the original solution, uh, to this kind of cool world where, uh, let's say, half the hint is actually quite close to your distribution, but the full

[18:46](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1126s) hint is very off. And you take a linear combination, and so then you're not shifting the model into unknown territory. So these are just some of the solutions to, uh, a lot of the challenges that we've had with scaling up OPSD.

[18:59](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1139s) Uh, but it is a very, very powerful algorithm. And, uh, with a lot of these combinations together, we're actually s- able to scale this up to, uh, a 120B model on Mercor Apex agents, uh, which often requires a hundred and pl-- or, or plus tool calls in order to achieve.

[19:15](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1155s) And it's a really powerful algorithm that has even surpassed RL as well.

[19:20](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1160s) So now we've finally arrived at a algorithm. It's not necessarily the algorithm to solve continual learning, but, uh, definitely one that is a huge step forward, that keeps the on-policy nature that makes RL so powerful.

[19:33](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1173s) But then it also has, uh, finally, the task distribution that is online. Uh, it has parallelism, uh, that is singular, so we don't need all of this infrastructure. And then finally, it is per-token dense reward.

[19:46](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1186s) So the, uh, the really exciting part for us and, and what we're really focused on now at Trajectory is, uh, this just gives you one taste of the entire continual learning loop.

[19:56](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1196s) And there is a really exciting world that is about to come where software, in general, just gets smarter every single time it's used. And that is the most exciting unlock that's gonna happen in twenty twenty-six, twenty twenty-seven, and as we scale up.

[20:10](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1210s) So a little bit about Trajectory. We are, uh, building the platform that turns every interaction into model improvement, harness improvement, uh, and the entire agentic loop just getting smarter over time.

[20:23](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1223s) Um, and we're, we're building this platform where we take in agent traces data from production. We're able to optimize that, uh, as a self-serve loop and then deploy that, uh, as a continually learning system.

[20:36](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1236s) And, uh, we have a control plane that goes over all of that. Um, so very quickly, our team is super awesome. We're from, uh, DeepMind, Meta Superintelligence, OpenAI, uh, and a lot of great product builders as well.

[20:47](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1247s) Um, and we're-- we've given early access to a lot of companies, so Harvey, Decagon, Rogo, um, and they're super excited about what we're doing. Uh, if you're interested in any of the research that we're doing, uh, or any of the product that we're building, uh, definitely let me know.

[21:01](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1261s) Keep in touch, and, uh, happy to answer any questions. [clapping]

[21:13](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1273s) Um, just, uh, how long does the training take? Is it continual in the sense of like very little latency so that it can run in real time, or is this something that happens in the back?

[21:24](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1284s) Yeah, that's a great question. Uh, so what I would say is a-as a research community right now, we're in this zone of what I call pseudo-continual learning, uh, where there's some still level of like batch updates offline, uh, and then re-uploading the model.

[21:40](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1300s) Uh, I think it's partly an infrastructure question. It's partly still an algorithmic question as well of how do you truly get-- When you have ten thousand rollouts going out in a product, merging those together, the infrastructure to pull all of those together.

[21:51](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1311s) So those are some of the problems that we are solving at Trajectory. Uh, but I wouldn't say we're anywhere close to the, the end-to-end solution.

[22:00](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1320s) Okay. The follow-up question, uh, about harness. Um, have you studied the effect of using different harnesses in terms of effectiveness of the-

[22:11](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1331s) Totally. I, I think that's actually one of the most underexplored and most exciting questions is not only just harness improvement, right? And, and I think there's some literature out there now starting to explore that.

[22:22](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1342s) But the really exciting part is how does the model and the harness interplay with each other as you're both updating them. That's some of the stuff that we're now exploring with our current customers and, and really doing those things online.

[22:32](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1352s) Uh, but it's completely underexplored territory, and there's some really exciting innovations to be made there as well.

[22:36](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1356s) We'll take all these questions outside 'cause we need to get up for the next speaker.

[22:41](https://www.youtube.com/watch?v=zL1kLftVTlo&t=1361s) Cool. All right. Thanks so much, guys. [clapping] [outro jingle]
