Scaling up Continual Learning

How can models learn from the work people already ask them to do? Following a talk about measuring continual learning, Ronak Malde turns to scaling it. His starting point is experience growing Windsurf’s research team and training SWE-1, before founding Trajectory. He describes giving up acquisition-related compensation to make that move; the Google transaction he references was a hiring and licensing deal, rather than a purchase of Windsurf itself.

The existing path to progress has been to build benchmarks, train against them, and replace them as they saturate. Malde describes saturation accelerating from years to months, while the remaining tasks become increasingly expensive: some take four, six, or twenty-four hours, and others take days. More investment in reinforcement-learning environments and data can sustain that process, but it does not ensure that the tasks resemble what users actually need.

Meanwhile, production inference generates examples of models succeeding, failing, and responding to the consequences of their actions. Malde estimates that inference consumes hundreds of trillions of tokens daily, without specifying a measurement source or accounting scope. The central opportunity does not depend on that exact total: real interactions contain training signal that is currently being left unused.

Continual learning would make those interactions part of the improvement process. Malde places this after internet-scale pretraining and benchmark-driven training, and points to growing interest from Ilya Sutskever, Andrej Karpathy, Sholto Douglas, Demis Hassabis, and Satya Nadella, alongside a recent Dwarkesh video. The technical question is how to turn that interest into a learning algorithm that fits actual usage.

The first distinction is between where tasks come from and which policy generates the training trajectories. Production tasks can match real usage while their recorded answers still come from an older policy. Conversely, a current model can generate fresh, on-policy answers to benchmark tasks that poorly represent production. These are separate axes.

Two further requirements concern infrastructure and feedback. Generating several alternative trajectories requires environments that can reproduce the same starting conditions and support parallel actions. Building faithful copies is expensive, and imperfections introduce training bias. Then, after collecting a rich interaction, reducing its outcome to one scalar discards much of the information that could explain which individual decisions were useful.

Malde’s historical comparison starts with supervised fine-tuning, associated here with instruction tuning and GPT-3.5 and ChatGPT, then moves through preference training to GRPO. The table captures the training setups he contrasts; it is not a claim that every implementation of each method has these properties.

SFT makes the infrastructure simple and supplies token-level supervision, but learns from a fixed collection of demonstrations. Preference training brings real user tasks into the picture while introducing pairs and sequence-level judgments. GRPO prioritizes fresh on-policy rollouts; Malde credits that with powerful capability gains and fewer of the forgetting problems associated with earlier methods. Its costs are grouped rollouts, substantial environment infrastructure, and a return to whole-sequence rewards.

The desired combination is therefore specific: online tasks, on-policy samples, one rollout, and dense token-level feedback. Getting all four would let a system learn from a real interaction without first reconstructing the world and replaying the task several times.

GRPO starts with one task and produces several parallel outputs, labeled O1 through O4 in the explanation. Each output receives an end-state reward. The algorithm compares those rewards with the group mean and shifts the policy toward outputs that did better and away from outputs that did worse. The group supplies a relative measure of which behavior to reinforce.

That still leaves a narrow feedback channel. Imagine writing an essay and receiving only 87/100 from a teacher. The score gives a judgment, but it does not identify a weak transition, an unsupported claim, or a good paragraph worth preserving. Many attempts may be necessary to infer what the teacher wants. GRPO combines this sparse-feedback problem with the earlier task-distribution mismatch and the need to run multiple copies of an environment.

Distillation provides a different kind of supervision. After a brief reference to Mythos and restrictions on distillation into open-source models, Malde starts with the conventional setup: a student and a smarter, usually larger teacher receive the same fixed dataset. Training fits the student’s token probabilities to the teacher’s. Instead of learning only whether a complete answer was good, the student learns how the teacher distributes probability at each position.

On-policy distillation changes the source of the trajectory. Rather than scoring a fixed demonstration, the teacher scores the student’s own rollout. The student therefore receives guidance at the prefixes its current policy actually produces. But frontier training creates a problem: if the student is already the strongest available model, where does a smarter teacher come from?

On-policy self-distillation, or OPSD, changes the teacher’s information rather than its underlying model. Give the same model privileged information—a hint about the task or environment—and it can make better predictions. The student receives the original prompt; the teacher receives that prompt with the hint prepended. Both evaluate the student’s trajectory, and training moves the unhinted student toward the hinted teacher’s probabilities.

The demonstration asks the student to find a function’s derivative. Guidance about how to solve it, or even a golden solution, makes the teacher’s job easier. The learning procedure is:

1. Generate a trajectory from the student using the original derivative question.
2. Add the guidance to the teacher’s prompt.
3. Evaluate the student’s trajectory under that better-informed context.
4. Train the student without the hint to match the teacher’s predictions.

The teacher does not need to generate a separate successful trajectory for the student to copy. Its advantage comes from knowing more while evaluating what the student already did.

This setup can use an actual online task, preserve sampling from the student’s current policy, and learn from a single trajectory. Malde contrasts that with an illustrative group of eight rollouts: the extra information comes from the teacher’s predictions, so it does not require eight independent attempts in cloned environments. At each generated position, the teacher supplies a distribution rather than one end-of-sequence score.

The feedback also extends beyond the token the student happened to choose. Consider the Python for loop and range example on the slide. At a position where the student favors one token, the teacher may favor a different token from the vocabulary. Malde uses a vocabulary of roughly 65,000 tokens to illustrate the breadth of that signal. The blue regions identify alternatives that were not the student’s top sampled token but that the teacher wants to make more likely.

A full-distribution matching loss can be expressed directly in Python. Here, each row represents the next-token distribution at the same student-generated prefix; the teacher rows were computed with the extra hint. response_mask selects the generated positions to train on.

import torch
import torch.nn.functional as F


def distillation_loss(
    student_logits: torch.Tensor,
    teacher_logits: torch.Tensor,
    response_mask: torch.Tensor,
) -> torch.Tensor:
    # Logits: [batch, response_positions, vocabulary]
    student_log_probs = F.log_softmax(student_logits.float(), dim=-1)
    teacher_probs = F.softmax(teacher_logits.detach().float(), dim=-1)

    per_position_loss = -(teacher_probs * student_log_probs).sum(dim=-1)
    mask = response_mask.to(per_position_loss.dtype)
    return (per_position_loss * mask).sum() / mask.sum().clamp_min(1)

This makes the distributional idea explicit: every vocabulary entry contributes to the target at a trained position. It does not require every implementation to materialize the full vocabulary distribution. The slide labels its vocabulary grid SDPO, while Malde calls the mechanism OPSD; those labels alone do not establish identical implementation details. His central comparison is that distillation can move probability toward teacher-preferred alternatives, beyond reinforcing the successful tokens already sampled by the student.

For short-horizon tasks, Malde reports that GRPO saturated around Sonnet-level performance on LiveCodeBench, while OPSD reached higher performance in their experiments. He does not specify the Sonnet version, exact scores, or evaluation window. He attributes the improvement to the ability to shift the output distribution into behavior that the original policy did not readily produce.

Malde also reports sharply reduced token counts for solving difficult tasks with OPSD, without giving a numerical efficiency ratio. He contrasts this with the tendency of RL-trained models to improve by spending more tokens reasoning. The attraction is therefore not just better answers, but a route to better answers without continually lengthening the reasoning trace.

For accessible implementations, he points to OpenClaw-RL, describing chatbots that learn user behavior from unstructured interactions. These examples establish a useful starting point: small models and short tasks can benefit from this learning signal. They do not settle what happens when an agent must maintain coherent behavior across a long sequence of tool calls.

Trajectory has experimented with scaling the approach to 120-billion-, 500-billion-, and one-trillion-parameter models, according to Malde. He reports instability emerging around 120 billion parameters on trajectories with 50–100 tool calls. Evaluation accuracy becomes erratic, run-to-run variance rises, and tool-call errors appear. Models also start departing from the formatting behavior established during instruction tuning.

The first failure mode is the “but wait” problem. On a short task, the student’s trajectory and the hinted teacher’s preferred behavior may remain close. On a longer task, the student keeps following its own policy and can drift far from the route the teacher would take. Whenever the teacher evaluates another position, it tries to redirect the trajectory.

Repeated correction creates pressure toward tokens such as wait, maybe, then, and but. Malde describes a word cloud in which these words become more prominent as training proceeds. Eventually, the policy can settle into an unhelpful local solution dominated by hesitation: the student and teacher distributions have diverged, and fitting between them produces a model that keeps reconsidering rather than making coherent progress.

One response is to measure step-level divergence. In an illustrative trajectory with 100 tool calls, compare the student’s and teacher’s distributions as the trajectory develops. Use their KL divergence to determine a multiplicative weight on the token loss within each step. This differs from adding a conventional KL penalty to an objective: the divergence controls how much that step contributes to learning.

Independent step weights allow three different treatments:

• A trajectory that stays in distribution: Keep the weights at one and learn throughout.
• A trajectory that diverges heavily: In Malde’s example, modify W1 first, concentrating the correction on getting the initial step right before proceeding.
• A trajectory that departs and later recovers: Downweight the off-track portion, then retain a mild update where the trajectory becomes useful again.

The recovery case is particularly valuable. A poor middle section need not make every later step unusable, and a good beginning need not justify equally strong updates everywhere afterward. Malde presents this as one way to stabilize long-horizon tool calling, without specifying an exact divergence-to-weight formula.

RL has reward hacking; self-distillation has an analogous problem in hint leakage. The teacher receives information that the student will not have at deployment. If matching the teacher teaches behavior that depends on already knowing that information, the student may learn to skip the very reasoning or investigation required to obtain it.

The numerical example asks for the last three digits of an expression. A hint includes the solution steps and reveals that the answer is 000. The resulting reasoning starts from knowing 000 and works backward to fit that answer into the trace. This is a shortcut that would not be available when facing a new problem without the privileged hint. Hint design must therefore distinguish guidance that helps the model reason from information that lets it bypass reasoning.

A simple remedy is to have an LLM filter the hint. Suppose a user cannot log in. The raw hint says the user’s SSO token has expired. A filtered version instead directs the model to inspect the logs. The useful action is preserved, while the answer the agent should discover is removed from the teacher’s guidance. Malde says this works decently well, though it is only one approach to the problem.

Residual guidance tackles the same problem through the teacher signal. A hint often reveals its full meaning only after the model has seen all of it. Cutting it in half creates a partial teacher: the same model has some useful information, but less than the full teacher. Both contexts can then be used to compute token probabilities.

Malde describes taking a linear combination of the partial- and full-teacher signals to assess the strength of the hint and moderate the resulting update. In his illustrated case, the partial-hint distribution remains close to the student, while the full-hint distribution lies much farther away. Combining their signals offers a way to use the extra information without pulling the student entirely into behavior supported only by privileged knowledge. The talk gives the mechanism, but not the combination coefficients or a precise implementation formula.

With these remedies combined, Malde reports that a 120-billion-parameter model surpassed RL on Mercor’s APEX-Agents tasks, which he says often require more than 100 tool calls. The talk does not provide the exact score, RL baseline configuration, or effect size for that comparison. The result matters here because it extends the approach beyond short answers into the long trajectories where naive self-distillation became unstable.

The intended properties remain intact: tasks can come from online usage, the student can generate its own on-policy trajectory, one rollout can supply a training example, and the teacher can provide dense per-token supervision. Malde presents this as a substantial step forward, not as an algorithm that has solved continual learning end to end.

A weight update is only one part of a continually improving product. Malde’s broader forecast is software that becomes more capable each time it is used, which he sees as an important opportunity for 2026–2027. At Trajectory, the improvement targets include the model, its harness, and the behavior of the complete agent system.

The platform he describes follows a production loop:

1. Ingest agent traces from real usage.
2. Optimize the system through a self-serve learning process.
3. Deploy the resulting improvements.
4. Oversee the process through a control plane.

He describes a team with backgrounds at DeepMind, Meta Superintelligence, and OpenAI, alongside product builders, and names Harvey, Decagon, and Rogo as early-access companies. The platform ambition is to connect the learning mechanism to collection, deployment, and operational control.

The first audience question makes the deployment issue concrete: how long does training take, and can learning happen in real time? Malde describes the current research stage as pseudo-continual learning. Data arrives through usage, but updates still happen in offline batches, followed by re-uploading the model. He gives no training-duration measurement.

Moving from that cycle to immediate learning is both an infrastructure and an algorithmic problem. Malde asks the audience to imagine 10,000 rollouts running through a product and the work required to bring them together. Online collection alone does not guarantee that each trace remains on-policy by the time training consumes it. Trajectory is working on this coordination problem, but he explicitly says the end-to-end solution is still far away.

The final question asks whether different harnesses change the effectiveness of learning. Malde points to emerging work on harness improvement, then identifies a harder question: what happens when the model and the harness both change? A model update changes the behavior available to the harness; a harness update changes the contexts and action sequences the model encounters. Trajectory is exploring that interaction online with customers, but Malde offers no comparative harness results. The remaining research problem is to make the whole agent improve coherently while both sides of that interaction are moving.