Beyond Static Intelligence: Evaluating Continual Learning

Ask a language model to solve a task, discard its experience, and ask it to solve another. Repeat across benchmarks, aggregate the independent scores, and build a leaderboard. This familiar evaluation pattern measures what a model can do from a fresh start. But where does it measure how much the model learns? That is the opening question from Parth Asawa, a PhD student at UC Berkeley.

Resetting a system between tasks is like restarting your life from scratch every time you do something: nothing learned previously can help with what comes next. An evaluation of learning ability should instead plot performance as a function of prior experience. The interesting signal is not just a high score, but improvement across successive instances because the system has encountered earlier ones.

Asawa defines continual learning as sample-efficient online learning that is stable over long horizons. Each part matters. A system must learn from limited new experience, update while operating, and preserve useful information over time. Retention alone is insufficient: the system must also revise its priors when new evidence arrives.

In the conventional training-and-deployment pattern described here, data goes into an offline training process and a frozen checkpoint comes out. That checkpoint is deployed without further weight changes. There are several places a language-model system could instead carry learning forward:

These mechanisms preserve experience differently, but an evaluation should be able to test all of them. The immediate objective is not to declare which one solves continual learning; it is to measure the ability the field wants to improve.

Continual learning has a long history in neural networks, especially research on catastrophic forgetting. A common experiment trains a model on task distribution A, then B, then C. The question is whether it can learn each new distribution without losing performance on earlier ones. Language-model research has adopted similar sequential evaluations, alongside tests of whether a model can recall facts from a very long conversation.

These tests cover important abilities, but Asawa identifies three gaps. Sample efficiency is not necessarily a first-order requirement. Remembering a fact does not necessarily demonstrate learning that improves future performance. And broad pretraining makes it difficult to construct genuinely new tasks for frontier models: much of the internet and many economically valuable activities are already represented in their training distributions. A useful benchmark needs room for online experience to make a difference.

Why not simply chain existing benchmark questions together? Asawa uses a sequence of AIME problems as the counterexample. Putting independently designed questions into an order does not establish that solving an earlier question should help with a later one. A sequence is necessary for this evaluation, but sequence alone does not create a learning opportunity.

Three design requirements make improvement from experience meaningful:

• Headroom: The task must require online adaptation. If additional offline training can eliminate the need to learn during evaluation, it is a poor test of continual learning.
• Shared structure: Instances must share latent structure in their environment. Earlier experience should reveal something the system can exploit on later tasks.
• A learning signal: The environment must provide evidence from which the system can improve, such as scalar rewards, error messages, or textual feedback.

Together, these requirements establish both a reason to learn and a realistic opportunity to do so.

Each task defines a per-instance reward, with higher values representing better performance. Depending on the domain, that reward can reflect efficiency, profit, or closeness to a correct prediction. Summing those rewards is useful, but cumulative reward can confound learning ability with initial model strength.

The talk illustrates this with red and black systems, plus a gray baseline representing the black system with its state reset between instances. The black system improves when allowed to retain experience. The red system can nevertheless accumulate more reward simply because it starts stronger. Assuming it still has headroom, its high score does not establish that it benefits from previous tasks.

To isolate that benefit, evaluate each system twice:

1. Run it statefully. Preserve experience across instances, whether that means an updated policy, notes, or a growing context.
2. Run a stateless baseline. Reset the system between every instance so earlier experience cannot carry forward.
3. Subtract the paired rewards. The difference measures the benefit of retaining experience for that instance.

Here, t identifies an instance. Gain on the fifth task asks how much experience from the first four tasks improved performance on the fifth. This is the raw gain definition used in the talk; the published benchmark also reports normalized gain, scaling the difference by the available headroom above the stateless baseline.

Gain does not replace reward. A system that learns substantially may still perform worse than a stronger starting model, and either system may cost more to operate. Asawa therefore treats reward, gain, and cost as a Pareto comparison: examine the trade-offs rather than compressing initial capability, improvement, and expense into a single supposedly sufficient metric.

Database exploration makes the learning opportunity concrete. A data engineer working with an unfamiliar database must discover schemas, table relationships, data formats, and local idiosyncrasies. A human carries those discoveries into subsequent work. An agent whose memory is reset repeatedly pays the discovery cost again.

In this task, the agent receives a natural-language question and answers it through SQL queries. Early questions require exploration before the agent can produce the answer. For an illustrative question—“What is the total order amount for each customer?”—the work might include inspecting two tables before writing the join:

SELECT * FROM customers LIMIT 5;
SELECT * FROM orders LIMIT 5;

SELECT
    c.customer_id,
    SUM(o.amount) AS total_amount
FROM customers AS c
JOIN orders AS o
    ON o.customer_id = c.customer_id
GROUP BY c.customer_id;

The table and column names make the example concrete; the learning target is knowledge of the schema and its relationships. If that knowledge persists, a later question can require fewer exploratory queries. Asawa’s comparison with a tenth instance illustrates this expected efficiency improvement, rather than reporting a measured query count.

Now introduce a database migration. Columns disappear, differently named columns appear, and data formats change. Experience that once saved work can now produce a failed query. The agent must detect which assumptions are stale, discard those assumptions, and retain knowledge that still applies.

This is the stability–plasticity trade-off inside an operational task. Stability preserves useful experience; plasticity lets new evidence change the system’s beliefs. Simply accumulating more memory does not solve the problem if the system continues to trust obsolete information. Concept drift appears across the benchmark’s tasks to test this selective updating, not just retention in a fixed environment.

Continual Learning Bench applies this design across six domains:

Each domain supplies a sequence of instances with defined rewards. Domain experts assess whether the tasks are learnable and realistic, whether the changes resemble plausible concept drift, and whether the information an agent is expected to retain actually matters in that environment.

The initial release compared context management systems, with parametric approaches planned for later evaluation. Asawa reports that vanilla in-context learning topped aggregate reward and remained favorable on reward-versus-cost and gain-versus-cost Pareto frontiers in the initial evaluation. This baseline simply places prior experience in context without the more elaborate memory management used by other systems.

The horizon matters to interpreting that result. These were medium-horizon tasks, which may not have pushed in-context learning far enough to expose its limits. Asawa does not present it as the eventual solution to continual learning. The immediate surprise is that more expensive context management systems performed worse on the tested tasks; extending the sequences is part of the roadmap.

Observed failures help explain why preserving state is not enough. Asawa groups them around the two sides of the stability–plasticity trade-off: retaining and reusing information, and learning from new information. A system can fail either by losing a useful constraint or by refusing to revise an existing belief.

The sales prediction example shows a stability failure. The agent forecasts sales over roughly five years. It interprets feedback on one submission as evidence of overprediction and revises its forecast downward. It then interprets subsequent feedback as evidence that the lower prediction went too far. Instead of combining both observations and moving toward an intermediate forecast, it returns to the earlier overprediction. The useful constraint from the first attempt has effectively been lost. The feedback is a composite score that the agent interprets; it need not explicitly tell the agent which direction to move.

The cohort-studies example fails on plasticity. This system has an external notepad, yet dismisses relevant information with: “These seem to be cohort definitions from a different study schema that doesn't apply here.” Asawa states that the study schema did apply. The problem was not merely the absence of storage: the system failed to recognize applicable evidence and use it to update its priors.

These evaluation results do not settle which mechanism will ultimately work best. Asawa’s own research preference is toward parametric methods that consider architecture, data, and learning algorithms together. That is a proposal to design for continual learning, rather than assume an existing checkpoint is the right starting point.

The training stack has already changed substantially. A simplified earlier description was pretraining, supervised fine-tuning, and reinforcement learning from human feedback. More elaborate stacks add mid-training, reinforcement learning for different teacher models, and multi-teacher on-policy distillation. Much continual-learning work then begins after all of this: take the resulting frozen model and find a method that lets it adapt.

But those models were not necessarily designed to be continual learners. Asawa questions whether insisting on that starting point reflects a sunk-cost assumption. If continual learning were a first-order requirement, the pipeline might instead have one phase devoted to training a continual learner, followed by deployment in which environmental interaction and weight updates continue together. This is a research hypothesis about a different training design, not a demonstrated replacement for today’s stack.

The same willingness to question the training pipeline extends to how AI research is organized. Asawa places this work within his group’s broader effort to challenge established practices, including questions about open science, consolidation of power, safety, academia, and independent research institutions. He points to his writing and invites wider participation in reimagining what third-party institutions and open research could become.

Continual Learning Bench is a collaboration across Berkeley, Snorkel, and UW–Madison, supported by Snorkel AI’s Open Benchmarks Grants program and the Laude Institute’s Laude Slingshots program. Its next steps extend both the evaluation and the methods being tested:

• Broader and longer tasks: Add more domain-specific environments and longer sequences of experience.
• More learning systems: Evaluate additional open-source models and parametric approaches.
• Richer simulation: Develop environments beyond easily verifiable, deterministic domains.

The simulation work is especially important for testing adaptation where success cannot be reduced to a straightforward deterministic check.

User personalization is one such future setting: a system should learn an individual’s needs through interaction. Asawa doubts that current models can adequately simulate users for that evaluation, but sees building that capability as a worthwhile objective. The benchmark’s purpose remains precise: continual learning is not a point capability. Measuring how experience changes performance is how the field can optimize for systems that actually learn.