Continual Learning

Learning across sequential experiences

Continual learning acquires capabilities from sequential training experiences while preserving specified earlier capabilities. Retention means continued demonstrated performance, not unchanged parameters. Each update inherits previous learning, but earlier examples may be unavailable. Data order and opportunities to revisit history therefore matter. [3]

Training changes learned numerical parameters; ordinary inference applies them. Supplying saved records can change answers without changing parameters. Training, fitted state, and inference explains this distinction; Agent Memory covers retained records. CL-Bench’s initial context-based experiments did not evaluate parameter updates, so their results cannot establish parametric retention. [4]

Experiences can recur or change gradually; clean task boundaries are optional. Starting from a pretrained model also brings inherited capabilities, discussed in Domain adaptation and retained capabilities. [3]

Online learning updates parameters from arriving observations or batches. Training can run separately from serving; online does not mean immediate deployment or guaranteed retention. [5]

Capabilities to acquire, preserve, and revise

An adaptation target needs accompanying retention requirements. Specify the task, affected population, target meaning, and acceptable degradation. Critical capabilities need their own limits: an average improvement can coexist with an unacceptable loss. These limits are application requirements, not universal constants. [6]

Distribution shift changes the patterns of inputs or outcomes. Different document formats need not change correct labels; revised labeling rules can. A discrepancy warrants investigation, not automatic retraining. Data change and verified recovery explains that distinction. [7]

Lineage connects records to their origins and transformations. Preserve enough identity to determine which examples and targets an update used; Collection, identity and permitted use explains the underlying records. [9]

Acquisition and specificity can diverge. In a narrow vendor experiment, continued training strengthened an unusual food-related spelling behavior while its restriction to relevant questions could worsen. Learning the requested behavior therefore needed a separate test from avoiding it elsewhere. [8]

Interference and catastrophic forgetting

Catastrophic forgetting is severe degradation of previously learned performance following later training. Shared parameters support multiple behaviors, so improving a new objective can disrupt earlier ones. The term describes severity without supplying a universal threshold. A small parameter change is not, by itself, evidence that capability survived. [10]

The stability–plasticity dilemma balances preserving useful learning against adapting. Excessive protection can also prevent acquisition; shared structure sometimes permits improvement on both. [2]

Acquisition and retention across checkpoints

A cohort is a specified group of assessment cases. Record $R_{i,j}$, performance on cohort $j$ after learning stage $i$. Include the starting checkpoint, acquisition scores, and later reassessments. Each column keeps a capability visible across updates. [11]

Keep replay training examples, development cases, and protected tests separate. Repeatedly selecting changes against protected results turns those cases into development evidence. Fixed historical cohorts measure retention; separately identified current-condition cohorts measure present usefulness. Coverage and independent assessment explains why neither population substitutes for the other. [12]

Prequential evaluation, or test-then-train, predicts and scores each arriving example before learning from it. Its accumulated score combines predictions from successive model states. It does not retest the current checkpoint on historical cases. MOA’s evaluation descriptions distinguish this protocol from periodic held-out assessment. [13]

Historical references and forgetting measurements

Check the preceding accepted checkpoint and a persistent historical reference. Three successive one-point losses total three points; a one-point-per-update allowance would conceal that accumulated loss. [2]

Attribution requires comparable cases, targets, prompts, decoding, tools, and evaluator versions. Otherwise, a changed score has competing explanations. Evidence of adaptation gains and Controlled offline comparisons describe these controls; Uncertainty in scores and differences explains why small differences may remain inconclusive. [12]

When revising a benchmark, explicitly compare relevant checkpoints under both versions on overlapping cases. This comparison design helps separate model change from assessment change; it does not make incompatible scores interchangeable. [12]

Replay as an earlier-training constraint

Replay, or rehearsal, includes earlier training examples in later updates. A replay buffer is their bounded collection. Mixing current and historical examples lets both influence learning; held-out testing measures behavior without supplying training examples. [14]

A batch containing eight current and two replay examples has a 20% replay example share. Equal per-example weighting gives that share of the averaged loss terms, not necessarily gradient influence. Token-weighted training needs another denominator; Sampling, token exposure and loss contribution distinguishes these quantities. [15]

Buffer coverage and historical validity

Reservoir sampling gives each of $n\geq M$ arrivals inclusion probability $M/n$ in a capacity-$M$ buffer. Uniform historical sampling does not guarantee class coverage. [14]

def admit(buf, item, n, M, rng):
    # n includes this arrival; M > 0.
    if len(buf) < M:
        buf.append(item)
    else:
        j = rng.randint(1, n)
        if j <= M:
            buf[j - 1] = item

# Sampling for training is a separate decision.
replay_batch = rng.sample(buf, min(batch_size, len(buf)))

Storage membership and training exposure differ. Repeated tiny-buffer training can overfit, but harm is not inevitable. [14]

For the classifier, protect valid rare-class examples separately from superseded labels. Replaying an obsolete target teaches the wrong current behavior; correction or exclusion follows the declared labeling rules. [7]

Replay also remains subject to permitted use and deletion requirements. Lifecycle fulfillment across derivatives covers propagation beyond the original record; a training buffer is another retained copy. [9]

Importance-weighted parameter protection

Regularization adds a training penalty. Elastic Weight Consolidation, or EWC, discourages departure from reference parameters, weighted by their estimated importance to earlier learning. [1]

With reference $(0,0)$, $F=(4,1)$, and $\lambda=1$, displacements $(0.5,0)$ and $(0,0.5)$ incur penalties 0.5 and 0.125. Equal movement receives unequal resistance. [1]

Long sequences require an explicit reference-refresh rule. Adding penalties around successive optima can double-count earlier evidence because later optima already incorporate it. A recursive formulation instead accumulates importance around the latest reference. This is a correction under an approximation, not a guarantee that arbitrary task requirements remain compatible. [16]

Restricting adaptation to added parameters is another possible constraint, but it does not establish retention either. The amount and kind of learnable change still require behavioral assessment; constrained capacity can also limit acquisition. [10][17]

Preservation through reference outputs

Output regularization discourages disagreement with earlier predictions on selected inputs. Learning without Forgetting records old-task probabilities before adaptation, then combines agreement with those targets and new-task supervision. Parameters can change; preservation acts on outputs. Distillation means learning from teacher-provided targets; Soft targets, temperature, and student updates explains the underlying training mechanism. [18]

Specify the reference checkpoint, selected inputs, aligned old-task classes, and preservation weight. Increasing that weight favors agreement over new supervision. New-task inputs may omit a rare historical class; replay can supply complementary coverage. On a revised-label slice, preserving old predictions may directly oppose the corrected targets. [18]

Fidelity is agreement with a reference; correctness is agreement with valid task targets. Either can improve while the other worsens. Independent assessment remains necessary even when the candidate closely matches its teacher. [19]

Reference refresh changes future constraints. Avalanche’s versioned implementation copies the model after each experience and records learned classes. [20]

Selective imitation can target relevant behavior rather than incidental style. Denton describes relevance-masked self-distillation, where a judge selects teacher tokens to learn from. That selection introduces its own reliability question: relevance judgments determine which differences receive training pressure. [21]

Adaptation cadence and training intensity

Cadence determines when updates start; intensity determines work within each update. Frequent starts cannot eliminate training delay. While training runs, the latest available model can continue predicting; a newer version becomes useful only when ready. [5]

Labels may arrive after predictions. Evaluation must preserve that chronology: predict first, score when the target becomes available, and learn only afterward. Granting future supervision early overstates adaptation. River’s progressive evaluator explicitly models prediction and learning events with label delays. [22]

Learning rate, update budget, ordering, replay exposure, and preservation strength jointly shape adaptation. Continuation experiments found rewarming could accelerate learning while increasing forgetting. These are regime-specific tradeoffs, not default settings; Data, learning-rate and sequence-length schedules supplies the scheduling foundations. [15]

Checkpoint acceptance under retention requirements

The lowest newest-task loss need not identify an acceptable checkpoint. Apply acquisition targets and separate retention limits, including critical cohorts. A release gate combines those declared requirements with evidence; Release and revision decisions explains the decision boundary. [6]

Compare against a frozen model and straightforward sequential training under stated data and compute budgets. Pooled-history training is another reference when feasible, not a guaranteed upper bound. [15][2]

Workflow compliance and task competence may need separate scores. Denton’s coding experiment distinguished calling the submission tool, passing tests regardless of submission, and achieving both. Learning to finish the workflow would otherwise conceal a loss in problem-solving ability. [23]

The complete learning-state version

An inference artifact supplies the model; resumable learning needs the state determining its next update. Checkpoint contents and training continuity establishes this distinction. A candidate record should bind the following components consistently. [24]

Stage method state with candidate weights. A rejected candidate can otherwise leave a refreshed teacher governing the next update. Consistent acceptance or rejection of the bundle is an engineering requirement, not an automatic property of the documented teacher lifecycle. [20]

Intentionally resetting an optimizer is a different intervention from resuming it. Record that choice rather than describing both as identical continuation. [15]

A checkpoint may reconstruct datasets from references instead of embedding their contents. Preserve the corresponding data versions; Dataset releases and reproducible lineage explains manifests and snapshots. [24]

Logical restoration does not guarantee bitwise-identical execution. Random-generator control and deterministic algorithms are separate measures, and framework releases or hardware changes can alter results even with identical seeds. [26]

Rejection, rollback, and verified recovery

Rejection withholds a candidate before promotion. Serving rollback switches production to an earlier model version after promotion. Neither action alone specifies restoration of the state governing future training. Shadow assessment can evaluate a candidate while the earlier model continues supplying outputs. [27]

After interruption, resume from a consistent supported boundary or declare a restart. Avalanche’s inspected checkpoint functions support experience-boundary recovery, not arbitrary mid-update resumption. [24]

For delayed regression, a recovery design should pause affected learning, identify dependent descendants, restore trusted state, and verify required capabilities before resuming. [28]

Restored weights do not undo earlier outputs or external effects. Those need separate handling; Snapshots, rollback and external effects explains the boundary. [27]

Recovery should change the next experiment: repair the responsible data, replay coverage, schedule, or preservation constraint. Repeating a harmful update unchanged does not resolve its cause. [29]

Open questions

1. 01

   Long-horizon evidence remains limited for repeatedly updated language models. Short evaluations can miss accumulated interference and changing environments. Progress would include sustained parameter-update sequences with protected inherited-capability cohorts, explicit resource budgets, and reported failures—not only final aggregate gains. [33][34]

2. 02

   Preservation must distinguish valid rare behavior from obsolete targets. Sparse inputs can leave the former unconstrained while reference agreement reinforces the latter. Progress requires independently testing both slices after selective preservation changes. [18]

3. 03

   Complete learning-state recovery needs end-to-end evidence. Serialized strategy state can omit attributes or depend on external data, making successful weight loading insufficient. Progress would demonstrate rejected-update and interruption recovery, including the next update, under documented checkpoint boundaries and preserved data versions. [24][26]