Pretraining and Midtraining

Training stages as capability interventions

Pretraining establishes broad capabilities by fitting predictions across a corpus. Subsequent training inherits learned parameters rather than learning everything again from an initial, untrained state. [3]

Midtraining means capability-focused continuation of an existing model. The terminology varies: the GLM-4.5 report includes instruction data in midtraining. A stage’s actual data, targets and updates define its operation more precisely than its name. [4]

Supplying request context ordinarily changes the input computation without updating model parameters. Training changes learned state. A request’s cached intermediate representations are also distinct from newly learned weights. [7]

Behavioral training uses demonstrations or feedback to shape responses. Its procedures belong in Post-training and Alignment. [8]

Prediction objectives and transferable learning

An objective measures prediction error; optimization changes parameters to reduce it. Better fit on training examples does not establish performance on new cases. Generalization and independent assessment explains that boundary. [9]

Self-supervision constructs prediction targets from the data itself. [10]

For causal prediction, future tokens must remain hidden when predicting them. Cross-entropy penalizes insufficient probability on the recorded target; Shifted targets and parallel sequence training supplies the computational detail. [11]

Transfer reuses learned parameters for another task. Shared language or domain structure can make those parameters useful across tasks, but fitting one distribution does not establish competence on unfamiliar inputs. Factual correctness, instruction following and task success require their own evidence. [12][13]

Corpus composition and capability coverage

A corpus is the collection available for training. A data mixture combines corpus components using specified sampling weights. [14]

Useful coverage depends on the intended tasks. Correct but repetitive examples can offer little additional information; varied but irrelevant examples can consume the budget without supporting the desired capabilities. Difficulty, information density and diversity must be judged together. [15]

Accessible material may omit the intended users or workflows; coverage and sampling explains that mismatch. Permitted use also constrains selection. Replit’s code-training example explicitly combined source selection with licensing restrictions; collection and provenance covers the underlying records. [17]

Generated material is another identifiable corpus component, not automatically independent coverage. Recipes differ on its role; Synthetic Data covers generation and verification. [18]

From documents to scored examples

Tokenization converts text into vocabulary IDs; Tokenization defines that representation. Sequence construction then determines which content fits together. Truncation drops content beyond a limit; chunking divides it into examples. Either can separate related information before the model receives it. [11][19]

Packing places multiple examples into a training sequence. Padding adds placeholders to fit a batch’s shape. Removing padding can avoid wasted work, while packing needs an explicit policy for whether separate examples can exchange information. [20]

An attention mask controls information access. A loss mask controls scored positions. Document-boundary attention resets, position resets and loss exclusion are independent settings. [21]

A token can supply context without being a scored target. Consequently, processed-token counts and scored-target counts can differ. Ignored targets must also be excluded from a mean loss’s denominator, rather than contributing zero while still enlarging it. [22]

Sampling, token exposure and loss contribution

Stored corpus size, selection probability and processed-token share answer different questions. If selection samples whole documents without truncation, longer documents contribute more tokens per selection. A domain’s sampling probability therefore need not equal its token share. [14]

Pooling all valid token losses gives the longer document nine times as many terms. Averaging document means gives each document equal weight. Explicit document or domain weights add another choice; PyTorch’s class-weight argument instead weights target classes. None of these weights specifies proportional capability gains. [22]

Oversampling selects a component more often than a reference mixture would. An epoch is one pass through a corpus. Mixed streams can accumulate different effective passes through each component, so one global epoch count can obscure concentrated repetition. [23]

Data, learning-rate and sequence-length schedules

A curriculum deliberately changes ordering or emphasis during training. A fixed mixture holds sampling policy constant; staged reweighting changes it at selected boundaries. Curriculum need not mean easy examples followed by difficult ones. [24]

Equal aggregate exposure leaves ordering unspecified. Earlier examples affect the parameters that later examples update, potentially under a different learning rate. To investigate ordering, match per-source token totals, objective, compute and other schedules; to investigate mixture weights, retain the ordering policy. [27][28]

GLM-4.5 changes data, lengths and packing across stages. Its combined recipe does not isolate an ordering-only benefit. [4]

Model size, training tokens and compute

Parameter count measures learned numerical values. A processed-token budget measures training exposure. FLOPs, floating-point operations, measure arithmetic work. Empirical scaling laws fit relationships between these quantities and prediction loss within studied training regimes. [6]

Kaplan’s historical fit favored increasing non-embedding parameters faster than data as compute grew, stopping large models before convergence. Chinchilla instead favored approximately equal proportional growth of parameters and tokens. It varied training horizons and matched learning-rate decay to them, helping explain why allocation conclusions differed. [6][2]

A training-only optimum minimizes prediction loss under a training budget. A lifetime resource objective includes expected inference demand. Substantial serving demand can favor smaller models trained longer, but that conclusion depends on the demand estimate and loss model; extrapolating ordinary token ratios can overstate gains from extreme exposure. [29]

Replit applied this reasoning to latency-sensitive code completion: spend more training effort on a smaller model and amortize it across later requests. [30]

FLOPs alone do not determine elapsed time or price. Memory capacity and execution efficiency constrain feasible runs; Distributed Training and Inference covers distributing work and state when one device is insufficient. [29][31]

Finite data and repeated exposure

Repeated passes can improve learning without adding unique material, but their marginal value declines. Some extensively repeated runs develop worsening validation loss. A saturation formula does not necessarily model that deterioration, and observed epoch ranges do not establish a universal safe limit. [23]

Deduplication removes unintended repetition from the inventory; deliberate epochs control repeated exposure after selection. These are compatible choices. Overfitting explains why improving training fit can coexist with worsening held-out performance. [34][9]

Small pilots can test whether data choices remain promising before a larger commitment. Preserve the intended scarcity and repetition regime; a small run with abundant fresh data may answer a different question from a large run repeatedly sampling a scarce component. [35]

Checkpoint contents and training continuity

An inference artifact identifies a learned model. A resumable checkpoint must additionally identify the training process that produced its next update. Successful weight loading establishes only part of that contract. [5]

A warm start reuses learned weights while intentionally resetting or changing process state. It begins a different trajectory. Document such resets at stage boundaries; they are design choices rather than inherently incorrect recovery. [25]

Verify restored counters, learning rate, next sample identities and loss continuity. A wrong data position can repeat or skip exposure; matching weights cannot detect that error. [36][37]

Logical-state restoration does not guarantee bitwise-identical execution across hardware or software changes. Random-state control and deterministic algorithms address different sources of variation, and deterministic execution can carry a performance cost. [39]

Weights alone do not record processed examples. Keep the recipe and recorded artifact state together, with accessible corpus versions and execution configuration. Dataset releases and reproducible lineage explains snapshots and manifests. [38]

Domain adaptation and retained capabilities

Domain-adaptive pretraining, or DAPT, continues language-model training on domain material. Task-adaptive pretraining, or TAPT, uses the target task’s unlabeled training inputs; task answers are not its prediction targets. The foundational study uses masked-token prediction. An autoregressive adaptation instead predicts continuations. Supervised response training belongs in Post-training and Alignment, even when both use token cross-entropy. [40]

Continuation inherits earlier learning; training from scratch must establish it again. Comparable incremental continuation budgets therefore do not imply comparable total training investment. Domain-trained weights also remain a predictive model, not an authoritative, automatically refreshed database. [43][44]

Catastrophic forgetting is substantial degradation of earlier capabilities after new training. General-data share, domain emphasis, duration and learning-rate policy can affect adaptation and retention. Replay and regularization algorithms belong in Continual Learning; here, retention remains a separate acceptance requirement. [25]

A reported legal midtraining example retained a majority of broadly representative data and improved domain performance without reported general regression. That supports testing mixed exposure, not prescribing a universal ratio or promising retention. [45]

Training progress and capability evidence

Perplexity comparisons require compatible tokenization, data, available context and scoring conventions. This ordinary autoregressive definition does not directly apply to masked language models. [46]

Track fixed validation slices and task probes against consumed tokens and compute. Define metric denominators explicitly; changing mixture composition must not silently change the population behind a progress claim. [33]

For base-model comparisons, fix prompts, demonstrations, decoding and scoring. In-context examples change elicitation without updating weights; assistant-style polish is not the only evidence of capability. [3]

Memorization can coexist with poor application. In Jack Morris’s financial-report demonstration, reproducing training text did not yield a useful differently formatted response. A transfer probe must require more than copying familiar sentences. [44]

Contamination and assessment independence

Contamination occurs when assessment information enters training or training choices across a claimed independence boundary. Familiarity with the subject is not itself leakage. Independent data boundaries separates fitting, development and protected assessment. [47][48]

Exact matching finds literal overlap. Matching n-grams, contiguous token sequences, can detect shared passages. Such checks are useful screening, but their thresholds determine which similarities are noticed and which material is removed. [49]

Semantic investigations retrieve related candidates and inspect equivalence. Paraphrases, translations and transformed code can evade literal detectors. Negative scans do not establish absence; detected overlap alone does not quantify its score effect. [47]

Adaptive reuse creates another dependency: assessment scores can guide later training decisions even when individual cases remain hidden. Use development evidence for iteration, then independently sampled untouched cases for an ordinary holdout claim. Dates alone cannot establish independence from an undisclosed inherited corpus. [48]

Controlled comparisons and checkpoint selection

An ablation changes or removes a component to investigate its contribution. Controlled offline comparisons hold assessment conditions fixed. Training comparisons additionally need explicit data, objective, schedule and budget controls. [50]

For a data comparison, match starting state, objective, sequence length and update schedule. Report remaining differences. Equal token counts need not mean equal compute, especially when model size or example construction changes. Small differences require uncertainty estimates, not merely a winning score. [6]

Select checkpoints against predeclared target gains, named retention slices and resource limits. A blended score cannot substitute for each requirement. Use development assessments for selection and reserve protected assessment for the selected configuration’s final claim. [48]

Bind the conclusion to checkpoint identity, recipe, corpus versions and evaluation configuration. Preserve unresolved inherited exposure as a limit on the claim. [38]

Open questions

1. 01

   Isolating curriculum order remains difficult because recipes often change data shares, lengths and update policies together. Progress would be a replicated comparison with equal per-source exposure and matched remaining schedules, showing an ordering effect on specified downstream capabilities rather than only final training loss. [27][28]

2. 02

   Small-corpus adaptation still needs reliable guidance on retaining earlier capabilities. Large incoming-data continuation studies do not determine the right policy for a narrow engineering collection. Progress would require separate target and retention curves across matched mixtures and durations, including regressions hidden by averages. [25]

3. 03

   A complete recovery contract must connect saved process state with corpus identity and subsequent samples. Component restoration alone leaves integration failures possible. Progress would be an interruption fixture that verifies the next samples, counters and updates under a specified environment, while separately testing behavior after environment changes. [36][39]