Distillation

Teacher supervision and the behavior to preserve

A teacher supplies supervision; a student learns from it. Distillation transfers behavior through training rather than copying parameters. The teacher can also be an ensemble that combines several models. [1]

Training changes learned parameters; inference applies the resulting model to inputs. A student can therefore use teacher-produced examples during training without calling the teacher during deployment. The broader distinction belongs to training, fitted state, and inference. [2][3]

Smaller size is optional. Born-Again Networks trained students with the teacher's architecture, demonstrating useful transfer without reducing architecture size. Reusing an architecture does not mean copying its learned weights. [4]

Behavioral fidelity means agreement with specified teacher behavior under stated conditions. Exact text, selected categories, probability distributions, and meaning are different comparison targets. Correctness requires a separate standard: faithfully repeating a teacher's error remains an error. [5][6]

An acceptance criterion makes a requirement observable. Numerical compression is a separate choice: quantization changes numerical representation, while distillation changes learned behavior through teacher supervision. They can be combined, but neither guarantees the other's benefits. [6][9]

Supervision interfaces and compatibility

A hard target selects a label or response. A soft target supplies probabilities over alternatives. Logits are scores before probability normalization; a language model produces one per vocabulary entry. Vocabulary readout explains how these scores become next-token probabilities. [10][11]

Token IDs obtain meaning from a vocabulary mapping. Equal IDs or vocabulary sizes do not establish compatibility. Different tokenizers also divide text differently, so prediction positions can diverge. Direct coordinatewise probability matching needs aligned outcomes and positions; generated text can instead be encoded with the student's tokenizer. [10]

Suppose a teacher reports probabilities 0.60 and 0.25 for two alternatives. Their sum is 0.85: 0.15 remains elsewhere. Renormalizing gives approximately 0.706 and 0.294, a distribution conditional on the retained alternatives. That changes the target. A service interface must specify which information and approximation it provides. [13]

An intermediate representation is a vector computed inside a model. FitNets trains a regressor from selected student features to a teacher layer's dimensions using squared error, followed by separate output distillation. This supports thinner, deeper students, but overly deep guided layers can overconstrain learning. [14]

Soft targets, temperature, and student updates

Alternative probabilities reveal distinctions discarded by selecting a winner. Temperature reshapes those probabilities: larger values soften the distribution. These example teachers both select Billing but disagree about the closest alternative. [1]

The $T^2$ factor approximately compensates temperature-dependent gradient scaling. Without independent labels, omit their term. Temperature and weighting remain empirical choices. [1]

In a fixed-teacher experiment, teacher targets remain constant while the optimizer updates student parameters. The label and teacher terms can disagree, so lowering their weighted sum need not improve either term separately. Losses specify the training pressure; gradients determine parameter updates. [2]

Softening does not establish calibration: whether predictions assigned a given confidence are correct at that frequency. Calibration and accuracy are different properties. A distribution can become less concentrated without becoming a better account of correctness. [15]

Selected responses and rationale targets

Sequence-level distillation trains on selected teacher responses. Selection matters: Kim and Rush used beam search, which keeps several candidate continuations, to produce translation targets. Their reference-based variant selected the candidate most similar to an existing translation. The same teacher can therefore supply different supervision under different selection rules. [12]

A prefix is the text available before a prediction. Teacher forcing conditions predictions on recorded predecessors instead of the model's generated choices. Shifted-target training uses each position to predict the following token. Here, each displayed symbol is one token in a deliberately small vocabulary. [16]

A loss mask selects prediction targets that contribute directly to training. Excluding prompt targets does not remove the prompt's conditioning role. Supervised fine-tuning, covered in Post-training and Alignment, increases the probability of demonstrated next tokens. It rewards the recorded response, not truth or execution success directly. [17]

A rationale is a written explanation supplied as another target. Distilling Step-by-Step trained pretrained T5 students on separate label-prediction and rationale-generation tasks, selected by task prefixes. Label inference did not require a teacher rationale. Some settings combined human labels with teacher rationales; the improvements were task-specific. [18]

Rationale usefulness and faithfulness require different evidence. An explanation may help a student without accurately describing how the teacher reached its answer. Hint-intervention experiments found that reasoning models sometimes omitted answer influences from their explanations. Generated reasoning is observable supervision, not direct access to hidden computation. [19]

The teacher's training method also need not become the student's method. DeepSeek-R1's smaller distills received supervised training on curated outputs. Using outputs from a reinforcement-learned teacher did not mean those students repeated its reward-based training procedure. [3]

Teacher feedback on student-generated prefixes

Token-distribution distillation matches next-token distributions at a prefix. Recorded prefixes can differ from those reached during student generation. On-policy distillation obtains teacher feedback at student-visited prefixes, including mistaken continuations. The student's sampled token determines visited context; it does not become ground truth. [20]

A rollout is a generated sequence of choices and resulting context. Generalized Knowledge Distillation mixes recorded examples with student rollouts. Its implementation differentiates the distribution-matching loss, without backpropagating through discrete sampling. This changes where supervision is supplied, not whether the teacher is correct. [20]

Scoring an existing sequence requires access to probabilities for its supplied positions. An API that generates a one-token response does not necessarily return those scores. Rollout collection can resemble reinforcement learning while supervision comes from teacher likelihoods rather than an environment reward. [21][22]

Transfer-data coverage and selection

The transfer dataset contains situations in which the student receives teacher supervision. Its inputs need not already have human labels: a teacher can generate targets for collected inputs. Input fitness and target correctness remain separate decisions. [23]

Coverage concerns which intended cases are accessible and represented. Selection bias arises when choosing records systematically changes that representation. More examples from a narrow source do not recover excluded conditions. Coverage and sampling provide the general framework. [24]

Easy-only selection can omit boundaries; disagreement-only selection can omit routine behavior and shared mistakes. Confidence filtering can exclude difficult conditions. Inspect rejected records as well as retained ones: C4's filtering analysis found benign material removed disproportionately. That demonstrates a selection hazard, not an optimal distillation sampling recipe. [26]

Generated scenarios can fill an initial collection gap. Rechat used an LLM to simulate user requests across supported workflows before substantial usage existed. Such synthetic data supplies possible inputs, not independent correctness labels or proof of deployment coverage. [27]

Keep transfer construction, development selection, and protected assessment separate. A random split within a restricted collection does not make it representative; repeatedly consulting final results converts assessment into development. Independent data boundaries apply equally to teacher-generated records. [28][25]

Target validity, privileged information, and lineage

Privileged information is available to the teacher during transfer but absent from ordinary student inputs. Richer teacher context can support useful learning: generalized distillation explicitly permits different teacher and student representations. However, a case-specific conclusion may be impossible to recover from what the student receives. [7]

Hinted self-distillation uses the same model as a better-informed teacher. A hint-enriched prompt supplies teacher probabilities while the student receives the ordinary prompt. Teacher scores must correspond to the original student prediction positions; adding context does not remove the alignment requirement. [29][30]

For a login ticket, a teacher might know that a single-sign-on token expired. If the deployed student can inspect logs, guidance to inspect them teaches an available procedure. Revealing the hidden diagnosis can instead teach an answer-first shortcut. Removing the hint from the student's prompt does not undo that target dependence. [31]

Target checks should follow the contract: parse the output, check categories against independent labels, and review whether justifications follow from permitted evidence. Evaluation workflows can route promising examples and failures through human review before updating training data. Label validity remains distinct from passing a format check. [32]

Teacher confidence cannot replace those checks. Token likelihood concerns a continuation; answer correctness is a different event. Even a generated confidence statement needs validation against observed outcomes before it can be treated as a probability. [33]

Record derivations across dataset releases, including excluded candidates needed to inspect selection. Provenance describes how a target arose; it does not certify its truth or authorize its use. Vendor and model-processing obligations govern the separate permitted-use boundary. [34][35]

Student capacity and task specialization

Representational capacity concerns distinctions a model can express. Architecture constrains that space; initialization and optimization affect the solution training reaches. Poor agreement alone does not identify insufficient capacity: students have failed to match teachers even with identical architectures capable of representing the teacher solution. [5]

A more accurate teacher need not produce a better student. Cho and colleagues observed deteriorating student accuracy with some larger teachers in their classification experiments. Transfer-fit diagnostics supported a capacity-mismatch interpretation there; increasing temperature did not resolve it. That finding does not exclude optimization or data limitations elsewhere. [36]

Task specialization changes what must fit. Predicting a bounded category set and justification is a narrower target than preserving every teacher capability. Distilling Step-by-Step compared student sizes, data quantities, and label-only versus rationale supervision. Such controls reveal task-specific gains without establishing general teacher equivalence. [18]

A student may also outperform its teacher on a bounded evaluation. Born-Again Networks demonstrated improvements with individual students; ensembles of successive generations were a separate configuration. Neither result implies universal superiority or that architecture reuse alone causes improvement. [4]

Available inputs impose another limit. If two cases look identical to the student but require different outputs because of hidden teacher information, increasing parameter count cannot reveal that missing case-specific fact. Change the available evidence or the task requirement before treating every failure as a size problem. [7]

Controlled transfer experiments and failure diagnosis

A checkpoint saves learned model state. Bind each candidate checkpoint to its initial student, target-dataset version, teacher configuration, loss weights, and training conditions. An ablation changes or removes a component to test its contribution. Controlled comparisons require identifiable alternatives, not merely two differently named models. [34][12]

Compare the initial student, teacher-supervised training, and independent-label training when those labels exist. Keep the student starting point and relevant data and training conditions explicit. Label-only and rationale-supervised baselines help distinguish improvement from additional supervision versus improvement already available through ordinary task training. [18]

Target defects and prefix mismatch can coexist. If repairs and student-prefix supervision are introduced together, an improvement cannot identify their separate contributions. With compatible teacher scoring available, cross original versus repaired supervision with recorded versus student-generated prefixes. Keep initialization and workload coverage fixed across the four arms. [37][8][28]

Use development evidence for repairs and selection. Freeze the selected configuration before protected assessment. Further changes after inspecting final results require a new assessment boundary. Failure analysis preserves competing explanations instead of treating every lower loss as a solved cause. [28]

Fidelity and independent task correctness

Top-label agreement measures shared selections; predictive-distribution comparison measures probability differences. Neither establishes correctness. In self-distillation experiments, greater agreement sometimes accompanied lower accuracy. Report the fidelity dimension promised by the contract alongside an independent task score. [5]

For ticket responses, score category correctness, supported justification, required format, and escalation behavior separately. Escalation recall is correctly escalated tickets divided by all independently labeled escalation-required tickets. Omitting difficult tickets from evaluation hides missed escalations; abstaining does not remove their reference requirements. [38]

A slice is a meaningful subgroup, such as language, format, or escalation type. Compare teacher, student, and baselines on matched cases in important slices. Whole responses must satisfy the task contract: favorable recorded-prefix loss can coexist with free-running failures. Uncertainty in differences matters especially when rare-case samples are small. [25][17]

The teacher should not be the sole judge of its transfer. Model judges can exhibit position, verbosity, and self-favoring biases. Validate judgments against independent human labels, inspect disagreements, and check these biases before using scores for selection. Automated judge reliability covers the broader procedure. [39]

When confidence controls automatic routing, evaluate whether stated probabilities match observed correctness frequencies. Distribution shift means the evaluated input distribution changes. Calibration established on familiar data can deteriorate under shift, so thresholds need deployment-relevant checks. Calibration and selective automation separates this requirement from accuracy and agreement. [40]

Deployment constraints and reassessment

Choose an operating point by declaring required quality and latency, then minimizing cost among configurations that satisfy them. Interactive completion and asynchronous work impose different requirements. Parameter count alone does not identify the best complete system; LLM Inference covers the serving mechanisms. [41]

A 95th-percentile latency is a duration at or below which approximately 95% of measured requests finish. Completed tasks per second differs from tokens per second: longer answers can increase token work without completing more tasks. Define measurement boundaries through task metrics and usage accounting, including retries and unresolved outcomes. [41][42]

Amortization spreads an initial investment over subsequent use. The cost ledger must include teacher-data generation, curation, student training, and validation, followed by recurring student inference, retries, and escalation. The video-judge team chose a smaller model for lower evaluation latency, accepting lower quality than a tested larger model. [43]

For a fixed operating horizon, compare total costs under the same completed-task definition and quality gates. Lower recurring cost only recovers the transfer investment after sufficient use; more escalation can erase that advantage. AI Cost and Performance Engineering develops the broader accounting. [43]

Instacart's semantic role labeling extracts product, brand, and attribute concepts from search queries. Its teacher pipeline populated a frequent-query cache and supplied data for a Llama-3-8B student serving uncached queries. The report describes similar F1, a combined precision–recall score, alongside different precision and recall. Similar aggregate quality therefore concealed different error balances. [44]

Meeting the latency target also required adapter merging—folding task-specific learned adjustments into the model—and a hardware upgrade. FP8 quantization reduced latency further but lowered recall, so the team deployed without it. The result belongs to a complete configuration, not distillation alone; undisclosed transfer-data composition limits conclusions about long-tail training coverage. [44]

Release requires the declared behavioral and resource limits before selecting the lowest-cost candidate. Reassess when workload, teacher, targets, student, or serving configuration changes: prior evidence describes its tested conditions. Evaluation records and reassessment connect each decision to the configuration and outcomes that support it. [40][44]

Open questions

1. 01

   Partial-distribution transfer remains sensitive to missing alternatives. Top-k interfaces reduce available supervision, but omitted mass can alter the learned target. Progress requires comparisons that vary truncation while holding teacher, student, and inputs fixed, measuring both distribution fidelity and independent correctness. [13]

2. 02

   Useful rationale supervision need not be faithful explanation. Separating procedural learning from imitation of hidden answer cues remains difficult. Progress would show gains on unseen problems while interventions on irrelevant hints no longer change answers through unacknowledged shortcuts. [19]

3. 03

   Hint filtering still lacks a general guarantee against inaccessible-information shortcuts. Procedural guidance may help, yet deciding what a student could discover depends on its deployed tools and observations. Progress requires leakage tests that withhold diagnoses while preserving legitimate discovery paths. [31][46]