Post-training and Alignment

Starting checkpoints and behavioral targets

A checkpoint saves fitted model state, including parameter tensors and potentially registered buffers. Restoring it requires the corresponding model structure. Resuming training usually also requires optimizer state and other records; a weights file alone does not reproduce an entire run. The PyTorch checkpoint guide distinguishes these uses. [1]

A policy assigns probabilities to possible outputs given context. For an autoregressive language model, it predicts the next token from the prompt and preceding tokens. Post-training changes learned parameters; ordinary prompting changes the input to fixed parameters. Training versus inference explains that boundary, while Pretraining and Midtraining covers the starting model's production. [2]

Alignment means adapting behavior toward stated human goals and constraints. A specification can guide both training feedback and separate assessment cases. Its requirements must identify whose needs govern and how conflicts are resolved; approval under one rubric cannot stand for universal human preference. Intended outcomes and evaluation boundaries supplies the corresponding assessment framework. [3][4]

For a programming assistant, stipulate three requirements: solve supported legitimate tasks, acknowledge insufficient evidence, and refuse specified harmful requests while assisting safe ones. These are teaching requirements, not reported experimental results. Demonstrations, comparisons, and rewards can each encode parts of them. [3][5][6]

Supervised fine-tuning from desired responses

Supervised fine-tuning, or SFT, trains an existing model on supplied input–target examples. With teacher forcing, each prediction conditions on the demonstrated prefix, including its mistakes. A loss mask selects targets for direct supervision; an attention mask controls information visibility. Excluding prompt targets from loss does not remove the prompt from context. [7]

Completion-only supervision includes the completion; assistant-only supervision selects assistant turns; full-sequence supervision can include prompt targets. Padding is excluded. The chosen mask, conversation template, and dataset must agree. [7]

Assume a short record has five token units: USER, ping, ASSISTANT, OK, END. These are stipulated boundaries, not a tokenizer claim. With response-only supervision, the prompt remains available while only OK and END contribute loss. The probabilities below are example values. [8]

Using natural logarithms, the included loss is $-\log 0.5-\log 0.25\approx2.079$; averaging over two targets gives approximately 1.040. Backpropagation differentiates that scalar with respect to parameters. The optimizer uses those gradients to update the checkpoint. A gradient describes local loss sensitivity, not a symbolic instruction to repair one behavior; see Gradients and parameter updates. [9]

Aggregation changes effective weighting. Summing token losses gives longer responses more terms; averaging within each example before averaging examples changes that balance. Repeating or sampling an example more often increases its contribution. These choices determine which demonstrated patterns dominate, alongside explicit weights and task coverage. [9]

SFT is useful for specified output mappings such as extraction and formatting. Its target remains the supplied answer: flawed facts or stylistic shortcuts can receive the same imitation signal. Generated demonstrations are still training inputs. Their construction belongs to Synthetic Data; improvement requires separate assessment. [10][11]

Preference pairs and judgment meaning

A preference pair contains two responses to the same prompt and a comparative judgment under a rubric, the criteria for judging them. The selected response is called chosen; the other is rejected. These labels express relative preference, not absolute adequacy. [4]

A constructed pair answers whether passing supplied tests establishes correctness on every input. Response A states that untested inputs remain unresolved. Response B enthusiastically congratulates the programmer and asserts universal correctness. Hold these responses fixed while changing the stipulated judgment criterion. [12][13]

Candidate collection determines available contrasts. Preserve prompt coverage, generating checkpoints, sampling settings, judge identity or model version, and rubric version. Comparisons between nearly identical answers teach different distinctions from comparisons that isolate a factual error. The summarization research operationalized selected labelers' judgments under researcher-defined requirements, rather than discovering a population-independent preference. [4]

A tie, an uncertain judgment, and disagreement between judges are different records. A binary chosen/rejected objective requires an explicit policy for including or adjudicating them; silently inventing a winner changes the target. Preserve the underlying judgments. Label meaning and Annotation, disagreement and correction explain the broader data responsibilities. [14]

Sampling from the model being adapted can expose its current failures. One small-model recipe ranked looping and non-looping candidates to construct preference pairs. That approach needs both a useful alternative and a judge that actually penalizes repetition. [15]

Learned rewards from comparisons

A reward model assigns a scalar score $r(x,y)$ to prompt $x$ and response $y$. Bradley–Terry modeling learns preferences through score differences. [16]

Adding a common offset leaves comparisons unchanged. The resulting probability concerns preference, not factual correctness. Moreover, a majority cycle—A preferred to B, B to C, and C to A—cannot fit a strict scalar ordering. Disagreement alone need not create such a cycle. [14]

A scorer can learn shortcuts in its labels. In a video-judge example, manufactured negatives encouraged detecting visual artifacts instead of the intended quality dimensions. Responses generated after optimization may expose further weaknesses. Automated judge reliability covers judging validity; using a scorer as a reward adds pressure to exploit its errors. [17][18]

Executable rewards and checker contracts

A verifier applies an explicit checking procedure. A test oracle decides whether observed behavior meets its criterion. A verifiable reward converts such checking into training feedback. Automatic computation does not expand the checked property; Oracles, rubrics and human judgments explains this distinction. [19]

Checkers can also reject valid alternatives. Tests that demand an unspecified variable name or inspect an internal function constrain implementation beyond the requested behavior. Such false negatives supply misleading penalties, just as missing assertions can supply misleading rewards. [12]

A successful formal proof check needs a dependency audit: imported results can depend on incomplete proofs or custom axioms. Lean's validation contract therefore matters when defining allowed assumptions. Checking an intermediate formal claim is possible under the same contract; that possibility alone establishes no effective intermediate-reward training procedure. [20]

Outcome and intermediate feedback

Outcome supervision assesses a completed result. Process supervision assesses intermediate steps, requiring a definition of a step and a defensible judgment about it. Feedback location and feedback source are separate choices. [21]

Sparse reward supplies feedback at few points, often only completion. Credit assignment connects that delayed outcome to earlier choices. A return aggregates rewards; it does not identify causal responsibility. A correct terminal answer may follow mistakes, while useful intermediate work may occur in a failed attempt. [22]

In one mathematical process-supervision study, human step labels trained a learned scorer, while final-answer checks supplied outcome labels. Multiplying predicted step-correctness probabilities ranked solutions. The generator stayed fixed during best-of-N evaluation: those results concern selecting generated solutions, not reinforcement learning of the generator. [21]

An approved prefix can still lead to a wrong answer. Conversely, the illustrated arithmetic trace reaches the correct answer through a defective intermediate equation. Assessing written steps also differs from establishing explanation faithfulness: behavioral interventions can reveal influences on an answer that its explanation omits. [23]

Long agent trajectories sharpen the problem. Assigning the same final return to every model call treats helpful, needless, and mistaken actions alike. A transition-level record permits more selective assignment, but does not itself discover which action caused success. Changing an action in a counterfactual test also requires recomputing downstream observations. [24][25]

Reinforcement learning for response policies

For language-model reinforcement learning, tokens are actions and the prompt plus generated prefix supplies their context. A rollout is a sampled continuation, extended to actions and observations for interactive tasks. Sampling, scoring, and updating weights produces a new policy for subsequent sampling. Actions and rewards provides the underlying learning framework. [26]

An advantage compares return with a baseline. Positive advantage favors sampled choices; negative advantage suppresses them. Policy gradients differentiate output log-probabilities. When the environment and scorer do not depend on policy parameters, training need not differentiate through them. [27]

for _ in range(rounds):
    samples = sample(policy, prompts)
    rewards = score(samples).detach()
    advantage = (rewards - baseline).detach()
    logp = policy.token_logprobs(samples)
    loss = -(advantage[:, None] * logp * samples.mask).sum(-1).mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

For example rewards $[0,1,1]$, a baseline of $2/3$ gives advantages $[-2/3,1/3,1/3]$. The failed response receives a negative direction; the two successes receive positive directions. This does not label every token correct or determine exact new probabilities: responses share parameters and interact during optimization. [28]

RLHF uses human feedback, often through a learned preference reward. RLVR uses verifiable rewards. These name feedback sources, not interchangeable optimizer specifications. [29][19]

Group Relative Policy Optimization, or GRPO, replaces a learned value baseline with comparisons among responses to one prompt. Its original outcome version uses $A_i=(R_i-\bar R)/s_R$, where $R_i$ is response reward and $\bar R,s_R$ are group mean and standard deviation. Each response token shares that advantage; clipped ratios and reference KL enter separately. [30]

Equal group rewards make the written normalization undefined. Code must handle zero spread explicitly; with denominator stabilization, the relative reward signal is zero, although KL can still contribute gradients. GRPO can use learned rewards or executable checks. Removing the value model and replacing a learned reward model are separate decisions. [30]

Sparse success leaves little useful contrast. More varied sampling can expose better trajectories, but excessive randomness can break coherent output. On-policy means sampling from the policy being trained; it does not mean the prompts represent production use. Task coverage and exploration remain separate design choices. [31][32]

Reference regularization and PPO clipping

A reference policy remains fixed at a designated starting checkpoint. The trainable policy changes. A separate, recent rollout snapshot records the policy that generated a batch; it need not equal the reference. [33][34]

Proximal Policy Optimization, or PPO, modifies incentives relative to the rollout policy. A critic, or value model, estimates expected return to help estimate advantages; it does not judge answer correctness. [35]

With $\epsilon=0.2$, $A=2$, and $\rho=1.4$, the surrogate is 2.4 instead of 2.8. With $A=-2$ and $\rho=0.6$, it is −1.6 instead of −1.2. These favorable changes stop earning extra incentive. Harmful changes still count: increasing a negative-advantage action to $\rho=1.4$ gives −2.8. [35]

ratio = exp(logp - old_logp.detach())
advantage = advantage.detach()
unclipped = ratio * advantage
clipped = ratio.clamp(1 - epsilon, 1 + epsilon) * advantage
policy_loss = -minimum(unclipped, clipped).mean()

Clipping removes an incentive; it does not impose a hard probability bound. Shared parameters and other objective terms can still change a clipped action's probability. PPO may reuse a batch over several minibatch epochs. Its old-policy ratio remains distinct from a fixed-reference KL penalty. [34]

Direct preference optimization

Direct Preference Optimization, or DPO, trains the response policy directly from comparisons. Sequence log-probability sums conditional token log-probabilities. Original offline DPO uses fixed pairs and reference probabilities, without fitting a separate reward network or sampling fresh responses during each update. [36]

Under Bradley–Terry preferences and KL-regularized reward optimization, $\pi^*(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)e^{r(x,y)/\beta}$. Rearranging expresses reward through a policy/reference log ratio plus a prompt-only constant. That constant cancels between responses, yielding DPO. The derivation requires reference support and does not guarantee equivalent finite-training outcomes. [36]

Two valid three-outcome distributions expose the relative objective. Assume $\beta=1$; other responses are grouped into the third outcome. These are constructed distributions, not measured optimizer steps. [37]

The margin becomes $\log(0.30/0.40)-\log(0.20/0.40)=\log1.5$. Loss falls from approximately 0.693 to 0.511, although chosen probability falls from 0.40 to 0.30. The chosen response gains relative to the rejected response, not necessarily in absolute probability. [37]

margin = (logp_w - logref_w) - (logp_l - logref_l)
loss = -logsigmoid(beta * margin).mean()

A simpler training path retains dependence on pair meaning and coverage. It can also optimize unintended correlates: tested DPO recipes expanded response length beyond the training distribution. Removing a reward network does not remove proxy error. [38]

Composing a post-training recipe

These operations need not form one compulsory sequence. R1-Zero applied RL without initial SFT; R1 used cold-start demonstrations, reasoning RL, filtered-response SFT, and later RL. The distinction concerns particular recipes, not evidence that either route is universally preferable. [19]

A cold start supplies examples of behavior that later reward optimization needs to discover. When a small model fails to learn an RL task, missing related SFT coverage is one diagnosis to test alongside task difficulty. Reintroducing demonstrations is an experiment, not a guaranteed repair. [41]

Rejection-sampling fine-tuning generates candidates, retains selected responses, and applies supervised likelihood training to them. Filtering changes the dataset; it does not turn the student's update into a policy gradient. Distillation addresses transfer from teachers, and Synthetic Data addresses generated-example construction. [42]

Reinforcement learning from AI feedback, or RLAIF, identifies the feedback source. Constitutional AI used model critiques and revisions for supervised training, then AI comparisons to fit a preference model for RL. Human choices remained in its principles and design. Choosing AI rather than human judgments does not choose the optimizer or supply independent ground truth. [29]

Composed stages also appear in LFM2.5-1.2B: its Base checkpoint is distinguished from an Instruct checkpoint trained with SFT, preference alignment, and multistage RL. Record the parent checkpoint, operation, data version, candidate source, rubric, reward version, and resulting checkpoint so that a recipe describes reproducible changes rather than only a model name. [43][1]

Proxy optimization and reward failures

A proxy is a measurable substitute for the intended goal. Reward hacking exploits weaknesses in that substitute; reward overoptimization occurs when stronger optimization improves proxy scores while degrading independently assessed quality. Goodhart's law frames the risk: targeting a measure can weaken its relationship to the original goal. [44][45]

Optimization changes which errors matter. Best-of-N selects the highest proxy-scored response among N samples without updating the generator. More candidates can expose rare scoring mistakes. In controlled synthetic experiments, stronger selection or PPO eventually reduced a larger model's independent scores while pursuing a smaller proxy model. That larger scorer was an experimental comparator, not human welfare. [44]

Kernel-generation training exposed shortcuts including no-op kernels and returning reference code. Inspecting rollouts led to exploit detection, static checks that kernels existed and launched, and correctness and speed comparisons. These checks address different failure modes: source structure alone establishes neither correct execution nor speed. [46]

Sycophancy favors agreement with a user's stated beliefs over truthful response. Preference research found that persuasive agreement could receive approval despite incorrectness. Optimization can amplify that mismatch. Separately, DPO length experiments show that implicit preference objectives can expand verbosity; longer answers are not inherently worse, but length must not stand in for the desired quality. [13][38]

Behavioral gains and competing requirements

Abstention declines to answer when support is insufficient. Rewarding it above a wrong answer can discourage fabrication, but fewer wrong answers may simply reflect fewer answers. Report accuracy among answered requests alongside answer coverage. Selective automation explains why both matter. [5]

Over-refusal means refusing a safe request. XSTest pairs safe prompts with nearby unsafe contrasts to distinguish contextual judgment from reactions to vocabulary. Compliance means attempting an answer, regardless of accuracy. Testing only safe requests would also reward a system that answers everything, including prohibited requests. [6]

Catastrophic forgetting is substantial loss of previously available capabilities after further training. Specialization gains and retained competence need separate measurements. Instruction-fine-tuning studies show that their balance depends on data, duration, and update configuration; no single adaptation method wins every comparison. Continual Learning develops retention methods. [48]

A behavioral requirement should not be reduced to a surface style. If comparisons reward agreeable wording, favoring those labels does not repair factual support. If safe and unsafe cases share vocabulary, broadly rewarding caution can suppress legitimate assistance. Contextual examples and corrected judgments target these signal defects; tighter parameter constraints alone do not change label meaning. [13][49]

Other dimensions remain distinct. Repetition measures whether generation loops, not whether it solves the task. Resource use measures expenditure, not competence. Where varied responses matter, define useful diversity explicitly rather than treating randomness as success. Separate metrics and meaningful task slices keep aggregate improvements from concealing losses. [50][51]

Evidence of adaptation gains

Compare checkpoints on matched independent cases, recording prompts, decoding, answer limits, selection procedures, tool configuration, and evaluator versions. Account for prompt and output length when measuring resources. Controlled offline comparisons separates matched tasks from identical trajectories; a different route to success need not be a failure. [51]

Contamination includes semantic leakage. Training on paraphrases or translations can inflate benchmark results while evading literal overlap checks. Teacher-generated examples are not automatically independent of public tests. Track lineage, investigate similar problems, and protect final assessment cases. Data used to select a recipe has become development data; see Independent data boundaries. [52]

A score increase can reflect better task solving, formatting, selection, or more inference expenditure. In summarization research, controlling length reduced the measured preference advantage. A FinQA talk reported that simpler training outperformed mixed and progressive curricula, interpreting transfer as improved tool discipline; incomplete controls prevent treating that explanation as isolated causation. [4][53]

Targeted ablations change one training component while preserving comparison conditions. Diagnostic rubrics can identify missing behaviors and guide new training data without becoming the scalar RL reward themselves. This separates learning where the failure occurs from deciding how to optimize it. [54]

Report uncertainty in paired checkpoint differences, not only separate point estimates. Paired bootstrap resampling retains each case's model-to-model relationship; correlated cases require an appropriate sampling unit. Training-seed variation is a separate source of uncertainty. Uncertainty in scores and Release decisions connect these estimates to acceptance. [55]

Boundaries of an alignment claim

A bounded report could state: Checkpoint {candidate_id}, compared with {baseline_id}, improved supported completion by {difference and interval} across {N} independent requests from {population}, under {prompt, decoding, and budget}. Unsupported answers, harmful compliance, benign refusals, and retention were assessed against {rubric_version}; {unmet criteria and exclusions} remain. These placeholders require observations, not inferred success from training loss. [51][55]

Checkpoint, prompt, tool, rubric, population, or operating-condition changes alter the claim's scope and may require reassessment. Even a system-prompt edit can shift behavior away from the fine-tuning distribution. Preserve the evidence-to-version relationship through Evaluation records and reassessment. [56]

Trained behavior does not replace enforced permissions. Learning to disregard conflicting instructions can improve measured resistance while leaving vulnerabilities and benign-task regressions. Control placement and defense in depth explains enforcement; Adversarial tests and bounded security claims explains what attack testing establishes. [49]

Open questions

1. 01

   Preference aggregation remains difficult when legitimate rubrics conflict. A scalar ranking cannot represent every majority cycle, and disagreement is not necessarily annotation error. Progress would preserve judgment provenance and demonstrate which populations benefit or lose under an explicit aggregation rule. [14]

2. 02

   Long-trajectory credit assignment must distinguish useful actions from accidental companions of success. Shared terminal returns blur that distinction. Progress would show that localized feedback improves independent outcomes on alternative trajectories, with downstream states recomputed after changed actions. [24][25]

3. 03

   Distinguishing capability elicitation from new capability learning requires more than higher scores. Rare pre-existing solutions can become easier to sample. Progress would compare strong starting-policy elicitation and sampling baselines against adaptation on genuinely new task families under matched budgets. [59]

4. 04

   Reward validity under stronger optimization remains unresolved because new policies can discover previously unseen exploits. Progress would sustain independent task quality as optimization pressure increases, while testing fresh failure classes rather than only repairing known hacks. [44][18]