[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Daniel Han frames modern language model development as a sequence of optimization stages rather than a single leap from random weights to a capable assistant. Pretraining teaches next-token prediction over broad data; mid-training can emphasize higher-quality material and extend context length; supervised fine-tuning (SFT) teaches conversational or instruction-following behavior; and later stages include preference optimization and Reinforcement Learning with Verifiable Rewards (RLVR). He distinguishes RLVR from preference fine-tuning because it uses reward functions tied to outcomes that can be checked. 13:02

This progression also explains naming conventions that otherwise look inconsistent. Han identifies pretrained and instruction-tuned variants through labels such as PT, IT, Base, Instruct, and Chat, while noting that naming practices vary across model families. The practical distinction is that a base model primarily continues text, whereas an instruction-tuned model has already been adapted to answer requests in a more conversational format. 10:57

Although Han says reinforcement learning can be applied directly to a pretrained model, he argues that skipping SFT is usually inefficient. A base model may not reliably produce answers in the format a reward function can recognize, leaving training runs stuck with little or no useful reward signal. His recommended path is to begin with an instruction-tuned checkpoint or briefly prime a base model with supervised examples before applying reinforcement learning. 15:48

Han introduces reinforcement learning through an agent, an environment, an action, and a reward. In a game such as Pac-Man, movement changes the environment, collecting an item can produce a positive reward, and encountering an enemy can produce a negative one. For a single-turn language model task, the prompt functions as the state and the generated response—including its reasoning trace, if present—functions as the action. 16:41

A simple arithmetic problem illustrates how much judgment enters reward design. An exact answer can receive a positive score, an incorrect numerical answer can receive zero or partial credit based on its distance from the correct value, and an invalid answer type can receive a stronger penalty. Han reports that distance-based scoring helped mathematical learning in his experiments, while emphasizing that binary pass-fail scoring is easier to implement and often more practical when meaningful distance is difficult to define. 19:31

The optimization target is not merely a raw reward in isolation. Han explains advantage as the reward relative to an expected or average baseline: positive advantage indicates an above-average action, while negative advantage indicates a below-average one. In conventional approaches, a separate value model estimates that baseline from the current state, and the language model’s own token probabilities provide the probability information needed to update the policy. 56:41

Han describes PPO as a reinforcement learning optimization method involving a trainable generating policy, a fixed reference policy, and a separate value model. The reference policy represents the checkpoint the run started from, while the generating policy is updated during training. The value model estimates expected reward, but maintaining and training another large model adds substantial memory, compute, and potential estimation error. 22:52

The extra terms in PPO are presented as safeguards against unstable updates, overfitting, and reward hacking. A likelihood ratio compares the updated policy with the policy that generated an action; clipping limits excessively large changes; and a KL divergence term discourages the trained model from drifting too far from its starting checkpoint. Han stresses that these protections involve tradeoffs: restricting movement improves stability, but may also limit exploration of substantially different model behaviors. 1:09:06

GRPO removes the separate value model and estimates relative performance from multiple sampled answers to the same prompt. Han’s example generates several responses, scores each one, and uses the group’s mean and standard deviation to compute a normalized advantage; each prompt forms its own comparison group. When GRPO is paired with RLVR, a directly implemented reward function can also replace a learned reward model, reducing the number of heavyweight components required for training. 48:04

Han repeatedly identifies the reward function, rather than the optimizer, as the difficult part of practical reinforcement learning. Mathematical answers can be checked directly, while code can be assessed through execution, imports, formatting, or other observable properties. More ambitious tasks expose the weakness of partial checks: a generated game might run and contain expected words or assets without actually being a good implementation of the requested game. 1:30:35

His notebook combines several reward components. A regular expression checks whether the model follows the requested reasoning-and-answer format; partial rewards recognize useful formatting progress before the full structure is correct; distance-based scoring gives higher marks to numerical answers closer to the target; and additional parsing handles formatted numbers. These examples show why reward shaping can help avoid an all-zero training signal, but also why each component must be tested to ensure that extraction and scoring work as intended. 2:09:03

Reward composition introduces additional judgment. If mathematical accuracy, code execution, and formatting are scored on different scales, their relative weights determine which behaviors training prioritizes. Han also warns that rewarding only a correct final answer does not establish that the intermediate reasoning was sound; likewise, an LLM as a judge can help evaluate subjective outputs such as summaries, but he cautions that repeatedly optimizing against another model’s judgments may eventually break down. 1:35:50

The demonstrated workflow uses Qwen3, Unsloth, VLLM, and LoRA to run a resource-constrained GRPO experiment. Han sets a maximum sequence length of 2,048 for the example, notes that larger values increase memory pressure, and explains that 4-bit loading can reduce memory requirements. LoRA limits training to added parameter-efficient weights, while sharing VLLM weights between inference and fine-tuning avoids maintaining separate full copies of the model. 1:59:50

Before reinforcement learning, the base-model demonstration uses a chat template and a small amount of supervised priming so that the model can produce recognizable conversational and reasoning formats. Han initially describes a dataset containing 7,000 rows, then clarifies that the actual run used only a much smaller subset, later identifying 118 rows. He also says an existing instruction-tuned model can bypass this additional priming stage altogether. 2:06:02

Sampling determines whether GRPO has meaningful alternatives to compare. Han recommends avoiding zero temperature, suggests temperatures around 1.0 to 1.2 in the notebook and discusses higher settings elsewhere, and pairs higher temperature with min P around 0.1 to control low-quality outputs. More generations per prompt improve coverage but increase memory and compute; gradient accumulation can reduce the memory burden of larger effective training workloads, although Han notes that the usual batch-size equivalence does not straightforwardly hold for GRPO. 1:23:41

The training example begins with negative rewards, occasionally discovers high-scoring outputs, and trends toward more positive results over time. Han says the run took two hours and 54 minutes on the demonstrated free Colab setup, but cautions against treating formatting rewards as evidence of real learning: the important metrics are the reward components that test whether answers are actually correct. His example contrasts a base model that produces irrelevant text for the square root of 101 with a trained output that supplies reasoning and an approximately stated numerical answer. 2:15:52

Han presents dynamic quantization as a selective compression strategy rather than a uniform reduction in numerical precision. He describes shrinking DeepSeek-R1 from approximately 730 GB to approximately 140 GB while acknowledging an accuracy tradeoff, and cites a Llama 4 Scout example in which a heavily quantized model remains close to a higher-precision result on the benchmark he discusses. His central claim is that some mixture-of-experts components can tolerate aggressive quantization, while attention layers, shared experts, and other sensitive components should retain higher precision. 2:31:38

The danger of indiscriminate quantization becomes clear in a vision-model example: applying 4-bit precision uniformly produces an incorrect image description, whereas preserving selected layers at higher precision restores the intended recognition behavior. Han recommends inspecting activation quantization error and weight quantization error to identify sensitive layers, because exhaustive testing of every possible layer combination would be prohibitively expensive. He also emphasizes that sensitivity patterns differ across models, including Qwen, Llama 3.2, and Pixtral. 2:34:07

Han further cautions that important weights are not necessarily the largest numerical outliers: a small value can still be critical to model behavior, making magnitude alone an unreliable guide to safe quantization. He discusses lower-precision formats including FP4 and MXFP4, while framing predictions about future GPU speedups as his personal view rather than established fact. His final practical recommendation is to experiment with Torch.compile, which he says can improve training speed and memory usage in some cases, while acknowledging that results vary and that compiler settings require tuning. 2:36:18