Popular talk #23
[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han
Synced transcript
Follow the talk
Automated overview
What this talk covers
Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.
This overview is derived from the transcript and has not been independently fact-checked by AI Engineer.
Generated highlights
Key moments
Unsloth fixed gradient accumulation bugs and introduced async offloaded gradient checkpointing.
Llama 1 trained on 1.4 trillion tokens; now models train on 10x more.
Yann LeCun's 2016 cake analogy: unsupervised learning is cake, SFT is icing, RL is cherry.
Agents and RL are connected: agent takes action in environment to maximize reward.
GRPO deletes the value model, using group statistics as baseline to save compute.
Reward function design: regex, code execution, LLM judge, even random rewards temporarily work.
These moment labels are generated from the unreviewed transcript and may contain errors.
Generated classification
Detected Q&A
Q: Is choosing 5 better than 0 for 2+2? A: Distance-based scoring rewards closeness.
Q: Why negative rewards? A: RL maximizes reward, so even negative values get maximized towards less negative.
Q: What models are generating and reference policies? A: Same model; generating is updated, reference is frozen.
Q: Are actions single token or full completion? A: Single turn chain-of-thought is the action.
Q: Does reward focus allow think-ahead inference? A: RL accentuates pre-training's forward prediction ability.
Q: Is distance-based scoring better than binary for math? A: Distance-based helps learn faster, but binary is simpler.
Podhood’s automated analysis classified these moments as Q&A. AI Engineer has not independently reviewed them.
Community discussion
Add context, evidence, or a useful disagreement
Build on the talk with an example, caveat, connection, or elaboration. Draft here, add the moment you’re discussing, then choose the direct-post pilot or the YouTube handoff.
Make it useful: add one concrete point, then support it with context, evidence, an example, or a caveat. Your words stay exactly as written.