# Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal Jason Mancuso, Modal [Talk page](https://ai.engineer/talks/3nDMYV-QHLs-training-your-own-agent-models-rl-is) · [Video](https://www.youtube.com/watch?v=3nDMYV-QHLs) ## Training Your Own Agent Models With RL Is Now Practical A failed production session can become a task an agent learns to solve. Jason Mancuso explains how Modal trained on new attempts even as samplers fell behind the trainer, then cut rollout costs by 15% in a run that mixed GPU types. ### What a model swap leaves to the agent team [0:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1s) A new model comes out. The agent team runs evals, A/B tests it, and rebuilds parts of the harness around whichever model offers the best quality at an acceptable price. Jason Mancuso of Modal sees a limit to that strategy. The team can choose among the combinations of capability, latency, and cost that providers offer. Its users care about how well the agent performs their particular task and what it costs to get it done. Modal customers had several reasons to take on the work of fine-tuning. - **Reliability.** Some teams wanted to run inference themselves to meet a service-level objective. An open model they could serve sometimes fell short on their tasks. Fine-tuning gave them a way to recover quality while retaining control over serving latency and cost. - **Proprietary data.** Some agents had to interpret private datasets and act on what they found. General-purpose models had never encountered those datasets during pretraining and could fail at the task altogether. Those teams turned to fine-tuning open models. - **Task performance per dollar.** In a competitive category such as coding, solving more of the user's task for the same spend matters. A team that can change both the harness and the model has another way to improve that ratio. Waiting for the next release has a real appeal. A new general-purpose model can outperform one a team painstakingly tuned from the previous generation. Mancuso expects more teams to invest anyway, especially where a deployment has a reliability requirement, private data, or a task-specific cost target that survives the next model release. ### Turn a failed trace into a task the agent can try again [3:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=215s) A recorded agent session follows one path through a tree. Supervised fine-tuning (SFT) on that trace teaches the model the actions taken along the recorded path. Once deployed, the model can take a different action early in the session. Now the next turn starts from a situation the demonstration never covered. Each additional turn creates more opportunities to diverge. In reinforcement learning (RL), the model attempts the task during training. Its own choices determine which situation it encounters next, and the outcome supplies feedback on the path it took. This is the sense in which Mancuso expects RL to generalize better for agents. Practice can reach branches missing from the recorded session. How much of the tree the model explores depends on the attempts it generates. A deployed agent gives the team a head start. Its harness already runs the model through a task. Its evals give the team a way to judge the result. Even the criteria used in A/B tests or to decide whether a harness change can ship can help define success in an RL environment. The harness supplies a place to act, and those criteria supply a basis for scoring an attempt. Production traces answer the harder question of what to practice. Find a session where the agent failed at something a user wanted to do. Use it to generate tasks that exercise that failure, run new attempts in the harness, and evaluate their outcomes. Even a failed trace is valuable here. It identifies a problem worth solving while leaving the new agent free to find a better path. ### When a version 1 rollout reaches a version 3 trainer [6:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=411s) The model has two jobs in this system. On inference GPUs, copies of it act as *samplers*. They generate rollouts, each a trajectory through the environment using the agent harness. The trainer consumes those rollouts and updates the model weights with gradient descent. It broadcasts the new weights back to the samplers, so subsequent attempts can reflect what the model just learned. The simplest schedule keeps the versions aligned. Generate rollouts, train on them, distribute the updated weights, then start the next round. Separate inference and training GPUs spend part of that schedule waiting for each other. Running both jobs continuously improves utilization, but the trainer can advance while a sampler is still working. An attempt generated by version 1 might arrive when the trainer is already on version 3. That gap is *staleness*. The question for version 3 is how likely it would have been to choose the tokens version 1 generated. A version gap is one reason those probabilities can differ. Small differences in the systems that run training and inference can also change the distribution, even when both sides intend to implement the same model. The challenge is to correct enough of that mismatch to keep learning from a rollout's outcome. ### Three corrections to asynchronous GRPO [9:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=578s) Modal starts with group relative policy optimization (GRPO), which uses the relative scores of attempts at a task to guide a policy update. The team adds three corrections so the trainer can use that signal when the samplers behave differently. - **Masked importance sampling handles token-level disagreement.** For a token in a rollout, compare the probability the trainer assigns it with the probability under the sampler that produced it, given the same preceding tokens. Their ratio measures how differently the two policies treat that choice. Masked importance sampling uses the comparison to adjust the token's contribution to learning. The ratio can reflect both the age of the sampler's weights and differences in how the two systems compute probabilities. - **Score centering handles drift across updates.** Token-level correction leaves some residual mismatch. Once an update incorporates that error into the weights, the next step starts from a slightly shifted model. Score centering builds a correction term that tracks and subtracts the accumulating drift. - **Account for the sampler's top-p cutoff.** At a setting such as 0.95 or 0.98, the sampler retains the most likely tokens until their cumulative probability reaches the threshold. Tokens in the excluded tail have zero chance of appearing at that position. If the trainer still assigns those tokens some of the probability mass, even its probabilities for the retained tokens describe a different choice. Its calculation has to account for which tokens the sampler could choose. Mancuso reports that with this treatment of top-p sampling, the recipe's score-centering correction becomes exact where it would otherwise be approximate. How much does the combination matter? Modal compared variants of the recipe in an ablation on coding environments. The run using the corrections together trained more stably than the alternatives the team tested. The result concerns stability over the course of training. Whether the resulting agent solves a particular product's failures still depends on which tasks the team designs and how it scores them. ### Send the weight change across clouds [12:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=754s) Modal could already run each part of the job. It had clusters for training, inference endpoints for sampling, and sandboxes for agent environments. Finding enough GPUs in one place was another matter. At the time of the talk, Modal pooled capacity from roughly 15 to 20 providers, including hyperscalers and smaller GPU clouds. That gave the scheduler options during a shortage. It also meant that plenty of GPUs could be free across the fleet while few were available together. Modal could schedule four or eight nodes fairly quickly. Finding a contiguous block of 16 or 32 took much longer. Agent RL made that constraint particularly expensive. At roughly two inference GPUs per training GPU, about two-thirds of a cluster went to generating rollouts. In an eight-node allocation, the trainer might get just two or three nodes. Moving the samplers elsewhere would free those slots, provided the trainer could keep distant copies of the model up to date. What has to cross the network after an update? A sampler that holds weights \(W_t\) needs only the difference \(\Delta_t = W_{t+1} - W_t\) to reconstruct the next version. Each full set of weights was dense and large, but Modal found the difference between consecutive steps to be highly sparse. Off-the-shelf compression could exploit that sparsity. Mancuso reports a roughly thousandfold reduction in the amount of weight-update data sent from the trainer to the rollout engines. In the coding run, the trainer stayed in one cluster while inference GPUs across multiple clouds generated rollouts. Compressed weight changes went out to the samplers. Completed trajectories came back as training data. Shrinking the outbound transfer let the team place inference work wherever it could find GPUs while keeping the training nodes together. What travels between one trainer and rollout engines in different clouds? Both sets of samplers can apply the same compressed change to their copy of the weights. Each returns the trajectories it generated. - Training cluster — Consumes trajectories and computes the next weights. - Compressed weight delta — Compress the sparse difference between successive weight sets. - Hyperscaler rollout engines — Run the model through tasks on available inference GPUs. - Other GPU-cloud rollout engines — Generate trajectories for the same run elsewhere in the fleet. - Training cluster → Compressed weight delta (Consecutive model versions) - Compressed weight delta → Hyperscaler rollout engines (Weight update) - Compressed weight delta → Other GPU-cloud rollout engines (Weight update) - Hyperscaler rollout engines → Training cluster (Rollouts) - Other GPU-cloud rollout engines → Training cluster (Rollouts) ### Let different GPU types generate rollouts for the same run [16:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=981s) Geography was only one source of fragmentation. The available GPUs also came in different generations. A trainer on B200s could use samplers on H100s or A100s, but those devices can run different kernels to implement the model's arithmetic. Those differences can alter token probabilities. Sparse deltas solve the problem of getting weights to those machines. The GRPO corrections address the separate problem of learning from the rollouts those machines produce. Which devices actually did the rollout work? Modal counted requests by GPU type over the course of training. Each type's count represents inference work that fed the same run. As requests went to different devices, the trainer had to learn from the mix of token distributions their kernels produced. ### More mismatch, stable training, and 15% cheaper rollouts [17:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1053s) Did the extra mismatch destabilize the updates? The comparison run used B200s for both training and inference. With mixed rollout hardware, Modal observed up to roughly four or five times as much trainer–sampler mismatch. Yet training reward and gradient norm, a measure of the size of the training gradient, behaved as stably as in the all-B200 run. The correction could tolerate a substantial increase in disagreement on this workload. Across the full training run, the rollout portion cost 15% less. Some inference work could run on cheaper GPUs for which there was less demand. The team got that saving using off-the-shelf inference configurations for each GPU type. Mancuso saw further savings as a possibility if the team tuned individual deployments, including their numerical precision. Even within a single GPU type, the trainer had to accommodate two precision choices. One H200 pool ran the model in BF16, the more conservative choice. Another H200 pool ran it in FP8. The lower-precision model could assign different probabilities to the next token even on the same hardware. The same corrections allowed the trainer to learn from those quantized rollouts alongside the others. ### What it means to own the agent stack [19:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1158s) Modal built its implementation on an existing reinforcement learning framework and inference engine. By the time of the talk, the team had published a technical write-up and planned to release the source code. Owning the stack, in this example, meant understanding those components well enough to change how the trainer and samplers worked together. [Recording at 19:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1167s) Mancuso expects that competence to matter most where teams already compete on task performance per dollar, as coding agents do. For a team with an agent in production, the first decision is close to the user. Which failure is worth turning into a training task? The eval determines which attempts succeed. The training recipe determines whether the team can learn from those attempts as the model changes. In Modal’s run, the recipe let the trainer learn from rollouts generated by H200s running at FP8. ## Key takeaways - A failed production trace can supply a task for RL. The harness lets the model try other paths, and existing eval criteria can score the outcomes. - Asynchronous sampling keeps GPUs busy while the trainer advances. Modal's GRPO recipe accounts for differences in token probabilities, drift across updates, and the set of tokens the top-p sampler could choose. - Compressing sparse weight changes made cross-cloud sampling practical. Correcting for differences among samplers let Modal mix hardware and precision. In the reported run, reward and gradient norm stayed as stable as in the all-B200 comparison, and rollouts cost 15% less. ## Slides ### Scaling Agent RL Across the Globe Jason Mancuso opens his talk on reinforcement learning for agents with a familiar setup. The agent harness connects to Model B, and engineers can swap in another model as new ones arrive. [0:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=11s) [The talk title appears in white over a dimmed diagram of an agent harness and three models.](https://assets.ai.engineer/sites/aiecode2025/sha256/021f274713cc6ce1a77d9b9e18da1fa24d0bf245aa5a837f1e081b60c9e189fd.webp) Scaling Agent RL Across the Globe Models as static backends AGENT HARNESS Model A Model B Model C Interface text: Close speaker notes window 2 / 27 ### Models as static backends The agent harness uses Model B, with A and C as alternatives. As new models arrive, engineers run evals and A/B tests, then rewrite the harness around the model that best balances cost and quality for their use case. [0:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=21s) [An agent harness connects to Model B, flanked by Models A and C. Curved arrows indicate model swaps. The caption lists swapping, running evals, A/B testing, rewriting the harness and repeating.](https://assets.ai.engineer/sites/aiecode2025/sha256/44333d1c2418b46c0f9697aaf8cf4fda2690b98af22472dc0219e28f428dc942.webp) Models as static backends AGENT HARNESS Model A Model B Model C swap · run evals · A/B test · rewrite harness · repeat Interface text: Close speaker notes window 2 / 27 ### A losing strategy once you compete on agent economics Mancuso argues that switching models for a better mix of price and quality falls short once teams have to compete on the cost of running an agent and give users a distinct experience. [0:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=52s) [Centered text reads “A losing strategy once you compete on agent economics,” with “economics” highlighted in green.](https://assets.ai.engineer/sites/aiecode2025/sha256/c22ca58c556aa28d31a8c69b45f04e0a361a372eeb95b5f456f8061534094704.webp) A losing strategy once you compete on agent economics Interface text: Close speaker notes window 3 / 27 ### Why do people finetune at all? Mancuso introduces fine-tuning through a question his customers face: when an agent has to compete on both cost and user experience, why train your own model? [1:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=69s) [The question “Why do people finetune at all?” appears at upper left with “finetune” in green. The slide body is empty.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8b79ab0727eedb92683de54df4e33fe617e531d4cbca8cff7b5623cd16c2099.webp) Why do people finetune at all? Interface text: Close speaker notes window 4 / 27 ### Why do people finetune at all? Teams fine-tune to improve the tradeoff among latency, quality and cost while meeting reliability targets. They can teach agents formats missing from pretraining and own the stack to compete on task performance per dollar. [1:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=73s) [Beneath the question, a single card shows an arrow from a dotted curve to a green point. Its text cites Pareto-optimal latency, quality and cost, plus reliability SLOs.](https://assets.ai.engineer/sites/aiecode2025/sha256/42108fb49098c0b94791c1aa259196f6fdff0b9d15c9ac7a67b010e90d5cb2d5.webp) Why do people finetune at all? Pareto-optimal latency, quality and cost plus real reliability SLOs Interface text: Close speaker notes window 4 / 27 [2:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=156s) [Three cards give reasons to fine-tune: balance latency, quality and cost while meeting reliability targets, handle proprietary formats absent from pretraining, and own the agent harness, model and infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d91a2d81f433064fa4549bf4c7d242d43ad671a95fa5d7e3d28a2acb8f5cda1.webp) Why do people finetune at all? Pareto-optimal latency, quality and cost plus real reliability SLOs Proprietary data formats absent from every pretraining set Agent harness Model Infra Own your intelligence compete on task-intelligence per $ Interface text: Close speaker notes window 4 / 27 ### Why RL, and not SFT? Supervised fine-tuning trains the agent on the recorded white path. A production session can follow the same first turn, then branch off. With reinforcement learning, the agent explores alternative paths during training. [4:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=265s) [A tree branches over five turns. A white path marks the recorded trace used for supervised fine-tuning, a dotted orange path shows production diverging from it, and green branches show paths explored by reinforcement learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/90926240342d76deae9eef7b7dbf6a0dd59aaa7d88e0d0b873373ccfbb7609bc.webp) Why RL, and not SFT? Timeline: task, turn 1, turn 2, turn 3, turn 4, turn 5 In prod: off the trace, out of distribution SFT: train on a recorded trace RL: the model explores the branches itself Interface text: Close speaker notes window 5 / 27 ### But isn't it hard to do RL? Your deployed agent harness and its evals can serve as the RL environment. To give the agent tasks, find failures in production traces, build tasks around those cases, and split them into training and test sets. [5:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=333s) [Arrows connect the agent harness and evaluations to an environment, and production traces to tasks split into training and test sets. Both lead to a box labeled “Ready for RL.”](https://assets.ai.engineer/sites/aiecode2025/sha256/1e832ea29a53abffdab6e5f1866b99450d92df14d7454f7f276e91fff52e0919.webp) But isn't it hard to do RL? YOU ALREADY HAVE Agent harness ✓ deployed in prod Evals ✓ for that deployment Prod traces ✓ from that deployment RL NEEDS Environment harness + evals Tasks train / test split Ready for RL congrats! Interface text: Close speaker notes window 6 / 27 ### Isn't the science hard? Mancuso scatters six references across the slide, from IMPALA to top-p mask replay, to illustrate why even a team that has deployed an agent might find reinforcement learning intimidating. [6:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=368s) [Six scattered citation cards name IMPALA, GRPO, masked importance sampling, score centering, top-p mask replay and token-in, token-out.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ce9db31d20b6fb2beb214b5f4f24c7a82971027631e33aa9c4017591df8c6a6.webp) Isn't the science hard? IMPALA arXiv 1802.01561 GRPO arXiv 2402.03300 Masked importance sampling arXiv 2510.18855 Score centering arXiv 2609.20807 Top-p mask replay arXiv 2512.02556 Token-in, token-out LMSYS blog, 2026 Interface text: Close speaker notes window 7 / 27 ### Isn't the science hard? Mancuso brings six techniques, including GRPO and masked importance sampling, into one reinforcement learning recipe. His team has had success with it, and he thinks it makes training simpler and more stable. [6:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=390s) [Six cards list GRPO, masked importance sampling, score centering, top-p mask replay, token-in/token-out and IMPALA. An arrow leads from the grid to a box labeled “A recipe.”](https://assets.ai.engineer/sites/aiecode2025/sha256/421be2e5135f8fbf9276a43df9bdaca6768ec11df9ec3d921341f38f72df6c2f.webp) Isn't the science hard? GRPO arXiv 2402.03300 Masked importance sampling arXiv 2510.18855 Score centering arXiv 2609.20807 Top-p mask replay arXiv 2512.02556 Token-in, token-out LMSYS blog, 2026 IMPALA arXiv 1802.01561 A recipe that we've found success with Interface text: Close speaker notes window 8 / 27 ### How does RL work? Eight samplers send trajectories and log-probabilities to one trainer. The trainer uses that data to update the model, then sends the new weights back to every sampler for the next round of rollouts. [6:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=409s) [Eight samplers connect through a central junction to one trainer. The samplers supply trajectories and log-probabilities, and the trainer sends new weights back to the samplers.](https://assets.ai.engineer/sites/aiecode2025/sha256/bcca4827d4e02bbca813eae1221117df948d6d8ee3323b28ecab675a8534ab3a.webp) How does RL work? SAMPLER SAMPLER SAMPLER SAMPLER SAMPLER SAMPLER SAMPLER SAMPLER TRAINER trajectories + log-probabilities new weights Interface text: Close speaker notes window 9 / 27 ### The math of RL prefers sequential steps The trainer uses rollouts from v0 to produce v1. Inference then uses v1 for the next batch of rollouts. The hatched blocks show the cost of taking turns: one set of GPUs sits idle while the other works. [7:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=459s) [On a two-row timeline, inference produces rollouts with versions v0, v1 and v2. Each rollout is followed by a trainer update to the next version. Hatched blocks mark the GPUs waiting during each step.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdf0d2f7b67719259ca4d4c39937c41f9f8c9ca25c269b1d9dfa027ff3b6393e.webp) The math of RL prefers sequential steps INFERENCE rollouts v0 rollouts v1 rollouts v2 TRAINER train → v1 train → v2 train → v3 time idle GPUs Interface text: Close speaker notes window 10 / 27 ### We want both sides async While the trainer builds v1 from the first batch of rollouts, inference uses v0 to generate a second batch. This overlap keeps GPUs busy, but by the time the trainer uses that second batch, its rollouts come from an older model. [8:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=487s) [Inference runs six consecutive rollout batches using v0 twice and then v1 through v4. The trainer waits during the first batch, then produces v1 through v5 while inference continues.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d72c0a10aa0977947497da1744a086fd06e5ebd703c132aa4eb9a2b8c8aa7ff.webp) We want both sides async INFERENCE rollouts v0 rollouts v0 rollouts v1 rollouts v2 rollouts v3 rollouts v4 TRAINER train → v1 train → v2 train → v3 train → v4 train → v5 time Interface text: Close speaker notes window 11 / 27 ### Training Qwen3.6-35B-A3B on agentic coding With SC + MIS + top-p, both samplers sustained gains in pass@4. On HARD-51 they started near 50% and finished around 67–75%. On the full set both rose from about 89% to 94%. Several other runs fell below their starting scores. [11:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=718s) [Two line charts plot pass@4 against training step for HARD-51 and the full set. Color identifies the training method, while solid and dashed lines distinguish mixed and homogeneous samplers. The bright green SC + MIS + top-p curves finish above their starting scores in both charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/7b745c23314a46f439d44f44c7560c535f696e4bfdde8394dc69e4b59127c09a.webp) Training Qwen3.6-35B-A3B on agentic coding pass@4 on HARD-51 Y-axis ticks: 0%, 20%, 40%, 60%, 80% X-axis ticks: 0, 100, 200 pass@4 on the full set Y-axis ticks: 50%, 60%, 70%, 80%, 90%, 100% X-axis ticks: 0, 100, 200 Training step Samplers — Mixed --- Homogeneous Method GRPO GRPO + MIS GRPO + top-p SC SC + MIS SC + MIS + top-p Interface text: Close speaker notes window 14 / 27 ### So, how far can we push this recipe? Combining the techniques made training more stable in the coding experiment. Mancuso asks whether the same RL recipe can scale and work on a wider range of problems. [12:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=751s) [A centered question asks “So, how far can we push this recipe?” The word “push” is green against the black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1f92653773db89b62c327bf01c86ce298e8a42240aa1ae5f2665443cd409dc6.webp) So, how far can we push this recipe? Interface text: Close speaker notes window 15 / 27 ### Why Modal for agent RL? On Modal, clusters train the model, inference servers run it, and sandboxes host the environments the agent interacts with. Developers can combine all three into an agent RL training system on the same serverless infrastructure. [13:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=796s) [Training clusters, inference servers and sandbox environments combine to support agent RL. A caption says all three are built on Modal’s serverless AI infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/63a1830cdccd5bfb41b21687b2cef511d009b8609b0d3b7f441315ff89775580.webp) Why Modal for agent RL? Training CLUSTERS + Inference SERVERS + Environments SANDBOXES = Agent RL we <3 it All built on Modal's serverless AI infra Interface text: Close speaker notes window 16 / 27 ### Modal's capacity model Provider A’s first region can fit four B200 GPUs together. No region shown can fit 16. With roughly two inference GPUs per training GPU in agent RL, inference accounts for about two-thirds of the cluster. [14:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=870s) [Colored squares show GPU capacity by provider and region. Four B200s fit in Provider A’s first region. The sixteen-B200 request has no fit. A note gives a 2:1 ratio of inference GPUs to trainer GPUs for RL.](https://assets.ai.engineer/sites/aiecode2025/sha256/519c4540903b52017cd58c1df961c2371da9d892669f77641acc4e81106f6aec.webp) Modal's capacity model PROVIDER A region 1 B200 H100 region 2 A100 H200 PROVIDER B region 1 B200 H100 A100 region 2 B300 RTX 6000 PROVIDER C region 1 H200 A100 region 2 B200 H100 B300 REQUESTS 4 × B200 fits 16 × B200 no fit RL: 2:1 inference to trainer GPUs Interface text: Close speaker notes window 17 / 27 ### The weight update is super sparse Only a few weights change from one RL step to the next. The orange cells mark those changes. Mancuso says compressing this mostly zero update cuts the data the trainer sends to rollout engines by a factor of 1,000. [14:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=899s) [Subtracting the weight grid at step t from the grid at step t+1 produces a mostly zero difference grid. A few orange cells mark the weights that changed.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c65346b339f10bd4b02564028e0f783dea8835e286709624982788003f48a2c.webp) The weight update is super sparse weights, step t+1 − weights, step t = diff: mostly zeros Interface text: Close speaker notes window 19 / 27 ### Disaggregated RL on heterogeneous compute Eight sampler configurations run rollouts in BF16, FP8 or NVFP4. They send trajectories and log-probabilities to a B200 trainer, which updates the model in BF16 and sends new weights back to the samplers. [16:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=975s) [Eight sampler configurations send trajectories and log-probabilities to a B200 trainer running BF16 and receive updated weights.](https://assets.ai.engineer/sites/aiecode2025/sha256/814062fb6b3d5f78a0702e306a9710e51ced339adddd5dc443e013b54437e10f.webp) Disaggregated RL on heterogeneous compute Sampler B200 NVFP4 · TRT-LLM Sampler H100 FP8 · FA3 Sampler H200 BF16 · FA3 Sampler A100 BF16 · FlashInfer Sampler RTX PRO 6000 BF16 · FlashInfer Sampler H100 BF16 · FA3 Sampler H200 FP8 · FA3 Sampler B300 NVFP4 · TRT-LLM Trainer B200 BF16 Ampere Hopper Blackwell Trajectories + log-probabilities New weights Interface text: Close speaker notes window 21 / 27 ### Many GPU types, many precisions The B300 and B200 together served nearly half of all requests during training, both running at NVFP4. Two H200 pools used different precisions: the FP8 pool served 18% of all requests, twice the BF16 pool’s 9%. [17:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1023s) [Eight rows track the share of requests served during training. The B300 running NVFP4 has the largest overall share at 29%, followed by the B200 running NVFP4 at 20%.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ffaa683e1a36b434baed084049fd5f053564e69d2c306e4e19d1fba516a1c06.webp) Many GPU types, many precisions Share of requests served Overall A100 BF16 · FlashInfer 3% RTX PRO 6000 BF16 · FlashInfer 4% H100 FP8 · FA3 8% H100 BF16 · FA3 9% H200 FP8 · FA3 18% H200 BF16 · FA3 9% B200 NVFP4 · TRT-LLM 20% B300 NVFP4 · TRT-LLM 29% Timeline: Training → Interface text: Close speaker notes window 22 / 27 ### Stable training despite massive mismatch Mixed samplers showed up to roughly 4–5 times as much per-token mismatch with the trainer as the B200-only run. Training reward rose along a similar path in both runs, and gradient norms stayed steady. [17:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1033s) [Solid green and dashed blue lines compare mixed and homogeneous samplers across training steps. Reward rises for both, while per-token KL divergence stays higher for mixed samplers. The lower plots track gradient norm and entropy.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6aa660c9fbcf047e2038702b69d9bfa16fe5c3cbb6c5c6a647744745088811f.webp) Stable training despite massive mismatch Training reward Y-axis ticks: 0.6, 0.5, 0.4, 0.3 X-axis ticks: 0, 100, 200 KL(sampler || trainer) per token Y-axis ticks: 0.01, 0.001 X-axis ticks: 0, 100, 200 Gradient norm Y-axis ticks: 0.03, 0.02, 0.01, 0.00 X-axis ticks: 0, 100, 200 Entropy Y-axis ticks: 0.3, 0.2, 0.1, 0.0 X-axis ticks: 0, 100, 200 Training step SC + MIS + top-p on the mixed samplers SC + MIS + top-p on the homogeneous samplers Interface text: Close speaker notes window 23 / 27 ### Competitive agents need the full stack Most agent work stops at the harness, one layer above the model. Mancuso argues that teams gain a competitive edge by integrating all six layers, from the infrastructure that runs the model to the product people use. [20:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1212s) [From bottom to top, the six layers are infrastructure, model, agent harness, environment, data flywheel and product. A green outline labeled “vertical integration” encloses all six, and an orange outline highlights the agent harness.](https://assets.ai.engineer/sites/aiecode2025/sha256/9519a4c0a83c8141ceab7436346746162492354d2d4362cea970d57716a88638.webp) Competitive agents need the full stack Product Data flywheel Environment Agent harness Model Infra most agent work stops “above the model” vertical integration Interface text: Close speaker notes window 25 / 27 ### modal.jobs Mancuso closes by sharing modal.jobs and inviting attendees to talk with the Modal team at its booth. [20:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1237s) [Centered on black, “modal.jobs” appears above “Come find us at the Modal booth.”](https://assets.ai.engineer/sites/aiecode2025/sha256/828f16e3486f597ec890420833833b62620080b455a564322cb9c14373908791.webp) modal.jobs Come find us at the Modal booth Interface text: Close speaker notes window 27 / 27 ## Full transcript [0:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1s) [music] [0:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=12s) Uh, I'm Jason. I'm from Modal and today [0:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=15s) Uh, I'm Jason. I'm from Modal and today [0:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=15s) Uh, I'm Jason. I'm from Modal and today I'm talking about scaling agent RL [0:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=18s) I'm talking about scaling agent RL [0:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=18s) I'm talking about scaling agent RL across the globe. Um, but first I think [0:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=23s) across the globe. Um, but first I think [0:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=23s) across the globe. Um, but first I think a lot of AI engineers view models as [0:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=26s) a lot of AI engineers view models as [0:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=26s) a lot of AI engineers view models as kind of like static backends that um [0:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=29s) kind of like static backends that um [0:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=29s) kind of like static backends that um that they [0:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=31s) that they [0:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=31s) that they kind of swap out like as the new models [0:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=33s) kind of swap out like as the new models [0:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=33s) kind of swap out like as the new models come out they'll maybe uh test the [0:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=36s) come out they'll maybe uh test the [0:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=36s) come out they'll maybe uh test the models, run evals, uh do AV AB tests and [0:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=40s) models, run evals, uh do AV AB tests and [0:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=40s) models, run evals, uh do AV AB tests and kind of like rebuild their harness [0:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=42s) kind of like rebuild their harness [0:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=42s) kind of like rebuild their harness around whatever model is cheapest but [0:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=45s) around whatever model is cheapest but [0:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=45s) around whatever model is cheapest but also highest quality for for their use [0:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=47s) also highest quality for for their use [0:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=47s) also highest quality for for their use cases. Um, and the claim I want to make [0:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=51s) cases. Um, and the claim I want to make [0:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=51s) cases. Um, and the claim I want to make for this talk is that this is actually a [0:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=54s) for this talk is that this is actually a [0:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=54s) for this talk is that this is actually a losing strategy. Uh, when you end up [0:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=56s) losing strategy. Uh, when you end up [0:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=56s) losing strategy. Uh, when you end up having to compete on like not only the [0:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=59s) having to compete on like not only the [0:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=59s) having to compete on like not only the agent economics but also the experience [1:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=61s) agent economics but also the experience [1:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=61s) agent economics but also the experience of the agent and like these two [1:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=63s) of the agent and like these two [1:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=63s) of the agent and like these two competing priorities of like offering [1:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=66s) competing priorities of like offering [1:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=66s) competing priorities of like offering differentiated value. Uh, and I think [1:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=69s) differentiated value. Uh, and I think [1:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=69s) differentiated value. Uh, and I think this is what we hear from our customers [1:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=71s) this is what we hear from our customers [1:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=71s) this is what we hear from our customers who do fine-tuning. Um it's like the [1:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=74s) who do fine-tuning. Um it's like the [1:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=74s) who do fine-tuning. Um it's like the first the first area where I think um [1:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=77s) first the first area where I think um [1:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=77s) first the first area where I think um fine-tuning took off is is people who [1:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=80s) fine-tuning took off is is people who [1:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=80s) fine-tuning took off is is people who had agent deployments who um [1:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=84s) had agent deployments who um [1:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=84s) had agent deployments who um need reliable SLOs's like like they they [1:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=87s) need reliable SLOs's like like they they [1:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=87s) need reliable SLOs's like like they they need reliability SLOs's if they if they [1:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=89s) need reliability SLOs's if they if they [1:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=89s) need reliability SLOs's if they if they need those then they need to own their [1:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=90s) need those then they need to own their [1:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=90s) need those then they need to own their own inference and uh the open source [1:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=92s) own inference and uh the open source [1:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=92s) own inference and uh the open source models out of the box were not always [1:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=94s) models out of the box were not always [1:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=94s) models out of the box were not always capable of supporting their deployments [1:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=97s) capable of supporting their deployments [1:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=97s) capable of supporting their deployments uh without you know uh degrading in in [1:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=100s) uh without you know uh degrading in in [1:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=100s) uh without you know uh degrading in in quality and I think fine-tuning [1:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=102s) quality and I think fine-tuning [1:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=102s) quality and I think fine-tuning especially [1:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=104s) especially [1:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=104s) especially you know after these deployments became [1:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=106s) you know after these deployments became [1:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=106s) you know after these deployments became more mature I think was the only path to [1:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=108s) more mature I think was the only path to [1:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=108s) more mature I think was the only path to owning your own inference uh but more [1:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=111s) owning your own inference uh but more [1:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=111s) owning your own inference uh but more importantly kind of like pushing [1:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=112s) importantly kind of like pushing [1:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=112s) importantly kind of like pushing yourself up this curve in terms of [1:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=115s) yourself up this curve in terms of [1:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=115s) yourself up this curve in terms of latency quality [1:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=116s) latency quality [1:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=116s) latency quality [clears throat] [1:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=116s) [clears throat] [1:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=116s) [clears throat] uh and cost al together like if you if [1:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=119s) uh and cost al together like if you if [1:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=119s) uh and cost al together like if you if you do fine tuning you like own all [2:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=120s) you do fine tuning you like own all [2:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=120s) you do fine tuning you like own all these yourself kind of um there were [2:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=123s) these yourself kind of um there were [2:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=123s) these yourself kind of um there were other use cases though I think one we [2:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=125s) other use cases though I think one we [2:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=125s) other use cases though I think one we heard which was interesting was uh [2:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=128s) heard which was interesting was uh [2:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=128s) heard which was interesting was uh proprietary data sets that their agents [2:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=130s) proprietary data sets that their agents [2:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=130s) proprietary data sets that their agents actually needed to reason about and [2:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=132s) actually needed to reason about and [2:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=132s) actually needed to reason about and understand and take action on. But uh [2:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=135s) understand and take action on. But uh [2:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=135s) understand and take action on. But uh these these data sets were like not [2:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=136s) these these data sets were like not [2:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=136s) these these data sets were like not present in any pre-training set. And so [2:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=139s) present in any pre-training set. And so [2:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=139s) present in any pre-training set. And so there like out of the box sometimes they [2:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=142s) there like out of the box sometimes they [2:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=142s) there like out of the box sometimes they they would build agents on that had to [2:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=144s) they would build agents on that had to [2:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=144s) they would build agents on that had to deal with this type of data. And Claude [2:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=147s) deal with this type of data. And Claude [2:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=147s) deal with this type of data. And Claude Gbt just like would not work at all uh [2:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=150s) Gbt just like would not work at all uh [2:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=150s) Gbt just like would not work at all uh for these tasks. And so they actually [2:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=152s) for these tasks. And so they actually [2:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=152s) for these tasks. And so they actually just had no other alternative but to [2:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=155s) just had no other alternative but to [2:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=155s) just had no other alternative but to fine-tune open source models. [2:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=158s) fine-tune open source models. [2:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=158s) fine-tune open source models. And lastly, I think like there's an [2:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=160s) And lastly, I think like there's an [2:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=160s) And lastly, I think like there's an emerging attitude around fine-tuning, [2:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=162s) emerging attitude around fine-tuning, [2:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=162s) emerging attitude around fine-tuning, which is um in order to build [2:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=165s) which is um in order to build [2:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=165s) which is um in order to build differentiated value with agents, uh [2:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=168s) differentiated value with agents, uh [2:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=168s) differentiated value with agents, uh like more and more people are deciding [2:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=170s) like more and more people are deciding [2:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=170s) like more and more people are deciding to vertically integrate across all the [2:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=172s) to vertically integrate across all the [2:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=172s) to vertically integrate across all the different levels of the stack. Um these [2:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=175s) different levels of the stack. Um these [2:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=175s) different levels of the stack. Um these are this is most common in industries [2:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=177s) are this is most common in industries [2:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=177s) are this is most common in industries where uh people are already competing on [3:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=180s) where uh people are already competing on [3:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=180s) where uh people are already competing on like some metric of task intelligence [3:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=183s) like some metric of task intelligence [3:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=183s) like some metric of task intelligence per dollar uh like coding. But I think [3:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=186s) per dollar uh like coding. But I think [3:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=186s) per dollar uh like coding. But I think um what we should see as the industry [3:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=189s) um what we should see as the industry [3:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=189s) um what we should see as the industry matures outside of just coding is like [3:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=192s) matures outside of just coding is like [3:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=192s) matures outside of just coding is like we should see this trend kind of [3:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=194s) we should see this trend kind of [3:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=194s) we should see this trend kind of continue elsewhere. Um so I think that's [3:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=197s) continue elsewhere. Um so I think that's [3:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=197s) continue elsewhere. Um so I think that's that's why I think a lot of engineers [3:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=199s) that's why I think a lot of engineers [3:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=199s) that's why I think a lot of engineers have not really been fully bought in [3:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=201s) have not really been fully bought in [3:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=201s) have not really been fully bought in because the the latest model comes out [3:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=203s) because the the latest model comes out [3:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=203s) because the the latest model comes out and it's better than any model they [3:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=204s) and it's better than any model they [3:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=204s) and it's better than any model they could fine-tune from the last [3:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=205s) could fine-tune from the last [3:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=205s) could fine-tune from the last generation. But I think there's like a [3:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=207s) generation. But I think there's like a [3:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=207s) generation. But I think there's like a lot of different reasons people do [3:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=208s) lot of different reasons people do [3:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=208s) lot of different reasons people do finetuning and in the long haul I think [3:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=211s) finetuning and in the long haul I think [3:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=211s) finetuning and in the long haul I think uh yeah this this is only going to [3:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=213s) uh yeah this this is only going to [3:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=213s) uh yeah this this is only going to become more common. [3:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=215s) become more common. [3:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=215s) become more common. So you may have heard of fine-tuning in [3:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=217s) So you may have heard of fine-tuning in [3:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=217s) So you may have heard of fine-tuning in the context of RL or SFT. And the [3:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=220s) the context of RL or SFT. And the [3:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=220s) the context of RL or SFT. And the question is why is this talk about RL [3:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=222s) question is why is this talk about RL [3:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=222s) question is why is this talk about RL and not SFT? [3:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=224s) and not SFT? [3:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=224s) and not SFT? Um [3:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=226s) Um [3:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=226s) Um I think the idea that I want to [3:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=228s) I think the idea that I want to [3:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=228s) I think the idea that I want to communicate here is that uh if you're if [3:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=230s) communicate here is that uh if you're if [3:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=230s) communicate here is that uh if you're if you're doing SFT you're basically [3:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=231s) you're doing SFT you're basically [3:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=231s) you're doing SFT you're basically training on traces recorded uh during [3:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=234s) training on traces recorded uh during [3:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=234s) training on traces recorded uh during production. Um but the problem is that [3:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=237s) production. Um but the problem is that [3:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=237s) production. Um but the problem is that agents in particular have this like [4:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=240s) agents in particular have this like [4:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=240s) agents in particular have this like branching structure uh based on the [4:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=242s) branching structure uh based on the [4:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=242s) branching structure uh based on the number of turns that they run. Um [4:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=247s) number of turns that they run. Um [4:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=247s) number of turns that they run. Um so in production if you train on this [4:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=249s) so in production if you train on this [4:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=249s) so in production if you train on this particular trace uh later on when you [4:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=251s) particular trace uh later on when you [4:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=251s) particular trace uh later on when you deploy that model it may see like a [4:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=253s) deploy that model it may see like a [4:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=253s) deploy that model it may see like a completely different branch in the kind [4:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=256s) completely different branch in the kind [4:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=256s) completely different branch in the kind of like session like the the tree of [4:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=258s) of like session like the the tree of [4:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=258s) of like session like the the tree of possible sessions and uh this is quite [4:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=261s) possible sessions and uh this is quite [4:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=261s) possible sessions and uh this is quite bad. SFT is quite brittle in in this [4:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=265s) bad. SFT is quite brittle in in this [4:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=265s) bad. SFT is quite brittle in in this regard. So kind of like why RL has [4:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=268s) regard. So kind of like why RL has [4:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=268s) regard. So kind of like why RL has emerged as this like dominant paradigm [4:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=270s) emerged as this like dominant paradigm [4:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=270s) emerged as this like dominant paradigm of fine-tuning is that um in RL the [4:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=273s) of fine-tuning is that um in RL the [4:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=273s) of fine-tuning is that um in RL the model actually explores all these [4:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=274s) model actually explores all these [4:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=274s) model actually explores all these branches uh during training time. At [4:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=277s) branches uh during training time. At [4:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=278s) branches uh during training time. At least that's kind of the hope and when [4:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=279s) least that's kind of the hope and when [4:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=279s) least that's kind of the hope and when people talk about RL like generalizing [4:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=281s) people talk about RL like generalizing [4:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=282s) people talk about RL like generalizing better that's often what they mean. [4:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=286s) I think one reason why this trade-off [4:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=290s) I think one reason why this trade-off [4:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=290s) I think one reason why this trade-off has been hard for people is that they [4:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=292s) has been hard for people is that they [4:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=292s) has been hard for people is that they view like post- training and fine-tuning [4:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=294s) view like post- training and fine-tuning [4:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=294s) view like post- training and fine-tuning in RL as difficult. Um, but I think my [4:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=297s) in RL as difficult. Um, but I think my [4:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=297s) in RL as difficult. Um, but I think my claim here is actually that if you have [5:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=300s) claim here is actually that if you have [5:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=300s) claim here is actually that if you have an agent deployment in production, you [5:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=302s) an agent deployment in production, you [5:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=302s) an agent deployment in production, you probably already have the ingredients [5:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=304s) probably already have the ingredients [5:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=304s) probably already have the ingredients you need. Um, so if you're in [5:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=307s) you need. Um, so if you're in [5:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=307s) you need. Um, so if you're in production, you have an agent harness [5:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=308s) production, you have an agent harness [5:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=308s) production, you have an agent harness that that's deployed somewhere and [5:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=310s) that that's deployed somewhere and [5:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=310s) that that's deployed somewhere and hopefully you have eval. [5:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=312s) hopefully you have eval. [5:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=312s) hopefully you have eval. Maybe calling it eval is not necessarily [5:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=315s) Maybe calling it eval is not necessarily [5:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=315s) Maybe calling it eval is not necessarily required. It could be AB tests. It could [5:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=317s) required. It could be AB tests. It could [5:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=317s) required. It could be AB tests. It could be um basically any any way you do [5:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=319s) be um basically any any way you do [5:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=319s) be um basically any any way you do change management for this harness, for [5:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=320s) change management for this harness, for [5:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=320s) change management for this harness, for this deployment. Um but those two [5:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=323s) this deployment. Um but those two [5:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=323s) this deployment. Um but those two together give you the the tools you need [5:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=325s) together give you the the tools you need [5:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=325s) together give you the the tools you need to basically build out something that [5:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=327s) to basically build out something that [5:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=327s) to basically build out something that you can consider an environment, an RL [5:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=329s) you can consider an environment, an RL [5:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=329s) you can consider an environment, an RL environment. Um [5:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=331s) environment. Um [5:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=332s) environment. Um and why production traces are important [5:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=333s) and why production traces are important [5:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=333s) and why production traces are important for this is that you need to kind of get [5:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=336s) for this is that you need to kind of get [5:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=336s) for this is that you need to kind of get high quality tasks to train on in this [5:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=338s) high quality tasks to train on in this [5:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=338s) high quality tasks to train on in this environment from somewhere. And the best [5:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=340s) environment from somewhere. And the best [5:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=340s) environment from somewhere. And the best place often to do that is to [5:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=343s) place often to do that is to [5:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=343s) place often to do that is to synthetically generate these tasks from [5:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=345s) synthetically generate these tasks from [5:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=345s) synthetically generate these tasks from actual live production traces of things [5:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=347s) actual live production traces of things [5:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=347s) actual live production traces of things that your users are are doing um with [5:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=350s) that your users are are doing um with [5:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=350s) that your users are are doing um with with your product product and yeah kind [5:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=353s) with your product product and yeah kind [5:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=353s) with your product product and yeah kind of forming uh finding examples where [5:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=356s) of forming uh finding examples where [5:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=356s) of forming uh finding examples where it's failing, mining those and creating [5:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=358s) it's failing, mining those and creating [5:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=358s) it's failing, mining those and creating tasks around them so that fine-tuning [6:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=360s) tasks around them so that fine-tuning [6:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=360s) tasks around them so that fine-tuning actually gives you some chance of [6:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=361s) actually gives you some chance of [6:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=361s) actually gives you some chance of improving your product in those areas. [6:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=365s) improving your product in those areas. [6:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=365s) improving your product in those areas. And this is basically all you need to do [6:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=367s) And this is basically all you need to do [6:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=367s) And this is basically all you need to do RL. [6:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=370s) RL. [6:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=370s) RL. But then the next part part about it [6:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=371s) But then the next part part about it [6:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=372s) But then the next part part about it being hard is that like okay once just [6:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=373s) being hard is that like okay once just [6:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=373s) being hard is that like okay once just because you have an agent deployment [6:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=375s) because you have an agent deployment [6:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=375s) because you have an agent deployment doesn't mean that your team may be [6:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=376s) doesn't mean that your team may be [6:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=376s) doesn't mean that your team may be capable or maybe that's how you feel. Um [6:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=379s) capable or maybe that's how you feel. Um [6:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=379s) capable or maybe that's how you feel. Um and probably because RL has a reputation [6:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=382s) and probably because RL has a reputation [6:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=382s) and probably because RL has a reputation of just being difficult generally. Um, [6:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=384s) of just being difficult generally. Um, [6:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=384s) of just being difficult generally. Um, but I think over the last couple of at [6:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=387s) but I think over the last couple of at [6:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=387s) but I think over the last couple of at least over the last year, over the last [6:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=389s) least over the last year, over the last [6:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=389s) least over the last year, over the last several months, uh, like we have kind of [6:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=392s) several months, uh, like we have kind of [6:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=392s) several months, uh, like we have kind of figured out the science of of RL. And [6:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=396s) figured out the science of of RL. And [6:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=396s) figured out the science of of RL. And that's what I'm going to basically talk [6:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=398s) that's what I'm going to basically talk [6:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=398s) that's what I'm going to basically talk about today is that, uh, a particular RL [6:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=401s) about today is that, uh, a particular RL [6:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=401s) about today is that, uh, a particular RL recipe that we've had a lot of success [6:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=402s) recipe that we've had a lot of success [6:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=402s) recipe that we've had a lot of success with that I think simplifies and [6:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=405s) with that I think simplifies and [6:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=405s) with that I think simplifies and stabilizes a lot of what, you know, was [6:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=408s) stabilizes a lot of what, you know, was [6:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=408s) stabilizes a lot of what, you know, was difficult before. [6:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=411s) difficult before. [6:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=411s) difficult before. But first, okay, just to kind of recap [6:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=413s) But first, okay, just to kind of recap [6:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=413s) But first, okay, just to kind of recap how how does reinforcement learning [6:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=415s) how how does reinforcement learning [6:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=415s) how how does reinforcement learning work. Uh it's it's this loop [6:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=417s) work. Uh it's it's this loop [6:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=417s) work. Uh it's it's this loop mechanically. This is this is what's [6:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=419s) mechanically. This is this is what's [6:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=419s) mechanically. This is this is what's happening. You have a bunch of uh [7:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=422s) happening. You have a bunch of uh [7:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=422s) happening. You have a bunch of uh inference GPUs that are running your [7:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=424s) inference GPUs that are running your [7:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=424s) inference GPUs that are running your sampler aka the model that you're [7:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=425s) sampler aka the model that you're [7:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=425s) sampler aka the model that you're training. Um [7:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=428s) training. Um [7:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=428s) training. Um and those rollouts you you basically [7:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=430s) and those rollouts you you basically [7:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=430s) and those rollouts you you basically generate rollouts in your environment uh [7:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=432s) generate rollouts in your environment uh [7:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=432s) generate rollouts in your environment uh with your harness. Those rollouts get [7:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=434s) with your harness. Those rollouts get [7:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=434s) with your harness. Those rollouts get collected and fed to a trainer which is [7:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=437s) collected and fed to a trainer which is [7:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=437s) collected and fed to a trainer which is consuming all this data and updating the [7:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=439s) consuming all this data and updating the [7:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=439s) consuming all this data and updating the model uh with you know gradient descent [7:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=442s) model uh with you know gradient descent [7:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=442s) model uh with you know gradient descent and then uh that updated model you you [7:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=445s) and then uh that updated model you you [7:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=445s) and then uh that updated model you you broadcast those new weights to all the [7:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=447s) broadcast those new weights to all the [7:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=447s) broadcast those new weights to all the samplers and um over time you the [7:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=451s) samplers and um over time you the [7:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=451s) samplers and um over time you the rollouts you're generating are from this [7:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=452s) rollouts you're generating are from this [7:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=452s) rollouts you're generating are from this like continuously improving model. Uh so [7:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=454s) like continuously improving model. Uh so [7:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=454s) like continuously improving model. Uh so you're you're you're kind of like going [7:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=456s) you're you're you're kind of like going [7:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=456s) you're you're you're kind of like going back and forth between them and [7:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=457s) back and forth between them and [7:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=457s) back and forth between them and iterating and and that's how the model [7:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=459s) iterating and and that's how the model [7:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=459s) iterating and and that's how the model improves. [7:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=461s) improves. [7:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=461s) improves. Uh so it turns out that like the math of [7:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=464s) Uh so it turns out that like the math of [7:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=464s) Uh so it turns out that like the math of RL prefers this to be done sequentially. [7:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=467s) RL prefers this to be done sequentially. [7:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=467s) RL prefers this to be done sequentially. So first you do your rollouts then you [7:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=469s) So first you do your rollouts then you [7:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=469s) So first you do your rollouts then you train and update your model then you you [7:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=471s) train and update your model then you you [7:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=471s) train and update your model then you you take that model update send it back to [7:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=473s) take that model update send it back to [7:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=473s) take that model update send it back to your rollout side so that the new [7:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=475s) your rollout side so that the new [7:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=475s) your rollout side so that the new rollouts you generate are from that new [7:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=476s) rollouts you generate are from that new [7:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=476s) rollouts you generate are from that new version of the model. Uh but the problem [7:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=478s) version of the model. Uh but the problem [7:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=478s) version of the model. Uh but the problem is that this wastes a ton of idle [8:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=480s) is that this wastes a ton of idle [8:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=480s) is that this wastes a ton of idle compute. Um there are ways to like share [8:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=483s) compute. Um there are ways to like share [8:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=484s) compute. Um there are ways to like share the GPUs between the two but I think [8:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=486s) the GPUs between the two but I think [8:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=486s) the GPUs between the two but I think especially with agents. Um what's much [8:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=489s) especially with agents. Um what's much [8:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=489s) especially with agents. Um what's much better is that if you actually are [8:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=490s) better is that if you actually are [8:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=490s) better is that if you actually are continuously doing both in parallel [8:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=492s) continuously doing both in parallel [8:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=492s) continuously doing both in parallel asynchronously. [8:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=493s) asynchronously. [8:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=494s) asynchronously. Um this is better because you can like [8:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=496s) Um this is better because you can like [8:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=496s) Um this is better because you can like pack your GPUs, get high utilization [8:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=498s) pack your GPUs, get high utilization [8:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=498s) pack your GPUs, get high utilization from them and uh not waste money. But [8:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=501s) from them and uh not waste money. But [8:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=501s) from them and uh not waste money. But the downside is that um your rollouts [8:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=504s) the downside is that um your rollouts [8:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=504s) the downside is that um your rollouts that you generated were very likely from [8:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=505s) that you generated were very likely from [8:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=506s) that you generated were very likely from an old copy of the model. And this is [8:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=509s) an old copy of the model. And this is [8:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=509s) an old copy of the model. And this is what people refer to as staleness in [8:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=511s) what people refer to as staleness in [8:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=511s) what people refer to as staleness in asynchronous RL. Basically uh rollouts [8:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=514s) asynchronous RL. Basically uh rollouts [8:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=514s) asynchronous RL. Basically uh rollouts were gener generated by V1. uh the [8:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=517s) were gener generated by V1. uh the [8:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=517s) were gener generated by V1. uh the current model that you're training is [8:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=518s) current model that you're training is [8:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=518s) current model that you're training is V3. Are those rollouts still useful for [8:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=520s) V3. Are those rollouts still useful for [8:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=520s) V3. Are those rollouts still useful for V3 if the model's gotten better since [8:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=522s) V3 if the model's gotten better since [8:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=522s) V3 if the model's gotten better since then? Um and this is like one of the [8:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=525s) then? Um and this is like one of the [8:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=525s) then? Um and this is like one of the reasons why it was unstable. Another [8:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=527s) reasons why it was unstable. Another [8:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=527s) reasons why it was unstable. Another reason why it was unstable is actually [8:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=529s) reason why it was unstable is actually [8:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=529s) reason why it was unstable is actually purely at the systems level. There's a [8:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=530s) purely at the systems level. There's a [8:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=530s) purely at the systems level. There's a lot of details that can show up as [8:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=534s) lot of details that can show up as [8:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=534s) lot of details that can show up as differences in between what the trainer [8:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=537s) differences in between what the trainer [8:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=537s) differences in between what the trainer thinks it's training and what the [8:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=539s) thinks it's training and what the [8:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=539s) thinks it's training and what the samplers think we're training. [9:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=541s) samplers think we're training. [9:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=541s) samplers think we're training. uh basically at all these different [9:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=544s) uh basically at all these different [9:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=544s) uh basically at all these different levels of the stack there can be these [9:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=545s) levels of the stack there can be these [9:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=545s) levels of the stack there can be these like small differences in behavior and [9:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=549s) like small differences in behavior and [9:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=549s) like small differences in behavior and that means that like in in the math of [9:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=551s) that means that like in in the math of [9:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=551s) that means that like in in the math of RL I think the the idea is that both [9:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=553s) RL I think the the idea is that both [9:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=553s) RL I think the the idea is that both both sides of this think they're doing [9:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=554s) both sides of this think they're doing [9:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=554s) both sides of this think they're doing the same thing they think they're [9:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=555s) the same thing they think they're [9:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=556s) the same thing they think they're learning from the same data training [9:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=557s) learning from the same data training [9:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=557s) learning from the same data training like the same distribution but actually [9:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=559s) like the same distribution but actually [9:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=559s) like the same distribution but actually it's like slightly different and so the [9:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=562s) it's like slightly different and so the [9:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=562s) it's like slightly different and so the the solution that I'll present here is [9:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=565s) the solution that I'll present here is [9:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=565s) the solution that I'll present here is accounting for these differences um [9:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=567s) accounting for these differences um [9:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=567s) accounting for these differences um broadly with like kind of the simplest [9:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=569s) broadly with like kind of the simplest [9:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=569s) broadly with like kind of the simplest set of fixes that will uh correct for [9:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=572s) set of fixes that will uh correct for [9:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=572s) set of fixes that will uh correct for this mismatch without drowning out the [9:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=574s) this mismatch without drowning out the [9:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=574s) this mismatch without drowning out the signal of the the training loop. [9:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=578s) signal of the the training loop. [9:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=578s) signal of the the training loop. So yeah, what's the recipe? Here it is. [9:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=580s) So yeah, what's the recipe? Here it is. [9:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=580s) So yeah, what's the recipe? Here it is. Uh we start from gRPO. This is uh what [9:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=583s) Uh we start from gRPO. This is uh what [9:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=583s) Uh we start from gRPO. This is uh what DeepS introduced for Deepseek R1 and [9:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=586s) DeepS introduced for Deepseek R1 and [9:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=586s) DeepS introduced for Deepseek R1 and since then has become kind of a [9:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=587s) since then has become kind of a [9:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=587s) since then has become kind of a cornerstone of of doing RL on language [9:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=590s) cornerstone of of doing RL on language [9:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=590s) cornerstone of of doing RL on language models. Uh we add something called mass [9:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=593s) models. Uh we add something called mass [9:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=593s) models. Uh we add something called mass importance sampling. This is correcting [9:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=595s) importance sampling. This is correcting [9:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=595s) importance sampling. This is correcting for token-wise mismatch. And this [9:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=598s) for token-wise mismatch. And this [9:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=598s) for token-wise mismatch. And this mismatch is not just the mismatch from [10:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=600s) mismatch is not just the mismatch from [10:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=600s) mismatch is not just the mismatch from all the systems level stuff that I [10:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=602s) all the systems level stuff that I [10:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=602s) all the systems level stuff that I pointed out. Uh it's also from the [10:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=603s) pointed out. Uh it's also from the [10:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=603s) pointed out. Uh it's also from the staleness as well. So if an old version [10:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=606s) staleness as well. So if an old version [10:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=606s) staleness as well. So if an old version of the model would have sampled a [10:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=608s) of the model would have sampled a [10:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=608s) of the model would have sampled a different token here, uh the ratio [10:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=609s) different token here, uh the ratio [10:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=609s) different token here, uh the ratio between the two can be can be very [10:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=612s) between the two can be can be very [10:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=612s) between the two can be can be very different. And then uh this is a [10:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=613s) different. And then uh this is a [10:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=613s) different. And then uh this is a mismatch that we want to correct for [10:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=615s) mismatch that we want to correct for [10:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=615s) mismatch that we want to correct for somehow and and uh I won't go too much [10:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=618s) somehow and and uh I won't go too much [10:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=618s) somehow and and uh I won't go too much into details here about how these [10:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=619s) into details here about how these [10:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=619s) into details here about how these mechanisms work, but uh I will you know [10:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=622s) mechanisms work, but uh I will you know [10:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=622s) mechanisms work, but uh I will you know reference them. Um and then the second [10:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=625s) reference them. Um and then the second [10:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=625s) reference them. Um and then the second is score centering. So even though we're [10:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=627s) is score centering. So even though we're [10:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=627s) is score centering. So even though we're correcting for each token-wise mismatch, [10:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=630s) correcting for each token-wise mismatch, [10:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=630s) correcting for each token-wise mismatch, what ends up happening is over the [10:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=632s) what ends up happening is over the [10:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=632s) what ends up happening is over the course of training, whatever mismatch is [10:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=634s) course of training, whatever mismatch is [10:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=634s) course of training, whatever mismatch is still remaining accumulates and it kind [10:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=636s) still remaining accumulates and it kind [10:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=636s) still remaining accumulates and it kind of accumulates in the the weights [10:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=638s) of accumulates in the the weights [10:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=638s) of accumulates in the the weights themselves like the latest version of [10:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=640s) themselves like the latest version of [10:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=640s) themselves like the latest version of the model is slightly different than uh [10:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=642s) the model is slightly different than uh [10:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=642s) the model is slightly different than uh what it should be even though we've like [10:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=644s) what it should be even though we've like [10:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=644s) what it should be even though we've like corrected for the most obvious [10:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=646s) corrected for the most obvious [10:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=646s) corrected for the most obvious differences. Um and so this score [10:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=648s) differences. Um and so this score [10:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=648s) differences. Um and so this score centering approach is it basically [10:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=650s) centering approach is it basically [10:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=650s) centering approach is it basically removes this drift over the course of [10:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=652s) removes this drift over the course of [10:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=652s) removes this drift over the course of training uh not just at each token or at [10:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=655s) training uh not just at each token or at [10:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=655s) training uh not just at each token or at each step. [10:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=656s) each step. [10:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=656s) each step. um it kind of like builds up a term that [10:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=658s) um it kind of like builds up a term that [10:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=658s) um it kind of like builds up a term that accumulates with the drift and removes [11:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=660s) accumulates with the drift and removes [11:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=660s) accumulates with the drift and removes it. Um and then the last thing is top [11:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=663s) it. Um and then the last thing is top [11:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=663s) it. Um and then the last thing is top heat sampling uh with mass replay. So [11:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=666s) heat sampling uh with mass replay. So [11:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=666s) heat sampling uh with mass replay. So top p is is one of the parameters you [11:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=667s) top p is is one of the parameters you [11:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=668s) top p is is one of the parameters you use when you sample from a language [11:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=669s) use when you sample from a language [11:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=669s) use when you sample from a language model. Basically says only consider the [11:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=671s) model. Basically says only consider the [11:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=671s) model. Basically says only consider the top p of the of the distribution. P [11:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=674s) top p of the of the distribution. P [11:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=674s) top p of the of the distribution. P being like 0.98 0.95 basically only the [11:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=677s) being like 0.98 0.95 basically only the [11:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=677s) being like 0.98 0.95 basically only the tokens that have probability mass [11:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=679s) tokens that have probability mass [11:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=679s) tokens that have probability mass greater than 95 uh or greater than 5%. [11:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=682s) greater than 95 uh or greater than 5%. [11:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=682s) greater than 95 uh or greater than 5%. Um, and so it just removes a bunch of [11:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=684s) Um, and so it just removes a bunch of [11:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=684s) Um, and so it just removes a bunch of very rare, likely bad tokens that can [11:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=688s) very rare, likely bad tokens that can [11:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=688s) very rare, likely bad tokens that can just never be sampled from. And so if [11:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=690s) just never be sampled from. And so if [11:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=690s) just never be sampled from. And so if your sampler decides to only sample from [11:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=692s) your sampler decides to only sample from [11:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=692s) your sampler decides to only sample from these tokens, but your trainer decides [11:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=694s) these tokens, but your trainer decides [11:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=694s) these tokens, but your trainer decides unluckily to sample one of these rare [11:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=696s) unluckily to sample one of these rare [11:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=696s) unluckily to sample one of these rare tokens, this mass mismatch can be [11:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=698s) tokens, this mass mismatch can be [11:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=698s) tokens, this mass mismatch can be basically catastrophic if you don't [11:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=699s) basically catastrophic if you don't [11:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=699s) basically catastrophic if you don't correct for it. So we correct for it. Um [11:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=702s) correct for it. So we correct for it. Um [11:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=702s) correct for it. So we correct for it. Um and it turns out actually in the math of [11:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=703s) and it turns out actually in the math of [11:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=703s) and it turns out actually in the math of score centering that if you're doing top [11:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=705s) score centering that if you're doing top [11:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=706s) score centering that if you're doing top sampling like this that actually the the [11:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=708s) sampling like this that actually the the [11:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=708s) sampling like this that actually the the technique we we use to do score [11:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=709s) technique we we use to do score [11:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=709s) technique we we use to do score centering uh is exact instead of [11:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=711s) centering uh is exact instead of [11:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=711s) centering uh is exact instead of approximate because if you don't do top [11:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=713s) approximate because if you don't do top [11:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=713s) approximate because if you don't do top it's actually only an approximate [11:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=715s) it's actually only an approximate [11:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=715s) it's actually only an approximate correction for drift. So all this is to [11:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=717s) correction for drift. So all this is to [11:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=717s) correction for drift. So all this is to say that uh with this recipe uh things [12:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=720s) say that uh with this recipe uh things [12:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=720s) say that uh with this recipe uh things get much much better much more stable. [12:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=722s) get much much better much more stable. [12:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=722s) get much much better much more stable. Um [12:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=724s) Um [12:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=724s) Um and this is this is basically just an [12:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=726s) and this is this is basically just an [12:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=726s) and this is this is basically just an ablation that we did on quen 3.635b 6 [12:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=729s) ablation that we did on quen 3.635b 6 [12:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=729s) ablation that we did on quen 3.635b 6 35B of thee. Um, it's trained on the [12:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=733s) 35B of thee. Um, it's trained on the [12:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=733s) 35B of thee. Um, it's trained on the MIMO code environments with evalu [12:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=738s) subset. Um, basically what this shows is [12:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=741s) subset. Um, basically what this shows is [12:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=741s) subset. Um, basically what this shows is an ablation of these different [12:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=742s) an ablation of these different [12:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=742s) an ablation of these different techniques uh putting them all together [12:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=744s) techniques uh putting them all together [12:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=744s) techniques uh putting them all together showing like much more stable training [12:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=746s) showing like much more stable training [12:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=746s) showing like much more stable training uh than any of uh the alternatives [12:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=750s) uh than any of uh the alternatives [12:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=750s) uh than any of uh the alternatives that we've tested here. Um, so yeah, I [12:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=754s) that we've tested here. Um, so yeah, I [12:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=754s) that we've tested here. Um, so yeah, I think the that's all well and good, but [12:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=756s) think the that's all well and good, but [12:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=756s) think the that's all well and good, but I guess the question is like how far can [12:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=757s) I guess the question is like how far can [12:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=757s) I guess the question is like how far can we push this recipe? Does it actually [12:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=758s) we push this recipe? Does it actually [12:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=758s) we push this recipe? Does it actually scale? Does it does it work for broad [12:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=760s) scale? Does it does it work for broad [12:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=760s) scale? Does it does it work for broad problems? Before I talk about that, I [12:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=763s) problems? Before I talk about that, I [12:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=763s) problems? Before I talk about that, I think I want to talk briefly about why [12:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=765s) think I want to talk briefly about why [12:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=765s) think I want to talk briefly about why we care about RL at modal and like some [12:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=767s) we care about RL at modal and like some [12:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=767s) we care about RL at modal and like some things about doing RL on modal are [12:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=770s) things about doing RL on modal are [12:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=770s) things about doing RL on modal are different. I kind of want to just like [12:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=771s) different. I kind of want to just like [12:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=771s) different. I kind of want to just like get into that a little bit first. Um, so [12:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=773s) get into that a little bit first. Um, so [12:53](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=773s) get into that a little bit first. Um, so yeah, first why modal for agent RL in [12:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=775s) yeah, first why modal for agent RL in [12:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=775s) yeah, first why modal for agent RL in particular? Well, we do training. We [12:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=777s) particular? Well, we do training. We [12:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=778s) particular? Well, we do training. We just released uh clusters as GA last [13:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=781s) just released uh clusters as GA last [13:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=781s) just released uh clusters as GA last week I think or week before. Um we also [13:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=784s) week I think or week before. Um we also [13:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=784s) week I think or week before. Um we also do inference really well. We have a [13:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=785s) do inference really well. We have a [13:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=785s) do inference really well. We have a bunch of endpoints. Uh we've we've done [13:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=787s) bunch of endpoints. Uh we've we've done [13:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=787s) bunch of endpoints. Uh we've we've done a a lot of the hard work around doing LM [13:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=789s) a a lot of the hard work around doing LM [13:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=789s) a a lot of the hard work around doing LM inference at scale. Um and the [13:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=791s) inference at scale. Um and the [13:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=791s) inference at scale. Um and the environments you need they run in modal [13:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=793s) environments you need they run in modal [13:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=793s) environments you need they run in modal sandboxes. So like just naively we have [13:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=795s) sandboxes. So like just naively we have [13:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=795s) sandboxes. So like just naively we have all of the kind of infrared layers like [13:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=798s) all of the kind of infrared layers like [13:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=798s) all of the kind of infrared layers like or infrop primitives that you'd want to [13:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=800s) or infrop primitives that you'd want to [13:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=800s) or infrop primitives that you'd want to be able to compose into like a full [13:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=802s) be able to compose into like a full [13:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=802s) be able to compose into like a full production agent RL uh training system. [13:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=806s) production agent RL uh training system. [13:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=806s) production agent RL uh training system. The problem, the only problem actually [13:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=809s) The problem, the only problem actually [13:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=809s) The problem, the only problem actually uh because all that sounds great, but [13:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=811s) uh because all that sounds great, but [13:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=811s) uh because all that sounds great, but the only problem is that our capacity [13:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=813s) the only problem is that our capacity [13:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=813s) the only problem is that our capacity model is very different compared to [13:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=814s) model is very different compared to [13:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=814s) model is very different compared to other people. Um the reason for that is [13:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=817s) other people. Um the reason for that is [13:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=817s) other people. Um the reason for that is we aggregate tons of capacity across [13:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=819s) we aggregate tons of capacity across [13:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=819s) we aggregate tons of capacity across many different providers here. There's [13:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=821s) many different providers here. There's [13:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=821s) many different providers here. There's three pictured, but in reality it's like [13:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=823s) three pictured, but in reality it's like [13:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=823s) three pictured, but in reality it's like 15 or 20 at this point. Um we we run all [13:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=826s) 15 or 20 at this point. Um we we run all [13:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=826s) 15 or 20 at this point. Um we we run all the hyperscalers, but we also run on a [13:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=827s) the hyperscalers, but we also run on a [13:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=827s) the hyperscalers, but we also run on a bunch of NeoClouds. Um and we handle all [13:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=830s) bunch of NeoClouds. Um and we handle all [13:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=830s) bunch of NeoClouds. Um and we handle all of the kind of like GPU health and [13:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=832s) of the kind of like GPU health and [13:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=832s) of the kind of like GPU health and reliability concerns. Uh but [13:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=837s) reliability concerns. Uh but [13:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=837s) reliability concerns. Uh but this is great but uh it's great for [13:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=839s) this is great but uh it's great for [13:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=839s) this is great but uh it's great for reliability. It's great for kind of like [14:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=840s) reliability. It's great for kind of like [14:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=840s) reliability. It's great for kind of like fast and flexible scheduling. It gets us [14:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=842s) fast and flexible scheduling. It gets us [14:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=842s) fast and flexible scheduling. It gets us around compute shortages. Uh but it also [14:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=845s) around compute shortages. Uh but it also [14:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=845s) around compute shortages. Uh but it also fragments our compute pool. So uh if you [14:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=848s) fragments our compute pool. So uh if you [14:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=848s) fragments our compute pool. So uh if you want to schedule large training [14:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=849s) want to schedule large training [14:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=849s) want to schedule large training clusters, it's actually quite difficult [14:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=851s) clusters, it's actually quite difficult [14:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=851s) clusters, it's actually quite difficult in this like multi-provider multi-reion [14:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=854s) in this like multi-provider multi-reion [14:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=854s) in this like multi-provider multi-reion fleet. Uh we've engineered it so that we [14:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=857s) fleet. Uh we've engineered it so that we [14:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=857s) fleet. Uh we've engineered it so that we can schedule clusters pretty quickly up [14:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=859s) can schedule clusters pretty quickly up [14:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=859s) can schedule clusters pretty quickly up to like four nodes or eight nodes. Uh [14:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=861s) to like four nodes or eight nodes. Uh [14:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=861s) to like four nodes or eight nodes. Uh but once you get past that to 16 nodes [14:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=863s) but once you get past that to 16 nodes [14:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=863s) but once you get past that to 16 nodes or 32 nodes uh it gets pretty painful. [14:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=865s) or 32 nodes uh it gets pretty painful. [14:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=865s) or 32 nodes uh it gets pretty painful. It takes a really long time to find like [14:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=867s) It takes a really long time to find like [14:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=867s) It takes a really long time to find like a contiguous block of all these nodes. [14:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=870s) a contiguous block of all these nodes. [14:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=870s) a contiguous block of all these nodes. Um and in RL this is particularly bad [14:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=873s) Um and in RL this is particularly bad [14:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=873s) Um and in RL this is particularly bad because in RL we generally have [14:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=875s) because in RL we generally have [14:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=875s) because in RL we generally have especially for agent RL we have like two [14:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=877s) especially for agent RL we have like two [14:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=877s) especially for agent RL we have like two two inference GPUs for every training [14:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=879s) two inference GPUs for every training [14:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=879s) two inference GPUs for every training GPU. So 2/3ish [14:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=881s) GPU. So 2/3ish [14:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=881s) GPU. So 2/3ish of your cluster is actually just stuck [14:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=883s) of your cluster is actually just stuck [14:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=883s) of your cluster is actually just stuck doing inference. Um, so if you're [14:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=886s) doing inference. Um, so if you're [14:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=886s) doing inference. Um, so if you're scheduling an eight node cluster max, [14:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=888s) scheduling an eight node cluster max, [14:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=888s) scheduling an eight node cluster max, you're probably only using two or three [14:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=889s) you're probably only using two or three [14:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=889s) you're probably only using two or three of those nodes for for the trainer side [14:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=891s) of those nodes for for the trainer side [14:51](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=891s) of those nodes for for the trainer side and most of it's just spent doing [14:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=892s) and most of it's just spent doing [14:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=892s) and most of it's just spent doing inference. [14:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=894s) inference. [14:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=894s) inference. Um, it'd be really great if we could [14:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=896s) Um, it'd be really great if we could [14:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=896s) Um, it'd be really great if we could move these inference GPUs out of the [14:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=898s) move these inference GPUs out of the [14:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=898s) move these inference GPUs out of the cluster. [15:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=901s) cluster. [15:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=901s) cluster. And one reason why we think we can do [15:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=903s) And one reason why we think we can do [15:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=903s) And one reason why we think we can do this is because uh during RL the weight [15:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=906s) this is because uh during RL the weight [15:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=906s) this is because uh during RL the weight updates, the actual like gradients that [15:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=908s) updates, the actual like gradients that [15:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=908s) updates, the actual like gradients that you're applying to your model end up [15:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=910s) you're applying to your model end up [15:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=910s) you're applying to your model end up being incredibly sparse. And there's uh [15:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=912s) being incredibly sparse. And there's uh [15:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=912s) being incredibly sparse. And there's uh two reasons for that. But um yeah, [15:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=916s) two reasons for that. But um yeah, [15:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=916s) two reasons for that. But um yeah, essentially if you if you take the model [15:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=917s) essentially if you if you take the model [15:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=917s) essentially if you if you take the model at step t and subtract the the new model [15:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=920s) at step t and subtract the the new model [15:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=920s) at step t and subtract the the new model t+1 from it or sorry take the model at t [15:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=923s) t+1 from it or sorry take the model at t [15:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=923s) t+1 from it or sorry take the model at t plus1 subtract t uh you get this delta. [15:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=926s) plus1 subtract t uh you get this delta. [15:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=926s) plus1 subtract t uh you get this delta. And it turns out that even though the [15:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=927s) And it turns out that even though the [15:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=927s) And it turns out that even though the full model weights at each step are very [15:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=929s) full model weights at each step are very [15:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=929s) full model weights at each step are very dense and very large. Uh the updates [15:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=932s) dense and very large. Uh the updates [15:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=932s) dense and very large. Uh the updates themselves are incredibly sparse. 100x [15:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=934s) themselves are incredibly sparse. 100x [15:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=934s) themselves are incredibly sparse. 100x or a thousandx as sparse. [15:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=938s) or a thousandx as sparse. [15:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=938s) or a thousandx as sparse. So you can actually just feed this to [15:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=939s) So you can actually just feed this to [15:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=939s) So you can actually just feed this to offtheshelf compression. And it turns [15:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=941s) offtheshelf compression. And it turns [15:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=941s) offtheshelf compression. And it turns out that the amount of data you need to [15:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=943s) out that the amount of data you need to [15:43](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=943s) out that the amount of data you need to communicate from the train uh yeah from [15:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=945s) communicate from the train uh yeah from [15:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=945s) communicate from the train uh yeah from the trainer to the rollout engines goes [15:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=948s) the trainer to the rollout engines goes [15:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=948s) the trainer to the rollout engines goes down by a factor of a thousand. This [15:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=950s) down by a factor of a thousand. This [15:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=950s) down by a factor of a thousand. This means that you can actually move your [15:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=952s) means that you can actually move your [15:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=952s) means that you can actually move your inference GPUs outside of the cluster. [15:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=955s) inference GPUs outside of the cluster. [15:55](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=955s) inference GPUs outside of the cluster. And so actually for the for the run that [15:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=957s) And so actually for the for the run that [15:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=957s) And so actually for the for the run that I mentioned uh of coin 3.6 we ran the [16:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=961s) I mentioned uh of coin 3.6 we ran the [16:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=961s) I mentioned uh of coin 3.6 we ran the training cluster I think it it landed [16:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=962s) training cluster I think it it landed [16:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=962s) training cluster I think it it landed somewhere in GCP but the inference GPUs [16:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=965s) somewhere in GCP but the inference GPUs [16:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=965s) somewhere in GCP but the inference GPUs actually ran all across our fleet and [16:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=966s) actually ran all across our fleet and [16:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=966s) actually ran all across our fleet and all the different hyperscalers and [16:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=967s) all the different hyperscalers and [16:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=967s) all the different hyperscalers and neoclouds that we run across. Um and it [16:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=970s) neoclouds that we run across. Um and it [16:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=970s) neoclouds that we run across. Um and it turns out with this like sparse delta [16:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=972s) turns out with this like sparse delta [16:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=972s) turns out with this like sparse delta compression technique you can do this. [16:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=974s) compression technique you can do this. [16:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=974s) compression technique you can do this. Uh it works well and it does not [16:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=976s) Uh it works well and it does not [16:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=976s) Uh it works well and it does not introduce like bad performance. Um [16:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=981s) introduce like bad performance. Um [16:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=981s) introduce like bad performance. Um not only that we we actually fragment [16:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=983s) not only that we we actually fragment [16:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=983s) not only that we we actually fragment our rollout fleet across different G GPU [16:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=985s) our rollout fleet across different G GPU [16:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=985s) our rollout fleet across different G GPU types. So again because the we have [16:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=988s) types. So again because the we have [16:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=988s) types. So again because the we have fragmentation due to all the different [16:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=989s) fragmentation due to all the different [16:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=990s) fragmentation due to all the different providers we run on. Uh but we also have [16:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=992s) providers we run on. Uh but we also have [16:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=992s) providers we run on. Uh but we also have fragmentation because we have a lot of [16:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=993s) fragmentation because we have a lot of [16:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=993s) fragmentation because we have a lot of different GPUs that different customers [16:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=995s) different GPUs that different customers [16:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=995s) different GPUs that different customers want to use. Uh it'd be really great if [16:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=997s) want to use. Uh it'd be really great if [16:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=997s) want to use. Uh it'd be really great if we could leverage as many of them as [16:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=998s) we could leverage as many of them as [16:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=998s) we could leverage as many of them as possible for RL runs and that's that's [16:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1000s) possible for RL runs and that's that's [16:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1000s) possible for RL runs and that's that's what we've done. Um each of these GPU [16:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1004s) what we've done. Um each of these GPU [16:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1004s) what we've done. Um each of these GPU types they they increase the mismatch [16:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1006s) types they they increase the mismatch [16:46](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1006s) types they they increase the mismatch potentially. So if the train is running [16:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1007s) potentially. So if the train is running [16:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1007s) potentially. So if the train is running in B200 uh but your rollouts are running [16:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1009s) in B200 uh but your rollouts are running [16:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1010s) in B200 uh but your rollouts are running in H100 or A100 there are hardware level [16:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1012s) in H100 or A100 there are hardware level [16:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1012s) in H100 or A100 there are hardware level differences in the the kernels the math [16:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1014s) differences in the the kernels the math [16:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1014s) differences in the the kernels the math you can run that implement the model. [16:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1016s) you can run that implement the model. [16:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1016s) you can run that implement the model. And so this like increases more and more [16:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1018s) And so this like increases more and more [16:58](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1018s) And so this like increases more and more mismatch which traditionally should [17:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1020s) mismatch which traditionally should [17:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1020s) mismatch which traditionally should cause your training runs to destabilize. [17:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1023s) cause your training runs to destabilize. [17:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1023s) cause your training runs to destabilize. Um this is just kind of like a graph to [17:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1026s) Um this is just kind of like a graph to [17:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1026s) Um this is just kind of like a graph to show uh how many requests are being [17:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1028s) show uh how many requests are being [17:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1028s) show uh how many requests are being processed by each GPU type in the run we [17:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1030s) processed by each GPU type in the run we [17:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1030s) processed by each GPU type in the run we did over the course of training. [17:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1035s) So actually the recipe that I mentioned [17:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1038s) So actually the recipe that I mentioned [17:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1038s) So actually the recipe that I mentioned uh is capable of handling this amount of [17:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1041s) uh is capable of handling this amount of [17:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1041s) uh is capable of handling this amount of mismatch that we throw at it which uh [17:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1045s) mismatch that we throw at it which uh [17:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1045s) mismatch that we throw at it which uh yeah I think is like kind of [17:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1046s) yeah I think is like kind of [17:26](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1046s) yeah I think is like kind of unprecedented. I think this just goes to [17:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1047s) unprecedented. I think this just goes to [17:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1048s) unprecedented. I think this just goes to show how much we've figured out the [17:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1050s) show how much we've figured out the [17:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1050s) show how much we've figured out the science of RL over the past year or two. [17:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1053s) science of RL over the past year or two. [17:33](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1053s) science of RL over the past year or two. Um basically this shows that you have up [17:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1057s) Um basically this shows that you have up [17:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1057s) Um basically this shows that you have up to like four or five times as much [17:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1059s) to like four or five times as much [17:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1059s) to like four or five times as much mismatch. Uh but the training reward and [17:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1062s) mismatch. Uh but the training reward and [17:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1062s) mismatch. Uh but the training reward and gradient norm everything is is training [17:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1065s) gradient norm everything is is training [17:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1065s) gradient norm everything is is training as stably as it was uh in the [17:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1068s) as stably as it was uh in the [17:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1068s) as stably as it was uh in the homogeneous case which is basically [17:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1070s) homogeneous case which is basically [17:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1070s) homogeneous case which is basically running training on B200s and inference [17:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1072s) running training on B200s and inference [17:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1072s) running training on B200s and inference on B200s. [17:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1076s) And kind of a bonus, like a side effect [17:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1077s) And kind of a bonus, like a side effect [17:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1077s) And kind of a bonus, like a side effect of this is that uh the rollouts that we [18:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1080s) of this is that uh the rollouts that we [18:00](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1080s) of this is that uh the rollouts that we run for this full training run are [18:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1081s) run for this full training run are [18:02](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1082s) run for this full training run are actually 15% cheaper. And that's because [18:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1084s) actually 15% cheaper. And that's because [18:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1084s) actually 15% cheaper. And that's because the the demand for the the high-end GPUs [18:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1087s) the the demand for the the high-end GPUs [18:07](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1087s) the the demand for the the high-end GPUs like B300s uh means that the price is [18:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1089s) like B300s uh means that the price is [18:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1089s) like B300s uh means that the price is just generally higher. But if you can [18:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1091s) just generally higher. But if you can [18:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1091s) just generally higher. But if you can actually run your rollouts cheaply on on [18:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1093s) actually run your rollouts cheaply on on [18:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1093s) actually run your rollouts cheaply on on cheaper hardware that there's less [18:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1094s) cheaper hardware that there's less [18:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1094s) cheaper hardware that there's less demand for like the overall cost of [18:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1096s) demand for like the overall cost of [18:16](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1096s) demand for like the overall cost of doing this run is is significantly [18:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1098s) doing this run is is significantly [18:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1098s) doing this run is is significantly lower. This is actually without any [18:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1100s) lower. This is actually without any [18:20](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1100s) lower. This is actually without any inference tuning for each GPU type. [18:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1102s) inference tuning for each GPU type. [18:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1102s) inference tuning for each GPU type. we're just doing like the basic [18:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1103s) we're just doing like the basic [18:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1103s) we're just doing like the basic off-the-shelf inference recipe for each [18:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1105s) off-the-shelf inference recipe for each [18:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1105s) off-the-shelf inference recipe for each GPU. If you actually decided to uh you [18:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1109s) GPU. If you actually decided to uh you [18:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1109s) GPU. If you actually decided to uh you know optimize each individual deployment [18:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1112s) know optimize each individual deployment [18:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1112s) know optimize each individual deployment for each GPU type with for example [18:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1114s) for each GPU type with for example [18:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1114s) for each GPU type with for example different levels of quantization in the [18:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1116s) different levels of quantization in the [18:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1116s) different levels of quantization in the model um you could actually probably [18:37](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1117s) model um you could actually probably [18:38](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1118s) model um you could actually probably make this significantly more cheap. We [18:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1119s) make this significantly more cheap. We [18:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1119s) make this significantly more cheap. We just uh this is not the focus of this [18:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1121s) just uh this is not the focus of this [18:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1121s) just uh this is not the focus of this work but uh but yeah oh I did also maybe [18:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1125s) work but uh but yeah oh I did also maybe [18:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1125s) work but uh but yeah oh I did also maybe I forgot to mention but in this case [18:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1128s) I forgot to mention but in this case [18:48](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1128s) I forgot to mention but in this case there are cases where like for example [18:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1130s) there are cases where like for example [18:50](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1130s) there are cases where like for example H200 we have two different pools of [18:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1132s) H200 we have two different pools of [18:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1132s) H200 we have two different pools of H200s that's because we run two [18:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1134s) H200s that's because we run two [18:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1134s) H200s that's because we run two different uh quantizations of the model [18:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1136s) different uh quantizations of the model [18:56](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1136s) different uh quantizations of the model uh on H200. So we have one H200 pool [18:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1139s) uh on H200. So we have one H200 pool [18:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1139s) uh on H200. So we have one H200 pool doing it at BF16 which is like the the [19:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1141s) doing it at BF16 which is like the the [19:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1141s) doing it at BF16 which is like the the safest way to do it but we have another [19:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1143s) safest way to do it but we have another [19:03](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1143s) safest way to do it but we have another running at FP8. Uh and again like the [19:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1146s) running at FP8. Uh and again like the [19:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1146s) running at FP8. Uh and again like the the correction that we do can like drown [19:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1148s) the correction that we do can like drown [19:08](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1148s) the correction that we do can like drown out the mismatch that's introduced by [19:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1150s) out the mismatch that's introduced by [19:10](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1150s) out the mismatch that's introduced by even excessive quantization of the model [19:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1152s) even excessive quantization of the model [19:12](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1152s) even excessive quantization of the model and still learn from these quantized [19:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1154s) and still learn from these quantized [19:14](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1154s) and still learn from these quantized rollouts. Um so yeah I think the the [19:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1158s) rollouts. Um so yeah I think the the [19:18](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1158s) rollouts. Um so yeah I think the the point I'm trying to make here is that um [19:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1161s) point I'm trying to make here is that um [19:22](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1162s) point I'm trying to make here is that um like we've released this there's a blog [19:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1163s) like we've released this there's a blog [19:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1163s) like we've released this there's a blog post that where we go into like all the [19:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1164s) post that where we go into like all the [19:24](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1164s) post that where we go into like all the technical detail. We'll be releasing the [19:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1167s) technical detail. We'll be releasing the [19:27](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1167s) technical detail. We'll be releasing the source code for it very soon. uh it's [19:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1169s) source code for it very soon. uh it's [19:29](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1169s) source code for it very soon. uh it's based on miles and silang the the [19:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1171s) based on miles and silang the the [19:31](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1171s) based on miles and silang the the reinforcement learning framework in the [19:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1172s) reinforcement learning framework in the [19:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1172s) reinforcement learning framework in the inference engine and um I think the the [19:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1176s) inference engine and um I think the the [19:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1176s) inference engine and um I think the the takeaway is that um I do believe that if [19:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1179s) takeaway is that um I do believe that if [19:39](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1179s) takeaway is that um I do believe that if you want to be competitive in a world [19:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1181s) you want to be competitive in a world [19:41](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1181s) you want to be competitive in a world where a bunch of different competitors [19:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1182s) where a bunch of different competitors [19:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1182s) where a bunch of different competitors are running agent stacks you do need to [19:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1185s) are running agent stacks you do need to [19:45](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1185s) are running agent stacks you do need to some degree own the full stack yourself [19:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1187s) some degree own the full stack yourself [19:47](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1187s) some degree own the full stack yourself or at least like build competency in [19:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1189s) or at least like build competency in [19:49](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1189s) or at least like build competency in every area of the stack. Um I think a [19:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1192s) every area of the stack. Um I think a [19:52](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1192s) every area of the stack. Um I think a lot of work traditionally ends at this [19:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1194s) lot of work traditionally ends at this [19:54](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1194s) lot of work traditionally ends at this agent harness but actually um the [19:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1197s) agent harness but actually um the [19:57](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1197s) agent harness but actually um the vertical integration across all of this [19:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1199s) vertical integration across all of this [19:59](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1199s) vertical integration across all of this is just extremely important in a in a [20:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1201s) is just extremely important in a in a [20:01](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1201s) is just extremely important in a in a hyperco competitive uh industry. We see [20:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1204s) hyperco competitive uh industry. We see [20:04](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1204s) hyperco competitive uh industry. We see this again in certain industries like [20:05](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1205s) this again in certain industries like [20:06](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1206s) this again in certain industries like coding. Um but I think it we we also see [20:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1209s) coding. Um but I think it we we also see [20:09](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1209s) coding. Um but I think it we we also see it in other industries and it's only [20:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1211s) it in other industries and it's only [20:11](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1211s) it in other industries and it's only going to keep growing I think. Um, and [20:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1213s) going to keep growing I think. Um, and [20:13](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1213s) going to keep growing I think. Um, and of course like if you're building agent [20:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1215s) of course like if you're building agent [20:15](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1215s) of course like if you're building agent deployments, you own everything up the [20:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1217s) deployments, you own everything up the [20:17](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1217s) deployments, you own everything up the stack as well. And vertically [20:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1219s) stack as well. And vertically [20:19](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1219s) stack as well. And vertically integrating across all of this is [20:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1221s) integrating across all of this is [20:21](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1221s) integrating across all of this is extremely important uh for maintaining [20:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1223s) extremely important uh for maintaining [20:23](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1223s) extremely important uh for maintaining kind of competency and and competitive [20:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1225s) kind of competency and and competitive [20:25](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1225s) kind of competency and and competitive edge. Um, yeah, you should own your own [20:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1228s) edge. Um, yeah, you should own your own [20:28](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1228s) edge. Um, yeah, you should own your own agent RL and it's actually possible now [20:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1230s) agent RL and it's actually possible now [20:30](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1230s) agent RL and it's actually possible now because the science has been figured out [20:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1232s) because the science has been figured out [20:32](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1232s) because the science has been figured out and there are very good tools and very [20:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1234s) and there are very good tools and very [20:34](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1234s) and there are very good tools and very good infrastructure that lets you do [20:35](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1235s) good infrastructure that lets you do [20:36](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1236s) good infrastructure that lets you do that. Um, that's all I got. So thanks. [20:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1240s) that. Um, that's all I got. So thanks. [20:40](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1240s) that. Um, that's all I got. So thanks. We're we're working on hard problems. [20:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1242s) We're we're working on hard problems. [20:42](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1242s) We're we're working on hard problems. Come visit our booth and and uh talk to [20:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1244s) Come visit our booth and and uh talk to [20:44](https://www.youtube.com/watch?v=3nDMYV-QHLs&t=1244s) Come visit our booth and and uh talk to us.