Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal

A new model comes out. The agent team runs evals, A/B tests it, and rebuilds parts of the harness around whichever model offers the best quality at an acceptable price. Jason Mancuso of Modal sees a limit to that strategy. The team can choose among the combinations of capability, latency, and cost that providers offer. Its users care about how well the agent performs their particular task and what it costs to get it done.

Modal customers had several reasons to take on the work of fine-tuning.

• Reliability. Some teams wanted to run inference themselves to meet a service-level objective. An open model they could serve sometimes fell short on their tasks. Fine-tuning gave them a way to recover quality while retaining control over serving latency and cost.
• Proprietary data. Some agents had to interpret private datasets and act on what they found. General-purpose models had never encountered those datasets during pretraining and could fail at the task altogether. Those teams turned to fine-tuning open models.
• Task performance per dollar. In a competitive category such as coding, solving more of the user's task for the same spend matters. A team that can change both the harness and the model has another way to improve that ratio.

Waiting for the next release has a real appeal. A new general-purpose model can outperform one a team painstakingly tuned from the previous generation. Mancuso expects more teams to invest anyway, especially where a deployment has a reliability requirement, private data, or a task-specific cost target that survives the next model release.

A recorded agent session follows one path through a tree. Supervised fine-tuning (SFT) on that trace teaches the model the actions taken along the recorded path. Once deployed, the model can take a different action early in the session. Now the next turn starts from a situation the demonstration never covered. Each additional turn creates more opportunities to diverge.

In reinforcement learning (RL), the model attempts the task during training. Its own choices determine which situation it encounters next, and the outcome supplies feedback on the path it took. This is the sense in which Mancuso expects RL to generalize better for agents. Practice can reach branches missing from the recorded session. How much of the tree the model explores depends on the attempts it generates.

A deployed agent gives the team a head start. Its harness already runs the model through a task. Its evals give the team a way to judge the result. Even the criteria used in A/B tests or to decide whether a harness change can ship can help define success in an RL environment. The harness supplies a place to act, and those criteria supply a basis for scoring an attempt.

Production traces answer the harder question of what to practice. Find a session where the agent failed at something a user wanted to do. Use it to generate tasks that exercise that failure, run new attempts in the harness, and evaluate their outcomes. Even a failed trace is valuable here. It identifies a problem worth solving while leaving the new agent free to find a better path.

The model has two jobs in this system. On inference GPUs, copies of it act as samplers. They generate rollouts, each a trajectory through the environment using the agent harness. The trainer consumes those rollouts and updates the model weights with gradient descent. It broadcasts the new weights back to the samplers, so subsequent attempts can reflect what the model just learned.

The simplest schedule keeps the versions aligned. Generate rollouts, train on them, distribute the updated weights, then start the next round. Separate inference and training GPUs spend part of that schedule waiting for each other. Running both jobs continuously improves utilization, but the trainer can advance while a sampler is still working. An attempt generated by version 1 might arrive when the trainer is already on version 3. That gap is staleness.

The question for version 3 is how likely it would have been to choose the tokens version 1 generated. A version gap is one reason those probabilities can differ. Small differences in the systems that run training and inference can also change the distribution, even when both sides intend to implement the same model. The challenge is to correct enough of that mismatch to keep learning from a rollout's outcome.

Modal starts with group relative policy optimization (GRPO), which uses the relative scores of attempts at a task to guide a policy update. The team adds three corrections so the trainer can use that signal when the samplers behave differently.

• Masked importance sampling handles token-level disagreement. For a token in a rollout, compare the probability the trainer assigns it with the probability under the sampler that produced it, given the same preceding tokens. Their ratio measures how differently the two policies treat that choice. Masked importance sampling uses the comparison to adjust the token's contribution to learning. The ratio can reflect both the age of the sampler's weights and differences in how the two systems compute probabilities.
• Score centering handles drift across updates. Token-level correction leaves some residual mismatch. Once an update incorporates that error into the weights, the next step starts from a slightly shifted model. Score centering builds a correction term that tracks and subtracts the accumulating drift.
• Account for the sampler's top-p cutoff. At a setting such as 0.95 or 0.98, the sampler retains the most likely tokens until their cumulative probability reaches the threshold. Tokens in the excluded tail have zero chance of appearing at that position. If the trainer still assigns those tokens some of the probability mass, even its probabilities for the retained tokens describe a different choice. Its calculation has to account for which tokens the sampler could choose. Mancuso reports that with this treatment of top-p sampling, the recipe's score-centering correction becomes exact where it would otherwise be approximate.

How much does the combination matter? Modal compared variants of the recipe in an ablation on coding environments. The run using the corrections together trained more stably than the alternatives the team tested. The result concerns stability over the course of training. Whether the resulting agent solves a particular product's failures still depends on which tasks the team designs and how it scores them.

Modal could already run each part of the job. It had clusters for training, inference endpoints for sampling, and sandboxes for agent environments. Finding enough GPUs in one place was another matter. At the time of the talk, Modal pooled capacity from roughly 15 to 20 providers, including hyperscalers and smaller GPU clouds. That gave the scheduler options during a shortage. It also meant that plenty of GPUs could be free across the fleet while few were available together.

Modal could schedule four or eight nodes fairly quickly. Finding a contiguous block of 16 or 32 took much longer. Agent RL made that constraint particularly expensive. At roughly two inference GPUs per training GPU, about two-thirds of a cluster went to generating rollouts. In an eight-node allocation, the trainer might get just two or three nodes. Moving the samplers elsewhere would free those slots, provided the trainer could keep distant copies of the model up to date.

What has to cross the network after an update? A sampler that holds weights (W_t) needs only the difference (\Delta_t = W_{t+1} - W_t) to reconstruct the next version. Each full set of weights was dense and large, but Modal found the difference between consecutive steps to be highly sparse. Off-the-shelf compression could exploit that sparsity. Mancuso reports a roughly thousandfold reduction in the amount of weight-update data sent from the trainer to the rollout engines.

In the coding run, the trainer stayed in one cluster while inference GPUs across multiple clouds generated rollouts. Compressed weight changes went out to the samplers. Completed trajectories came back as training data. Shrinking the outbound transfer let the team place inference work wherever it could find GPUs while keeping the training nodes together.

Geography was only one source of fragmentation. The available GPUs also came in different generations. A trainer on B200s could use samplers on H100s or A100s, but those devices can run different kernels to implement the model's arithmetic. Those differences can alter token probabilities. Sparse deltas solve the problem of getting weights to those machines. The GRPO corrections address the separate problem of learning from the rollouts those machines produce.

Which devices actually did the rollout work? Modal counted requests by GPU type over the course of training. Each type's count represents inference work that fed the same run. As requests went to different devices, the trainer had to learn from the mix of token distributions their kernels produced.

Did the extra mismatch destabilize the updates? The comparison run used B200s for both training and inference. With mixed rollout hardware, Modal observed up to roughly four or five times as much trainer–sampler mismatch. Yet training reward and gradient norm, a measure of the size of the training gradient, behaved as stably as in the all-B200 run. The correction could tolerate a substantial increase in disagreement on this workload.

Across the full training run, the rollout portion cost 15% less. Some inference work could run on cheaper GPUs for which there was less demand. The team got that saving using off-the-shelf inference configurations for each GPU type. Mancuso saw further savings as a possibility if the team tuned individual deployments, including their numerical precision.

Even within a single GPU type, the trainer had to accommodate two precision choices. One H200 pool ran the model in BF16, the more conservative choice. Another H200 pool ran it in FP8. The lower-precision model could assign different probabilities to the next token even on the same hardware. The same corrections allowed the trainer to learn from those quantized rollouts alongside the others.

Modal built its implementation on an existing reinforcement learning framework and inference engine. By the time of the talk, the team had published a technical write-up and planned to release the source code. Owning the stack, in this example, meant understanding those components well enough to change how the trainer and samplers worked together. Recording at 19:27

Mancuso expects that competence to matter most where teams already compete on task performance per dollar, as coding agents do. For a team with an agent in production, the first decision is close to the user. Which failure is worth turning into a training task? The eval determines which attempts succeed. The training recipe determines whether the team can learn from those attempts as the model changes. In Modal’s run, the recipe let the trainer learn from rollouts generated by H200s running at FP8.