Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

How do you take an open model and make it better at your company’s production applications, internal tasks, and workflows? The goal is more than a customized checkpoint: you need a way to train, deploy, and keep improving the model as it encounters real work. Will Brown, who introduces himself as Prime Intellect’s applied research lead, approaches that problem through two open-source libraries: verifiers for environments and PrimeRL for training and inference.

Prime Intellect calls the surrounding infrastructure its open superintelligence stack. At the bottom is compute; Brown reports a global fleet of more than 10,000 GPUs, with hundreds or thousands within individual clusters. Above that sit PrimeRL, environments built with verifiers, and Lab, which brings together the Environments Hub, hosted training, evaluations, inference, and sandboxes. These layers support the company’s Intellect model series and training projects with customers.

The practical question is how to assemble these pieces without first building a large research organization. Brown uses an evolving alpha Lab Cookbook to frame the workshop: broaden environments beyond reinforcement learning, overhaul their interfaces, and support asynchronous training and algorithms such as on-policy distillation and self-distillation. Large-model experimentation has to be fast, affordable, and reliable enough to repeat; the walkthrough therefore covers both research flexibility and the systems that make iteration economical.

An environment specifies what you want a model to do: the data, the scenario, the interaction rules, and the scoring that distinguishes good behavior from bad. Its first use can simply be evaluation. You can improve prompts or the agent harness before changing model weights, and you can keep frontier API models in the system wherever they work best. Evaluations reveal where a trained model belongs and where an existing API is the better fit.

The logic needed to evaluate a model is also much of the logic needed to train it. A task suite can help choose between GPT and Claude, or among Opus, Sonnet, and the model Brown refers to as Mythos, according to quality and cost. That same suite can then drive post-training. The familiar recipe is supervised fine-tuning, or SFT, followed by reinforcement learning, or RL; on-policy distillation adds another route. For several distinct capabilities, Brown describes training separate RL experts from the same base model, then distilling those teachers into one checkpoint. He finds this generally more reliable than trying to acquire all those capabilities together.

The resulting deployment can be a full-weight checkpoint or a LoRA adapter. Deployment then supplies more evidence about what needs improvement, making post-training a recurring process whose compute cost can be amortized against inference spending. The difficult connection is between production feedback and the next environment: extracting useful learning signals from real-world interactions remains an engineering and research problem. Owning a trainer does not by itself close that loop.

Earlier verifiers environments owned a multi-turn loop into which users plugged tools. That worked well when the main challenge was moving from single-turn answers to tool use. Supporting CLI agents and MCP repeatedly required new extensions, exposing a more useful division: the task describes the work, while the harness controls how the agent attempts it. Some tools, skills, and system prompts belong to a particular task; others are part of the harness. Both remain components of the complete environment.

The common procedure is straightforward:

1. Put a model into a harness and give it a task.
2. Run the interaction to collect a rollout.
3. Verify what happened.
4. Use the result for evaluation or training.

This also changes how to think about SFT data. Rather than collecting trajectories, saving them offline, and repeatedly converting formats for upload and training, treat the teacher as another rollout source. The teacher can be a model or a replay of an existing dataset. On-policy distillation uses student rollouts, then scores them with the teacher’s likelihoods or log probabilities rather than relying only on a task reward.

Environments are Python packages, so they can bring their own dependencies and fetch data from whichever sources the task requires. That packaging keeps the same task-and-rollout abstraction available for evaluating API models, generating supervised data, and post-training models you control.

The verifiers V1 architecture separates three independently loadable components. Here, V1 names the architectural overhaul, not a stable package version 1.0: the demonstrated interface was a preview under verifiers.v1, alongside the still-working legacy interface, with a stable release pending.

A task set often represents a backend, server, or stateful world. It does not have to own the agent loop. Brown names Hugging Face datasets, Harbor, NeMo Gym, and OpenEnv as integrations; the accompanying V1 announcement distinguishes full Harbor support from alpha NeMo Gym and OpenEnv integrations.

The default harness retains the familiar system-prompt-and-tools loop. Other harnesses can run recursive language models, CLI agents such as Codex, Claude Code, or OpenCode, research agents such as mini-SWE-agent, or custom implementations using LangChain or DSPy. Basic compatibility checks compare a task set’s requirements with a harness’s supported capabilities. This lets a task remain fixed while you change how the model works on it.

The runtime supplies somewhere to execute the harness as code. Local execution, Docker, Prime sandboxes, another sandbox provider, and custom backends are all possible. The implementation retains asyncio patterns while making greater use of subprocesses; uv scripts help contain executable dependencies. Combine the three components, run a task, and the output is a trace. Brown says the new implementation was available on the repository’s main branch and as a development release.

Task authoring can start with a dataset row. In the new namespace, decorators, functions, and loaders replace the old Rubric pattern. Pydantic supplies typed validation, while TOML configuration and CLI overrides let users adjust runs without burying settings in code. The intent is to catch configuration errors before execution. The same composition supports SWE-grep for agentic code search, Wordle, document search with judges, and Harbor’s terminal-based benchmarks: choose the task set, then pair it with an appropriate harness and runtime.

Rewards and metrics are functions that consume rollout records and return numbers. Rewards drive the learning objective; metrics expose behavior such as tool-call counts and errors. Some useful rewards need access to several rollouts at once. Pairwise judging, ranking, and a bonus for the shortest correct answer all require a group-level view, which becomes awkward if the infrastructure assumes every rollout is entirely independent.

Conciseness illustrates why that access matters. Models can keep extending their reasoning if the reward gives them no pressure to be efficient. A fixed token cap is difficult to choose: the right length differs across problems and changes as the model improves. Multiple samples provide a relative standard. Compare their lengths, identify successful solutions, and reward the more concise ones while retaining a separate correctness reward.

This Python example implements a shortest-correct bonus. For the teaching task 17 + 25, the two correct rollouts use 80 and 30 tokens; an incorrect rollout uses 10. A configurable bonus goes to the shorter correct rollout, so being brief cannot compensate for being wrong.

from dataclasses import dataclass


@dataclass(frozen=True)
class Rollout:
    answer: str
    token_count: int
    correct: bool


def group_rewards(
    rollouts: list[Rollout], bonus: float = 0.2
) -> list[float]:
    correct_lengths = [r.token_count for r in rollouts if r.correct]
    shortest = min(correct_lengths, default=None)
    return [
        float(r.correct)
        + (bonus if r.correct and r.token_count == shortest else 0.0)
        for r in rollouts
    ]


rollouts = [
    Rollout(answer="42", token_count=80, correct=True),
    Rollout(answer="42", token_count=30, correct=True),
    Rollout(answer="41", token_count=10, correct=False),
]
rewards = group_rewards(rollouts)

The relative comparison avoids prescribing a universal solution length. The bonus magnitude still expresses a design choice about how strongly to favor efficiency alongside correctness.

A task can supply capabilities beyond those already built into the agent harness. These may appear as skills or MCP servers. Verifiers uses MCP as a backend for both tools and user simulators: the simulator runs as a server, but its output reaches the model as a user message rather than a tool result. A script or an LLM supplied with context can therefore play the user during a multi-turn rollout. The harness connects to these modular components as uv scripts, and grading can also run through a script instead of a basic reward function.

To train an existing harness, verifiers introduces an interception server rather than requiring the harness to implement an RL-specific loop:

1. Give each rollout a substitute API base URL with an OpenAI-compatible or Anthropic-compatible interface.
2. Intercept the harness’s model requests.
3. Apply training requirements such as temperature settings and log-probability collection.
4. Forward requests to the training run’s inference server and return its responses to the harness.

The harness behaves as though it were making ordinary model API calls. Its code can remain the same when moving between training and deployment.

The evaluation CLI exposes the same separation during experimentation. A task set can run with an RLM harness or Codex, locally, in Docker, or in sandboxes. This makes it possible to prototype on a MacBook, then move the workload to cloud execution without redesigning the environment. It also makes harness quality an experimental variable: which harness works best for a model, and does that advantage carry across tasks?

Supporting subagents and parallel execution requires a trace graph rather than only a linear conversation log. The graph must represent message-level branches while preserving the sequential dependencies needed for RL. Harnesses naturally work with messages and text, but trainers need the exact tokens sampled by the model. Decoding tokens into text and then tokenizing that text again can produce a different sequence. Brown describes the resulting numerical problems as especially subtle late in large training runs. The trace therefore stores enough history to map logical messages back to their token sequences.

The standalone Renderers library addresses the boundary between those representations. A chat template might strip an extra newline from a model response. The next inference request can then contain different tokens from the history expected by the trainer, creating a trainer–inference mismatch. It can also make a conversation appear to branch in token space even though it remains a single continuation in message space.

Brown identifies OpenAI’s Harmony format with gpt-oss and Thinking Machines’ Tinker cookbooks as precedents for renderer-style abstractions. Prime’s implementation is a Python library independent of the rest of its stack. It turns chat templates into programmable artifacts that manage token input, token output, and concatenation. By consulting trace history, a renderer can recognize that messages share a logical prefix even when freshly retokenizing those messages would fail to reproduce the original token prefix.

Preserve both the logical conversation and its exact token history. Brown connects this need to increasingly stateful agent APIs and offers it as an explanation for the stateful design of OpenAI Responses. That is his interpretation of the API’s design motivation; the concrete systems requirement is the dual representation. Harness authors should be able to work in text while the trainer and inference engine retain token-level fidelity.

The interception server can route requests to either an evaluation client or a training client. Training needs capabilities such as log-probability access and temperature control. Brown’s example is a reasoning API that hides its reasoning trace: it cannot supply all the token log probabilities required by this training path, but the model remains usable for evaluation. The distinction belongs in the client layer rather than in the harness.

OpenAI Chat Completions, Responses, and Anthropic interfaces become API dialects translated behind interception. On the training side, a request can pass through a renderer to produce the appropriate token representation. PrimeRL then consumes the assembled environment, including its task set, harness, runtime, interception server, and renderer.

PrimeRL is asynchronous by design. In synchronous training, a batch can wait for its slowest rollout before the model advances. For agents with long and unpredictable trajectories, Brown prefers accepting some off-policy data—rollouts generated by earlier model weights—so that rollout execution and training can overlap. Strictly matching every rollout to the current policy comes at the cost of waiting for stragglers.

The orchestrator connects separate inference and trainer processes running on separate GPUs. It maps the environment’s endpoint to the inference server, launches rollouts, packages results into batches, and sends those batches to the trainer. The trainer constructs the sequences required by the configured loss. A client-server environment engine provides the communication boundary.

That boundary also separates scaling decisions. You can change the number of environment workers, the number of inference replicas, or the use of sandboxes without making the trainer understand those details. Keeping the components independently runnable matters before training begins, too: an environment should still work for evaluation and experimentation when no RL trainer is attached.

Brown describes scaling work on GLM-5 and the Kimi K2.5/K2.6 family. The reported run used GLM-5 before GLM-5.2 was released; he also reports support for the newer model. Brown reports a GLM-5 training step in under five minutes on 28 nodes for long-horizon coding tasks with 131K context. He gives an associated estimate of 1,000 steps in three days for approximately $50,000 at rental prices. The duration and cost are his run estimate, not quantities established by the under-five-minute bound alone.

That expenditure is substantial, but it becomes a plausible enterprise experiment when compared with sustained inference spending. The necessary starting point is a set of tasks representative of the company’s coding workflows, plus evaluations that can show whether training improved them. Brown’s comparison with monthly token bills motivates investigating the economics; it does not establish a measured savings result.

Coding agents make the latency problem concrete: a task might take half a minute, a few minutes, or hours. Asynchronous RL lets a completed rollout enter the first batch that can accept it even if training has advanced since the rollout began. Meanwhile, the inference server keeps receiving the latest model weights. Brown says his experiments often operate around an average off-policy value of 16, but does not define its unit or exact staleness statistic.

This overlap absorbs more than slow generation. Sandbox startup, weight synchronization, environment execution, and judge grading all consume time without necessarily consuming the trainer’s GPUs. Letting them proceed concurrently reduces the need to make every stage finish at the same moment. Brown cites DPPO among the loss-side techniques used to stabilize this training. Brown reports stability through thousands of steps, with reliable 10,000-step runs still an experimental target.

Large batches and mixture-of-experts models require parallelism on both sides of the system:

• Trainer: Expert parallelism distributes expert computation, while context parallelism handles long sequences.
• Inference: Wide expert parallelism spreads experts across nodes, and disaggregated prefill separates a major phase of inference work.
• Supporting systems: FP8, wide data parallelism, routing, KV offloading, and memory management contribute additional efficiency.

The separation between trainer and inference lets serving optimizations carry into the RL stack.

Router replay creates a less obvious storage problem. Reproducing routing requires metadata for every layer and potentially multiple selected experts at each layer. That can substantially exceed the storage needed for tokens and log probabilities alone. Multimodal rollouts add artifacts such as images. Brown describes moving these heavier objects to object storage or a filesystem instead of keeping everything in memory, while refining model-specific recipes for context parallelism and other interacting constraints.

PrimeRL uses TorchTitan as its training foundation. Brown favors its hackability and modularity, contrasting it with the more monolithic Megatron: when a new model or research idea arrives, the team wants to change the relevant pieces quickly. Brown says the research team maintaining PrimeRL had fewer than 10 people and the company had fewer than 40 at the time of the talk. Modular infrastructure lets that team divide the work while continuing to adapt the stack.

The same desire to experiment motivates PrimeRL’s algorithm abstraction. Brown names on-policy distillation, OPSD, ECHO, SFT, and MaxRL as methods the team wanted to support without repeatedly threading new conditionals through the whole trainer. The useful division is between an algorithm, which prepares training data, and a loss, which supplies the gradient objective.

An algorithm class can handle scoring, rollout groups, and advantage assignment, then target a loss with the appropriate signals and masks. Configuration selects an entry from an extensible registry, including on a per-environment basis. This separates the question of how a trajectory gets its learning signal from how that signal enters the gradient computation.

Two questions organize the methods: Who generated the rollout, and what learning signal is assigned to it?

For SFT, the unit-advantage view recovers ordinary cross-entropy or negative log-likelihood training. For RL, the baseline is often the group mean. Factoring these choices out allows the surrounding infrastructure to remain shared even when the learning objective changes.

For on-policy distillation, the student has already generated the sequences. The additional operation is to send those sequences to a teacher and obtain reference log probabilities. Brown describes using a one-token generation request to trigger prefill scoring; this requires an inference backend that exposes the prompt-token scores, not merely the probability of the newly generated token. Those reference scores can then be attached to the existing student sequences.

Self-distillation changes the teacher’s information rather than necessarily changing the model. Add a hint to the teacher prompt, use a renderer to pack the altered input, then slice the returned scores back into alignment with the original sequence. The teacher receives extra context, while the student learns from scores associated with its original rollout.

ECHO combines objectives over different parts of the interaction: cross-entropy predicts environment-observation tokens, while an RL objective trains the agent’s action tokens. This learns about the observations resulting from actions within the training objective; it does not require describing a separately deployed simulator. More generally, sampling source and algorithm can vary independently by environment. Brown’s mixed configuration includes both ordinary on-policy distillation and GRPO, sharing the same training infrastructure.

Hosted training packages these capabilities as managed PrimeRL. At the time of the talk, the broadly available self-serve offering was multi-tenant LoRA, focused mainly on RL. Brown describes forthcoming full fine-tuning with more freedom to change the model and PrimeRL itself, while the platform handles GPU allocation, autoscaling, automatic restarts, dashboards, and unified billing for services such as sandboxes and judges. Environment development stays local: build a Python package on a CPU laptop, push it to the platform, and select it in the run configuration.

The customization boundary depends on the experiment:

• Reward function: Change what counts as success inside the environment.
• Trainer configuration: Select a loss or adjust settings such as learning rate.
• Algorithm class: Change scoring, grouping, or advantage assignment.
• Loss or trainer implementation: Introduce a new objective or modify deeper training behavior.

Many applications need only the first layer. The deeper layers remain available when the research question requires them.

Multi-tenant LoRA makes shared serving possible because tenants use one model architecture and base-weight copy, while each has a separate adapter. Brown compares this with shared inference and a managed KV pool: hot-swapping adapters lets a single inference pool serve customized models without replacing the base weights. That arrangement supports token-based pricing without asking every user to reserve GPUs.

Full fine-tuning is the route Brown describes for large-scale SFT, mid-training, and experiments that need more freedom than an adapter provides. He says multi-tenant LoRA was already live and forecasts full fine-tuning for the following couple of weeks. The V1 preview and cookbook were available, with a stable V1 release still expected. These are the release statuses presented in the recording.

Brown closes by inviting people to join a growing team based mostly in San Francisco, with further team-scaling news promised later that week. His description of the business connects the workshop’s open tooling to its commercial use: the company publishes research code on GitHub while earning revenue by training large models. The invitation to prospective hires is to work on both sides of that arrangement—open research infrastructure and the demanding training workloads that keep improving it.