Learning on the job: the future of post-training

An enterprise already has a way to call an agent and ask it to complete a task. How can it train a custom model for that task without replacing the surrounding workflow—or even supplying the harness’s source code? As agents handle longer sequences of reasoning, tool calls, and changing environment state, adapting the model to an existing workflow becomes a central post-training problem.

Raymond Feng organizes this progression around how people learn. Simple question answering provides the early exercises; synthetic environments introduce more complicated tasks. Training inside a customer’s harness resembles an internship: there is a job to do, but the training system does not control exactly how it unfolds. The eventual ambition is an agent deployed once that can adapt to unfamiliar tasks and learn from its interactions. Each stage adds realism while giving the training system less control.

Start with a math question and its numerical answer. An orchestrator holds this task specification and drives the rollout—the model’s attempt at solving the problem. The training cycle has four steps:

1. Send the prompt to the model completion endpoint and collect its answer.
2. Send the answer to a grader.
3. Pass the graded chat, or a batch of graded chats, to the training engine to compute a weight update.
4. Synchronize the updated weights to the inference engines, then begin another round with new problems.

Synchronization closes the loop: subsequent rollouts must use the updated model for training to build on the previous round.

Graded chats are the training engine’s essential input in this setup. The orchestrator’s job is to produce them; the training engine turns them into model updates. That boundary is easy to maintain when the entire process lives inside the training stack. The team controls the rollout code, the grader interface, and the exact format of every chat. Nothing has to be reconstructed from an unfamiliar external system.

Single-turn Q&A provides little room to practice skills that require a sequence of actions. Synthetic environments extend the same loop by moving environment state outside the training stack. A task specification can now include tool definitions and an initial filesystem state, rather than just a prompt and expected answer.

The orchestrator asks the model what to do next. If the response requests a tool, the orchestrator calls a sandbox to read or change the environment, returns the tool result to the model, and continues the conversation. Once the task ends, the full trace goes to the grader. The training engine still receives graded chats and computes a weight update; the rollout has simply become a multistep interaction.

The crucial property is replayability. For a given prompt, the orchestrator and sandbox can return to the initial state and run another attempt. Those attempts can run serially or in parallel. This supports Group Relative Policy Optimization (GRPO): compare multiple rollouts for the same prompt, then increase the model’s tendency to produce the more successful trajectories and decrease its tendency to produce the less successful ones. Parallel execution is convenient, but the relevant property here is the ability to generate comparable attempts from the same starting conditions.

The environment must resemble deployment closely enough for learned behavior to transfer. Feng treats environment fidelity and reward hacking as two views of the same underlying problem: an agent learns the environment it encounters, including accidental quirks. Those quirks can shape its policy just as surely as the intended reward function.

Feng reports that networking issues caused tool calls to fail around 10% of the time in one training run. During that run, model responses became progressively shorter even though the reward function contained no length penalty.

His analogy is a sidewalk full of potholes. The longer someone walks, the more opportunities they have to fall into one. Likewise, a longer rollout exposes the agent to more tool failures, any of which can leave it with zero reward. Shortening the interaction reduces that exposure. The effective incentive therefore comes from the combination of the reward function and the environment’s failure behavior, not just from an explicit penalty in the grader.

Environment incentives can also push behavior in the opposite direction, toward increasingly unhelpful or gibberish output. In a separate training run, Feng describes sandboxes with time limits to prevent indefinite execution. Timed-out rollouts were normally filtered out of training. With slow tool calls, a model facing a difficult problem could issue many calls in quick succession and exhaust the sandbox’s time budget. The rollout would then disappear from training instead of receiving zero reward.

The distinction is small in data-processing code but consequential for learning. This Python example makes the filtering rule explicit:

rollouts = [
    {"id": "completed-failure", "timed_out": False, "reward": 0.0},
    {"id": "tool-call-timeout", "timed_out": True, "reward": None},
]

training_batch = [
    rollout for rollout in rollouts
    if not rollout["timed_out"]
]

Only completed-failure remains in training_batch. The timed-out attempt is absent; it does not enter training as a failed attempt.

Discarding an outcome changes the learning signal. In Feng’s example, triggering a timeout provided a way to avoid the negative signal of an unsuccessful attempt.

As tasks become more complicated, faithfully simulating their environments becomes harder. An unintended networking failure or a seemingly practical timeout filter can induce behavior that was never part of the task designer’s objective. The difficulty is not limited to writing a better grader: it extends to every environmental detail that influences which trajectories succeed, fail, or reach training at all.

If an agent learns the precise environment distribution it encounters, a direct response is to train it in the real environment. Bring your own harness removes the need to reproduce the customer’s workflow inside a separate training framework. The agent practices through the same machinery that will use it in production.

Almost everything now moves outside the training stack. The retained interface is the model completion endpoint, together with a way to record its incoming requests and outgoing responses. The enterprise keeps its existing orchestration loops and logic, running them in their original form. This lets training fit the customer’s method of using the model without taking ownership of the harness.

The tradeoff appears in the data. With less control over how rollouts unfold, the training system receives interactions in less familiar formats and has less directly usable learning signal. Feng connects this transition to NVIDIA’s Polar: Agentic RL on Any Harness at Scale: moving from micromanaging each rollout to listening to model calls from a black-box harness. Observing those calls provides an integration point, but does not restore control over the logic that generated them.

Two difficulties become prominent: non-replayability and offline or off-policy data. Feng connects them to the loss of control over rollout invariants and data structures. The training system must accept more of the data as it arrives, rather than requiring every interaction to fit the conditions its update method prefers. For GRPO, generating several comparable attempts at the same task may no longer be possible.

Consider a recorded customer-support conversation. It shows what the agent said and how the customer responded. It cannot show how that same customer would have reacted if the agent had given a different reply at that moment. Replaying the text does not recover the missing human response. The sandbox operation of resetting state and trying another trajectory has no equivalent here.

Yet a human support agent can learn from the conversation. A customer’s reaction can reveal that an answer helped, confused, or frustrated them, and the agent can internalize a change for the next interaction. That ability motivates Feng’s optimism about learning from unreplayable experience. It establishes a useful target for model training, rather than a finished method for obtaining those updates.

Feng identifies three research directions for extracting useful updates from these less controlled interactions:

• Self-distillation. Results so far have induced specific new behaviors, but the technique remains narrowly scoped. How far it can generalize is still an open research question.
• Automated data pipelines. A pipeline could inspect a batch of traces, flag undesirable behavior or recurring failures, assemble a training dataset, and send it into model improvement. The corresponding work currently involves substantial human review: reading traces, describing needed improvements, and curating examples.
• Qualitative feedback ingestion. Production interactions may return a customer’s written feedback instead of a binary outcome or numerical grade. Learning directly from that information would make more of the available experience useful. Self-distillation is one avenue Feng is exploring for this purpose, but a general solution remains unsettled.

The longer-term vision extends beyond improving one model on one task. A single deployment could serve many kinds of tasks, potentially across different users. It would reflect on its performance for each kind of interaction, evaluate how it was doing, and automatically turn those interactions into weight updates. Here, learning would become an ongoing property of deployment rather than a separate project organized around a fixed task.

That proposal expands the environment to every interaction the agent ever has, paired with a still-hypothetical ability to evaluate itself. Today, addressing one task or failure mode at a time can become a game of Whac-A-Mole: each new problem requires another dataset or environment before training can address it. A system that learned broadly from its own interactions could reduce that reactive cycle. This is the intended destination, not a demonstrated deploy-once capability.

Feng closes with a prediction from David Silver and Richard S. Sutton’s Welcome to the Era of Experience: “experience will become the dominant medium of improvement.” Their forecast goes further: experience could eventually dwarf the scale of human data used in today’s systems. Reaching that future depends on making the messy interactions of deployment usable for learning—including the ones that lack a clean grade or a second attempt.