Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

How do you train an agent when you can look back at its actions and recognize good or bad work, but cannot write a complete scoring rule beforehand? Reinforcement learning with verifiable rewards, or RLVR, makes progress where correctness can be checked. Many real tasks offer something weaker: an evolving understanding of what the agent should have done. Will Brown, who introduces himself as leading applied research at Prime Intellect, starts from this gap between retrospective judgment and an explicit objective.

The practical problem is to turn that imperfect feedback into a repeatable learning process. This requires more than choosing an RL algorithm. It requires constructing tasks, worlds, and scoring methods that remain useful as agents encounter situations their designers did not anticipate.

An agent is a model plus a harness: the surrounding machinery that lets it act. An environment supplies a task and a world. That world might include a Docker image, a repository, tools, skills, applications, or browser tabs, together with a scoring rule. The agent acts, the environment returns observations, and the interaction continues until the resulting rollout can be rewarded.

A reward becomes useful for learning through an advantage: how much better or worse a rollout was than a baseline, possibly with additional scaling. The basic relationship is:

Here, R is the reward and b is the baseline. Rollouts and their advantages support a gradient update to the policy—the model weights—that nudges the model toward higher-reward behavior. Brown places GRPO, REINFORCE, and CISPO within this policy-gradient framework. Their details differ, but they share the need for a useful reward signal.

Prime Intellect’s tooling spans this loop. GPU orchestration supplies compute; PRIME-RL handles large-scale training; composable task sets, harnesses, and Verifiers assemble environments. Brown describes Lab as the platform for hosted training, evaluation, inference, experiment monitoring, run management, and deployment. The intended workflow starts from an open model and lets domain experts steer its weights toward their own tasks. Making that accessible remains a goal: the research and operational details are still difficult.

The environment is useful before any weight update happens. A task, harness, and reward definition can serve as an evaluation, generate synthetic data for supervised fine-tuning, support RL or on-policy distillation, or drive prompt optimization with GEPA. It can also be a test bed for comparing harness designs. These uses share the same underlying question: what happens when this agent attempts this task, and how should the result be assessed?

Deterministic checks make some assessments straightforward:

The difficulty changes when the agent writes a research report, books a flight, buys something, or handles a refund. A completed transaction does not fully answer whether it was the right choice or whether the user’s situation was handled well. These tasks still need reliable, scalable feedback, but there may be no single clean best answer to compare against.

Handcrafted benchmarks are expensive to build. Brown describes researchers working with experts, and sometimes data vendors, to refine task sets carefully. That process becomes harder to scale when deployment is open-ended: the production distribution is something the team discovers as users interact with the agent. Training for a known distribution does not, by itself, define successful behavior outside it.

A loose scoring rule introduces another problem. Reward hacking occurs when optimization finds a way to increase the proxy score without satisfying the intended objective. The poorly specified boundaries of a reward become opportunities for the model to exploit. A rising reward curve can therefore coexist with behavior that is getting less useful.

Brown uses continual learning in an operational sense: deploy a model in a realistic setting, let it act, observe its mistakes, and use those observations to avoid repeating them. Prompt and harness changes can contribute, but the larger aim is a system that improves over time while humans supervise at a useful level of abstraction. Achieving that requires automating more of the low-level work of designing and running learning experiments.

Online RL is one part of this process. It can refine skills as experience accumulates, but Brown distinguishes that from incorporating dense new knowledge into weights. A production learning system may need both. Its experiments should also be monitorable, traceable, and replayable, so changes in behavior can be investigated rather than merely observed. The goal is to make this kind of optimization available on strong open models for tasks beyond the distributions covered by their original training.

Three complementary techniques help create supervision where clean labels are missing:

• Grounding: Give a model source material, then compare what it can do with and without that material in context. The capability gap provides something to learn from.
• Judges: Use an LLM’s existing reasoning ability to assess whether an action was good or bad. Carefully configured judges spend inference compute on evaluation.
• Search: Spend additional compute improving the tasks, the worlds in which agents act, and the criteria used to judge their rollouts.

These techniques move work into constructing better supervision rather than assuming the right reward already exists.

Production traces are especially useful raw material. User prompts and calls from an orchestrator to a subagent reveal what the deployed system is actually being asked to do. As traces accumulate, they begin to define the task distribution. They do not yet provide quality labels, but they tell you where to look for tasks and failures.

Document collections and repositories provide similar anchors for search and coding. Environment construction therefore does not begin from nothing: it begins from real artifacts whose structure and content constrain what useful learning should look like. Even without supervision, the production world supplies evidence about what matters.

Documents make backward task construction easy to see:

1. Sample a document and generate a question-answer pair grounded in it.
2. Ask other models, still given the document, to check that the question is answerable.
3. Remove the initial search result from the learner’s starting point, so it must find the supporting material itself.

The construction process has access to evidence that the learner must recover. Verify the easier problem, then train on the harder upstream problem. This creates supervision without requiring someone to author every search task from scratch.

For code, a completed pull request provides a description, a diff, tests, and evidence of a reachable end state. Withhold tests, or remove implementation files while retaining tests, and the completed artifact becomes a reconstruction task. The following small Python example illustrates the latter transformation: the task input retains the test, while the reference implementation is kept separately.

completed_files = {
    "pricing.py": (
        "def total(price, quantity):\n"
        "    return price * quantity\n"
    ),
    "test_pricing.py": (
        "from pricing import total\n\n"
        "def test_total():\n"
        "    assert total(7, 3) == 21\n"
    ),
}

reference_files = dict(completed_files)
task_files = {
    path: contents
    for path, contents in completed_files.items()
    if path != "pricing.py"
}
task_prompt = "Restore pricing.py so the retained test passes."

This produces a starting state, not a solved rollout. In a real repository, the PR’s description, diff, and tests provide much richer constraints on what should be reconstructed.

The general property is known reachability. Start with an end state that exists, take steps backward, remove the solution, and ask the learner to find a path forward again. That principle extends beyond code wherever the environment lets you control the starting state.

Production tools and applications often prevent that kind of backward construction. An agent may interact with an MCP tool, a CLI, or a website without the training team being able to program its backend state. World simulators offer a way to regain control over those interactions.

Brown describes combining general backend infrastructure with test-time search, then iterating between simulated and real behavior using production traces as grounding. He reports that this can produce high-fidelity simulators, with additional production data helping refine them over time. Their crucial training property is backend controllability: the designer can plant an answer, establish a reachable end state, and work backward to generate a task. Verifiability can then be built into the simulator even when the real deployment gives no advance assurance that a user’s request is solvable.

A completed trajectory contains evidence that was unavailable when an action was chosen. Later events may reveal exactly where the agent went wrong. Brown illustrates this with multiple models agreeing that a particular action was mistaken; agreement is a way to aggregate judgment, not a guarantee of correctness. Expensive retrospective analysis can then be distilled into rubric questions that are cheaper to apply repeatedly. Those questions also identify failure modes that new tasks should target.

Search can improve several parts of this process:

• Trace mining: Revisit offline production traces with more inference compute and more model analyses.
• Difficulty calibration: Generate and select tasks on which rollouts show useful outcome variation. Uniform success or uniform failure provides little separation for a group-relative advantage signal.
• Simulator refinement: Have agents explore complex tool or application behavior and improve the simulator implementation.
• Verification: Strengthen judgments both during training and while building rubrics offline. Use adversarial prompt optimization to probe backdoors, then inspect rewarded traces for proxy exploitation rather than actual task fulfillment.

The useful allocation of compute is not limited to making the acting model think longer. It also includes making the learning environment more informative and harder to exploit.

Reward hacks can emerge unexpectedly, but Brown observes that judges are often able to recognize the kinds of hacks humans find obvious in hindsight. Recognition after a rollout does not guarantee prevention during it. The practical response is to accumulate examples, use them to understand recurring failures, and spend inference compute refining both the environment implementation and its rewards.

Some weaknesses become visible only after optimization begins. Small training runs are part of environment design, not merely a downstream use of a finished environment. Train an individual model in one environment, inspect how its behavior changes, and record more than aggregate reward. Tool-call patterns and judge-derived questions about traces can reveal what the policy is learning to do. Those observations guide the next environment revision.

The most consequential questions should reach the human expert: is this behavior good, is this the intended goal, and is this an acceptable way to pursue it? Automation can mine traces and refine checks, but the desired boundary is human direction over objectives and acceptable behavior. That expert judgment is part of how confidence in an environment is earned.

General Agent brings several of these pieces together for tool use: generate tasks, attempt to solve them, gate them by pass rate, and use the selected tasks for training. Brown presents this as an online task-evolution loop and points to benchmark uplift. The published implementation makes a narrower distinction: it synthesizes a fixed corpus offline, leaving evolution during training to future work. Its reported benchmark transfer gains come from supervised fine-tuning; the separate RL experiment reports training reward. The task-generation mechanism is concrete, while a continuously evolving training distribution remains the broader direction.

Task selection does not solve every learning problem. Some information in the environment may never be acquired through RL exploration alone. ECHO: Terminal Agents Learn World Models for Free, which Brown attributes to external researchers he describes as collaborators, combines learning from agent actions with supervised signal from the environment. Prime Intellect conducted a separate investigation of this approach.

The mechanism is to model the likelihood of tokens generated by the environment itself. Instead of learning only which actions receive reward, the model also learns what responses to expect from the world. That predictive signal can support an internal world model, help the agent navigate, and bring new information into its weights. Prime Intellect’s related experiments found environment-dependent outcomes, so this is a complementary learning signal rather than a universal improvement guarantee. The distinction is between refining a policy’s skills and also teaching it about the world those skills operate in.

Detailed research work still remains in these workflows. Environments provide an anchor for progressively automating it: use compute to mine real-world data, refine tasks and rewards, and move human involvement toward higher-level decisions about goals and guardrails. The intended closed loop is concrete. Agents encounter issues in production; those issues become new training tasks; training produces behavior that can be evaluated and deployed again. The environment is the mechanism that makes experience reusable, while human direction defines what improvement should mean.