What RL Means for Agents

How do we turn models that answer difficult questions into agents that reliably carry out difficult tasks? That question requires looking beyond the next model release to how systems learn from experience. Will Brown approaches it through his background in multi-agent reinforcement learning at Columbia and language-model research at Morgan Stanley. His February 2025 talk is a research outlook, drawing on emerging results and personal open-source experiments rather than a production deployment or established best practices.

The starting point is a progression from chatbots to reasoners to agents. In OpenAI’s five-level framing, chatbots converse; reasoners tackle harder questions; agents take actions. Models such as o1, o3, DeepSeek-R1, Grok 3, and Gemini make longer deliberation useful for question answering and interactive problem-solving. Turning that capability into autonomous action usually requires chaining model calls together, with prompts, tool calling, evaluations, operational infrastructure, and human oversight.

Those systems can accomplish useful work without having much autonomy. Brown calls systems with substantially engineered decision trees and control flow pipelines: developers decide how calls connect, what happens after each result, and how prompts should change. Many successful products also keep users close to the loop. Cursor, Windsurf, Replit, and research-oriented search interfaces do some work and return quickly for more direction.

Devin, Operator, and OpenAI’s deep research stand out in Brown’s account as systems that work for longer before returning to the user. They raise a practical question: how can developers build more systems with that degree of autonomy? Waiting for a stronger underlying model is one answer, but it leaves the improvement process outside the application developer’s control.

In reinforcement learning, an agent interacts with an environment to pursue a goal, then learns from repeated attempts. That last part changes the engineering loop: an interaction produces both an outcome and evidence for improving future behavior. Brown’s hypothetical is an agent at 70% success after substantial prompt tuning, with a target of 90%. These figures describe an engineering problem, not a measured result: once prompt changes stop helping, what makes the next attempt more likely to succeed?

Brown places that question against several model-development trends. His assessment is that pretraining still reduces loss but offers diminishing returns on capital, while other techniques solve different parts of the capability problem.

Verification and rejection sampling matter because they distinguish useful generated attempts from unsuccessful ones. Brown connects this shift toward outcome feedback to test-time scaling in reasoning models: training can teach a model to use more computation effectively at inference, without requiring humans to curate every successful reasoning trace.

The DeepSeek-R1 paper made this approach unusually concrete. Rather than merely showing that a model could reason for longer, it described mechanisms for training that behavior. Brown reduces the core feedback loop to three actions: ask a question, check the answer, and update the model toward attempts that worked.

Longer reasoning can emerge because it is a successful strategy. The learner need not receive a prescribed long reasoning trace for every question. This point applies most directly to R1-Zero, which omitted preliminary supervised fine-tuning; the released R1 also used cold-start reasoning examples and subsequent supervised and RL stages. The distinction preserves the useful mechanism without treating outcome-only training as the complete R1 recipe. Brown’s broader point is that reinforcement learning discovers strategies for solving problems, and longer deliberation can be one of them.

That opens several routes for the open-model community. Researchers can try to reproduce the training process, while others distill reasoning outputs from stronger models into smaller ones. Replication and distillation are different activities, but both make experimentation with reasoning behavior more accessible.

The basic RL rhythm is exploration and exploitation: try alternatives, observe which work, and favor successful behavior. Brown illustrates it with a model writing code to pass tests. A reward can recognize several properties of the candidate: whether it follows the required format, uses the correct programming language, and ultimately passes the test cases. The model generates these attempts—its rollouts—and their numerical scores feed back into training.

Group Relative Policy Optimization, or GRPO, gives a compact way to understand that loop:

1. Give the model a prompt.
2. Sample N candidate completions for that prompt.
3. Score every completion using the reward functions.
4. Update the model to favor the higher-scoring completions.

This is Brown’s conceptual explanation, not the complete optimization objective. Its value is that it makes the training signal tangible: multiple attempts at the same task give the learner a basis for preferring some behavior over other behavior.

So far, the unit of interaction is still a single completion. An autonomous agent must instead act, receive information from its environment, and decide what to do next. Extending the feedback loop across those multiple steps is the next challenge.

Deep research provides a concrete connection between RL and extended action. Brown points to OpenAI’s disclosure that it trained the system with end-to-end reinforcement learning for browsing and reasoning. Brown describes deep research as potentially making up to 100 browsing or querying tool calls to synthesize an answer; that is his account, not a verified tool-call limit. The important change is the scope of the behavior being learned: a sequence of information-gathering actions contributes to a final answer.

An impressive research result does not establish general competence. Brown distinguishes browsing and synthesis from working in a repository or solving difficult software-engineering tasks. He also cites anecdotal difficulty with out-of-distribution requests, such as filling a table through many manual calculations. These observations are capability qualifications, not a controlled comparison. RL offers a way to teach particular skills through environments, tools, and verification; it does not establish that a trained agent will generalize to every other kind of work.

The infrastructure Brown describes is still largely organized around RLHF: single-turn interactions scored by reward models derived from human feedback. Developers might eventually connect their own environments to large-lab training services, but at the time of the talk he describes reinforcement fine-tuning access as limited and not yet offering the multi-step tool-calling workflow he wants. That is a historical assessment of the available workflow, not a statement about current trainer or API capabilities.

Several questions determine whether this becomes practical:

• Cost: What does repeated interaction and training cost?
• Model size: How small can a model be while still learning the required behavior?
• Generalization: Does training on one task improve performance on others?
• Rewards and environments: How should developers define success and expose opportunities to learn?

These questions create room for open-source infrastructure, training services, and supporting products. The ecosystem needs ways to turn an application’s interaction loop into something a trainer can use.

Not every useful feedback loop needs to update model weights. Brown places DSPy nearby: downstream scores can guide improvements to prompts and the surrounding system. This is adjacent to RL in its use of feedback to improve behavior, while remaining distinct from reinforcement learning that trains the underlying model.

The weekend after R1’s release, Brown tried the idea in a single Python file. He used Hugging Face’s GRPO trainer with a Llama 1B model, math questions, a simple system prompt, and manually designed rewards. The goal was to let the model reason and then produce an answer. The linked gist has since changed its default to Qwen2.5-1.5B-Instruct, while retaining Llama-3.2-1B-Instruct as a commented option; the experiment described here used Llama.

He shared an output on X that appeared to show self-correction. Brown reports improving accuracy and response length that first fell as the model learned the format, then rose as longer reasoning became useful. He supplies no numerical accuracy result or held-out evaluation protocol in the talk, and the example’s apparent self-correction does not by itself establish a particular reasoning process. Brown explicitly declines to call the experiment a true R1 replication.

Over the following weeks, people forked the script, modified it, adapted it into notebooks, and wrote explanations. Brown attributes its appeal to its small size: a single file made the experiment easy to understand and change. That accessibility exposed a particularly approachable part of RL engineering—the rules that decide which outputs deserve credit.

Brown calls this rubric engineering. The simplest reward asks whether the model’s answer equals the expected answer. But a rubric can also recognize intermediate properties: a correctly placed XML tag, the required response structure, or an integer answer even when that integer is wrong. His example of giving a point for a tag illustrates partial credit; it is not a specification of the script’s reward constants.

A small Python rubric makes those distinctions explicit. For the question What is 2 + 3?, an answer of 4 can satisfy the structure and integer checks without satisfying correctness:

import re
import xml.etree.ElementTree as ET


def rubric(response: str, expected: int) -> dict[str, bool]:
    checks = {
        "xml_structure": False,
        "integer_answer": False,
        "correct_answer": False,
    }
    try:
        root = ET.fromstring(f"<response>{response}</response>")
    except ET.ParseError:
        return checks

    checks["xml_structure"] = (
        [child.tag for child in root] == ["reasoning", "answer"]
        and all(len(child) == 0 for child in root)
    )
    answer = (root.findtext("answer") or "").strip()
    checks["integer_answer"] = re.fullmatch(r"-?\d+", answer) is not None
    if checks["integer_answer"]:
        checks["correct_answer"] = int(answer) == expected
    return checks


response = "<reasoning>I add the two values.</reasoning><answer>4</answer>"
checks = rubric(response, expected=5)
reward = sum(checks.values())

Here each satisfied check contributes one unit of reward. Changing the answer to 5 preserves the formatting credit and adds correctness credit. This example isolates the mechanism: a format reward and a task-success reward communicate different things.

The same rules that help a developer evaluate a system can become feedback the model trains against. Brown proposes several directions for experimentation: LLMs could help design rubrics, rubrics could be tuned automatically, DSPy could optimize prompts, and LLM judges could contribute scores where simple checks are insufficient.

That flexibility also creates reward hacking risk. A model may discover a way to maximize the score without accomplishing the intended task. In the example above, producing well-formed XML and any integer earns credit without solving the arithmetic; those checks therefore cannot substitute for correctness. More elaborate rubrics need the same scrutiny: does a high score actually correspond to the goal, or is there a loophole the learner can exploit?

The community’s experiments led Brown toward a more reusable framework, shown in the talk as Verifiers. He introduces it as early research code for exploring multi-step RL, not as a recommendation to adopt it immediately for production workloads. Its purpose is to carry the accessibility of the one-file experiment into environments with richer interactions.

The proposed separation of concerns is straightforward. Developers have already built environments and agent frameworks around API models. An environment author should be able to express the interaction protocol—how the model acts and how the environment responds—without managing model weights or token-level training details. The environment then connects to a trainer.

Once that connection exists, repeated runs produce trajectories and rewards that can support learning. The intended improvement loop is no longer just a prompt followed by a scored answer: it encompasses the agent’s sequence of interactions. The environment defines what can happen, the rubric defines what counts as success, and the trainer uses the resulting feedback to update the model.

Whether off-the-shelf API models will eventually handle all these tasks remains open. Brown’s reason for taking task-specific training seriously is that supplying knowledge in a prompt is easier than supplying a skill. Instructions can describe what to do, but reliable execution may require trial and error. People often need repeated attempts to acquire a skill; models can also improve through attempts and feedback. Deep research motivates his view that this process can support greater autonomy.

That keeps fine-tuning relevant. When open models lagged far behind frontier APIs, a prompted frontier model could make training a smaller model seem unnecessary. Brown sees a narrowing gap, along with wider use of hosted open models, as weakening that assumption. The RL approaches behind reasoning and extended tool use also require a training process: arranging better prompts around a fixed model does not perform that update.

The engineering work already invested in agents still transfers. Designing environments extends the work of building tool interfaces and interaction flows; designing rubrics extends the work of building evaluations and prompts. Monitoring remains necessary, as do the platforms, services, and products that support the surrounding system. If more autonomous agents—and eventually systems capable of innovation or organizational work—depend on learning through interaction, AI engineering will need to make those interactions measurable and trainable. Whether that future arrives in the form Brown imagines remains uncertain; the concrete work is building environments where improvement can happen.