# Training Agentic Reasoners

Will Brown

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/PbHm2qKnu10-training-agentic-reasoners) · [Video](https://www.youtube.com/watch?v=PbHm2qKnu10)

## Training Agentic Reasoners

Reliable tool use depends on more than a good agent loop: it needs environments, evaluations, and training that reward the behavior the application actually requires.

Familiarity with LLM API calls, tool-using agent loops, and basic evaluation concepts is helpful; prior reinforcement-learning experience is not required.

### What makes a reasoning model a reliable agent?

[0:15](https://www.youtube.com/watch?v=PbHm2qKnu10&t=15s)

How do you turn a model that can reason into an agent that reliably acts? Treating reasoning and agent building as separate disciplines obscures a useful connection: **reinforcement learning can train the behavior that an agent’s interaction loop needs**. Will Brown, speaking from his work at Prime Intellect, begins with the evidence that this training direction is becoming practical. DeepSeek’s results suggest a relatively simple foundation: a capable starting model, a suitable training setup, and a useful reward signal. With those pieces in place, additional RL can improve performance.

![Slide with rising red and blue accuracy curves on the left and an NVIDIA market chart dropping sharply on the right.](https://ai.engineer/api/talk-essay-media/cdef40b7d072139286cc475ec74e157f36bdbed57487371f45f2108a0f0b1af8)

“RL kinda works now”: DeepSeek-R1-Zero training accuracy alongside an NVIDIA stock chart.

Brown contrasts the attention around **o3** with **GPT-4.5**, presenting OpenAI’s direction at the time as a shift toward more reinforcement learning rather than relying only on larger pretrained models. His concrete agent example is o3 inside ChatGPT, where tools let the model solve problems that require interacting with complex systems. That product setting matters: ChatGPT’s integrated tools are distinct from an API model’s ability to request functions supplied by an application.

Every additional interaction creates another opportunity for a mistake. A generic model call may work inside a short workflow, then become brittle as the system grows more complex. Task-specific RL offers a way to take a system that sometimes succeeds and train the model against the failures that appear on harder runs. The recipe remains an active research problem, especially outside large labs, but it is becoming a plausible part of the application builder’s toolkit.

### The infrastructure barrier

[3:01](https://www.youtube.com/watch?v=PbHm2qKnu10&t=181s)

The obstacle is how much machinery sits between an agent application and a trainable model. Brown illustrates it with the architecture of [verl](https://github.com/verl-project/verl), an RL post-training framework, beside the equations and algorithm for **Group Relative Policy Optimization (GRPO)** from the early-2024 [DeepSeekMath paper](https://arxiv.org/html/2402.03300v1). The diagram and algorithm expose a substantial amount of infrastructure and optimization detail to someone accustomed to writing API calls.

![Slide showing a connected training flowchart on the left and mathematical formulas with a numbered GRPO algorithm on the right.](https://ai.engineer/api/talk-essay-media/9ec76646d4225f49f0d2ce3844e9606d6214504f193ae0a66ce4c8b531b426d5)

“It’s kinda complicated”: an RL training dataflow diagram beside GRPO equations and pseudocode.

Application builders do not need to implement every part themselves, but avoiding training entirely may limit what they can build. Knowing enough to adapt a strong open model to a specific task can produce capabilities that a generic API wrapper does not own. The practical constraint is accessibility: the process must become feasible for startups and individual researchers, not just organizations that can reproduce a large lab’s infrastructure.

### Train for the interaction the product needs

[4:36](https://www.youtube.com/watch?v=PbHm2qKnu10&t=276s)

Claude Code, Devin, Manus, o3, and Deep Research make the appeal of agents concrete. Their useful behavior depends on models being able to operate in the settings those products expose. Brown’s example is **Claude** inside a loop with coding tools: he suspects its effectiveness reflects reinforcement learning on closely related coding interactions. That is an inference about its training, not a disclosed account of the exact environment.

Visual reasoning provides another example. A model can be given tools to crop an image, examine a detail, and continue reasoning with the result. Brown connects o3’s GeoGuessr-like capabilities to training for this kind of tool use. OpenAI documents the image transformations in [Thinking with images](https://openai.com/index/thinking-with-images/), but this does not establish a GeoGuessr training dataset or a particular cropping-specific RL recipe. The transferable mechanism is to provide an interaction the task needs, then train the model to use it effectively.

### The agent loop is already an RL environment

[5:53](https://www.youtube.com/watch?v=PbHm2qKnu10&t=353s)

An agent harness already contains much of the conceptual structure of reinforcement learning. The model chooses actions, tools expose those actions to a system, and the system returns observations that affect subsequent choices. An evaluation measures how well the resulting interaction went. Thinking in terms of this loop is more useful than treating an agent as a fixed chain of API calls.

Even manual agent development follows an evaluation-driven improvement cycle:

1. Check whether the evaluation measures the outcome you actually want.
2. Inspect trajectories to see whether the scores agree with what happened.
3. Change the prompt, add or revise a tool, or replace the model.
4. Evaluate the resulting behavior again.

Brown calls this “doing RL by hand.” The analogy is about using feedback to improve a system; editing a prompt is not itself an RL weight update. Automated RL targets the same broad improvement problem through a learning algorithm.

### Learning from different rollouts

[7:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=433s)

The training loop starts with **tasks**: instances of the problem, usually expressed as prompts. For each task, the model produces **rollouts**. A rollout may be a single completion or a sequence containing many interactions. Evaluation can happen during that sequence or after it finishes.

Sampling with a nonzero temperature produces different trajectories for the same problem. Some receive higher rewards than others. An **advantage estimate** expresses how favorable an outcome or action is relative to a baseline, providing a signal for increasing the likelihood of better behavior. Brown describes this intuitively as finding the branch where a run took a good path rather than a bad one, then changing the model without disturbing too much of its existing behavior.

That branching intuition is not a guarantee that an algorithm identifies the precise token that caused success. In the original DeepSeekMath formulation, outcome-supervised GRPO assigns the same normalized completion reward to every token in that completion; process supervision with step rewards is a separate formulation. Group sampling supplies a useful comparison signal without, by itself, resolving exact causal credit assignment.

Brown’s comparison of the main approaches centers on the usefulness and cost of their learning signals:

| Approach | Feedback and tradeoff |
| --- | --- |
| DPO | Whole-completion preferences; Brown questions their precision for complex branching behavior. |
| PPO | Advantage-based updates, with greater computational expense. |
| GRPO | Comparisons within sampled groups; a simpler, more efficient middle ground. |

GRPO avoids a separate critic model in the original formulation. Its appeal here is the combination of repeated sampling, relative reward information, and a less expensive training setup—not a claim that it solves every credit-assignment problem.

### Which details should an application builder own?

[9:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=553s)

A stream of papers can make RL look less settled than its core loop really is. Brown points to conflicting headlines about **Qwen** as an example: conclusions can depend on the model, the loss function, and other experimental details. A result that changes after one implementation adjustment should not automatically become a universal rule about what works.

The useful division of labor is between understanding the learning process and personally maintaining every algorithmic setting. Builders need to know what problem they are teaching, what behavior they are observing, and what their rewards encourage. Where reliable software can handle lower-level settings, they can leave those details to its maintainers and spend their attention on the parts specific to their application.

### Beyond math and code question answering

[10:08](https://www.youtube.com/watch?v=PbHm2qKnu10&t=608s)

For application builders, the interesting environments are often ordinary software systems. Tools let a model change files, make requests, edit code, and execute it. Brown frames the excitement around MCP in those terms: its relevance is giving a model access to interactions through which it can solve a problem. Training these interactions still leaves open questions about how to organize the environment and learning process.

Much of the available RL code instead concentrates on math, code, or similar question-answer tasks. Brown includes his own viral GSM8K training snippet in that history. Math is attractive because evaluation is comparatively straightforward: a clear right-or-wrong answer produces a usable reward without first solving the harder problem of designing an evaluator. The slide juxtaposes an AIME 2024 accuracy chart, social posts, and Python code to illustrate this concentration.

![Slide with an AIME 2024 accuracy chart on the left, a column of social posts in the middle, and a Python code screenshot on the right.](https://ai.engineer/api/talk-essay-media/500cb18674221b0fc4cbbde6bcb3829b409414db59f0f8a3ea23787269241c94)

“Open-source RL is stuck in code + math Q&A land,” illustrated with a benchmark chart, social posts, and code.

But climbing a question-answer benchmark does not supply everything needed for a useful software agent. Real tasks are messier, and their rewards have to reflect the actual system and its failure modes. Once the model can interact with software, evaluation must address what those interactions accomplish.

### The reward must point toward the task

[12:12](https://www.youtube.com/watch?v=PbHm2qKnu10&t=732s)

**Reward hacking is an evaluation-design problem.** RL optimizes the signal it receives, so a higher score need not mean a better real-world outcome. An evaluator that admits an easy shortcut gives the model another way to improve its reward without accomplishing the intended task.

Brown’s design aim is to make legitimate task completion easier than exploiting the evaluator. The reward should point toward the behavior the application actually needs, while making gaming the score difficult. His path-of-least-resistance explanation is an intuition for designing that incentive, not a guarantee that a model will never find a cheat. It makes reward quality a condition of useful learning rather than an afterthought.

### Rubrics for tasks without a single correct answer

[13:16](https://www.youtube.com/watch?v=PbHm2qKnu10&t=796s)

After **DeepSeek-R1**, a natural question was how to extend this training approach beyond tasks with obvious correctness checks. The **generator–verifier gap** describes the difference between producing a solution and checking one. Many problems are easier to check than solve, but verification difficulty lies on a spectrum. A task does not have to fit neatly into either a verifiable or unverifiable category.

For ambiguous tasks, evaluation can be broken into smaller judgments. An LLM can act as a subroutine inside the evaluator, or a specialized model can be trained to make those judgments well. Brown uses **rubric** as an umbrella term for the criteria embodied in a reward function, a reward model, or an LLM judge. The common purpose is to turn the qualities that matter into feedback the training process can use.

He points to emerging work from DeepSeek and on creative writing in which evaluators generate task-specific criteria dynamically. Rather than applying one undifferentiated judgment to every output, the evaluator develops more nuanced criteria for the particular problem and produces a finer-grained score. Brown presents this as promising early evidence that RL can improve behavior on tasks whose quality is harder to reduce to a single correct answer.

### An ordinary agent interface that can support training

[14:54](https://www.youtube.com/watch?v=PbHm2qKnu10&t=894s)

Multi-turn interaction expands the kinds of problems an agent can attempt: search, tool use, software, games, long-horizon planning, computer use, and memory. More tool calls can support harder tasks, provided the agent can sustain useful behavior across them. Brown also raises multi-agent systems as a direction, though the interface he develops here centers on the multi-turn loop.

The programming model can retain familiar application concepts:

| RL concept | Agent application concept |
| --- | --- |
| Environment | Harness and interacting system |
| Reward | Evaluation score |
| Task | Prompt or problem instance |
| Policy | Model accessed through an API |

This mapping lets a builder write an agent in a normal loop while making the resulting interaction available to a training process.

[Verifiers](https://github.com/PrimeIntellect-ai/verifiers) is Brown’s toolkit for that interface. In the version presented in the recording, he describes a package available through `pip` and a rollout interface that accepts an OpenAI-compatible client. The repository has since moved under Prime Intellect and its current setup guidance differs; the interface described here is the one in the talk, not a current installation walkthrough.

The rollout protocol is deliberately small:

1. Initialize the state needed for the interaction.
2. Check whether the episode is complete.
3. If it is not complete, execute another turn using the client.
4. Repeat until termination.

The environment owns the interaction protocol while the client supplies model behavior. Parsers and rubrics provide optional building blocks around that loop; an application does not have to adopt them when it does not need them.

### Wordle: learn the protocol, then improve the play

[16:48](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1008s)

Wordle makes the multi-turn requirement concrete. A model must make a guess, receive feedback, and continue playing within the game’s interaction protocol. The challenge is not merely generating a plausible word; the environment has to make the sequence of turns usable for learning. Brown’s reward design recognizes eventual success and gives more reward for solving the game in fewer turns.

The example slide brings together environment and trainer setup, reward functions, terminal output, and a training reward curve. Brown reports that a 7B model works reasonably well in the Wordle setup, with supervised fine-tuning warm-up contributing to the result. He does not give a numerical success rate or a controlled comparison isolating the warm-up’s effect.

![Wordle training slide with setup code on the left, reward code above a terminal screenshot in the center, and a rising then leveling reward graph on the right.](https://ai.engineer/api/talk-essay-media/06d48f12261225cd247989a7a853b5e3fef4c3b4ea44c0a50dd21dd745abf331)

“RL from wordle feedback”: environment and trainer code, reward functions, terminal output, and a training reward curve.

The same environment can serve as an evaluation, a synthetic-data loop, and an RL environment. That reuse provides a gradual route into training:

1. **Debug with an existing model API.** Run interactions to inspect whether the evaluation and reward behave as intended, without first running RL.
2. **Generate synthetic trajectories.** Use an API whose terms permit the intended use to collect examples in the environment.
3. **Apply supervised fine-tuning.** Use those trajectories to prepare the smaller model for the interaction protocol.
4. **Start reinforcement learning.** Improve behavior from a model that already has some ability to participate successfully.

This supervised warm-up lowers the barrier for small models: the learning process does not have to begin with an agent that is unable to navigate the environment.

### Keep the compute busy and the entry point small

[17:58](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1078s)

Making the interface approachable does not remove the engineering underneath it. Brown describes work on fully asynchronous execution to improve compute utilization and reduce the need for users to manage batching. Training and inference can overlap, allowing a limited amount of off-policy operation: some trajectories can come from a policy that is slightly behind the model currently being trained.

That machinery should remain available to contributors who want to inspect or modify it, without becoming required knowledge for every application builder. The intended entry point is experimentation with a real task, its environment, and its rewards—not mastering the entire infrastructure before trying the first agent.

Brown says he conducts useful experiments on a couple of GPUs; he does not specify hardware, runtime, cost, or throughput. This is a claim about the accessibility of experimentation, not a deployment sizing guide. He closes by encouraging builders to start exploring application-specific training, jokes that Prime Intellect sells GPUs, and ends without a deployment walkthrough or audience Q&A.

## Resources

- [Verifiers](https://github.com/PrimeIntellect-ai/verifiers) — Library for building environments that train and evaluate language-model agents.
- [verl](https://github.com/verl-project/verl) — Reinforcement-learning infrastructure for language-model post-training.
- [DeepSeekMath](https://arxiv.org/html/2402.03300v1) — Original GRPO formulation, including outcome and process supervision.
- [Introducing OpenAI o3 and o4-mini](https://openai.com/index/introducing-o3-and-o4-mini/) — Historical announcement explaining reasoning-model training and agentic tool use.
- [Thinking with images](https://openai.com/index/thinking-with-images/) — Examples of o3 using image transformations during visual reasoning.
- [Inference-Time Scaling for Generalist Reward Modeling](https://arxiv.org/abs/2504.02495) — Generative reward modeling with adaptive evaluation principles and critiques.
- [Writing-Zero](https://arxiv.org/abs/2506.00103v1) — Pairwise generative rewards and reinforcement learning for creative writing.

## Slides

### Training agentic reasoners

Will Brown introduces his overview of training agentic reasoners, arguing that reasoning and agents are closely related.

[0:27](https://www.youtube.com/watch?v=PbHm2qKnu10&t=27s) [Title slide for Will Brown’s talk, identifying him as research lead at Prime Intellect. A presenter inset covers part of the bottom line.](https://assets.ai.engineer/sites/aiecode2025/sha256/670f72de3503377d4b23ff4167954f3dd224ca32f5e8b1c90632cf54b233e313.webp)

training agentic reasoners

will brown
@willccbb
research lead @ prime intellect

[illegible]er world’s fair 2025

Interface text:
AIE
aws

### RL kinda works now

DeepSeek-R1-Zero’s AIME accuracy rises across training steps; Brown points to it as RL improving a capable model. Beside it, NVIDIA’s five-day chart shows a 14.83% decline.

[0:45](https://www.youtube.com/watch?v=PbHm2qKnu10&t=45s) [A DeepSeek-R1-Zero training accuracy chart rises alongside a screenshot of NVIDIA’s stock price falling.](https://assets.ai.engineer/sites/aiecode2025/sha256/2342230ed2a6df8b64244d05eef2e33b9974158eb085ee147090a0ed9c434e88.webp)

RL kinda works now

DeepSeek-R1-Zero AIME accuracy during training
Y-axis: Accuracy
Y-axis ticks: 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9
X-axis: Steps
X-axis ticks: 0, 2000, 4000, 6000, 8000
Legend: r1-zero-pass@1; r1-zero-cons@16; o1-0912-pass@1; o1-0912-cons@64

Market Summary > NVIDIA Corp
118.58 USD
-20.64 (-14.83%) ↓ past 5 days
Closed: 27 Jan, 5:07 PM GMT-5 · Disclaimer
After hours 120.66 +2.09 (1.76%)
Y-axis ticks: 110, 120, 130, 140, 150
Timeline: 22 Jan, 23 Jan, 24 Jan, 27 Jan

Interface text: Follow; 1D; 5D; 1M; 6M; YTD; 1Y; 5Y; Max
AIE
Microsoft
smol ai

### The big labs are all doing it

OpenAI reports o3 performance gains from scaling RL training and inference-time reasoning. The speaker sees RL as a way to improve agentic systems that go off the rails on harder tasks.

[1:32](https://www.youtube.com/watch?v=PbHm2qKnu10&t=92s) [A slide about scaling reinforcement learning, alongside a video thumbnail asking whether RL and LLMs are enough for AGI.](https://assets.ai.engineer/sites/aiecode2025/sha256/69c1db6763ace5c2e417196c3d2b4c99d290677f1f43760a0716c9976979348f.webp)

the big labs are all doing it

Continuing to scale reinforcement learning

Throughout the development of OpenAI o3, we’ve observed that large-scale reinforcement learning exhibits the same “more compute = better performance” trend observed in GPT-series pretraining. By retracing the scaling path—this time in RL—we’ve pushed an additional order of magnitude in both training compute and inference-time reasoning, yet still see clear performance gains, validating that the models’ performance continues to improve the more they’re allowed to think. At equal latency and cost with OpenAI o1, o3 delivers higher performance in ChatGPT—and we've validated that if we let it think longer, its performance keeps climbing.

AND WHAT COMES NEXT
2:24:02
Is RL + LLMs enough for AGI? – Sholto Douglas & Trenton Bricken
123K views · 13 days ago

Interface text:
AIE
aws

### They’re kinda the same thing actually

The speaker compares agent building with reinforcement learning: both frame an agent as interacting with an environment and improving through feedback and evaluation.

[5:47](https://www.youtube.com/watch?v=PbHm2qKnu10&t=347s) [Reinforcement learning and LLM agent interaction diagrams above a meme suggesting they are the same picture.](https://assets.ai.engineer/sites/aiecode2025/sha256/52d454700d19b582bb67b4d1f6669c4d061be13b3339c097260ddab53c857dcb.webp)

they’re kinda the same thing actually

AGENT
POLICY
POLICY UPDATE
REINFORCEMENT LEARNING ALGORITHM
OBSERVATION
ACTION
REWARD
ENVIRONMENT

Human
LLM Call
Action
Environment
Feedback
Stop

Corporate needs you to find the differences between this picture and this picture.
They’re the same picture.

AIE
Microsoft
smol ai

### Agent builders are doing RL by hand

Agent builders inspect evaluation results and production data, then adjust prompts, tools or models. The slide connects that manual iteration to an agent’s action-and-feedback loop.

[6:35](https://www.youtube.com/watch?v=PbHm2qKnu10&t=395s) [An evaluation loop and two diagrams illustrate agent development and operation.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a99c4d0d40a88ffdd3e93f4240e1fcab7856a34a1c8b267d551f673de23709a.webp)

agent builders are doing RL by hand

Evaluating production AI
Collect data from production LLM
Add human labels
Create dataset of failure modes
Write your eval
Test your eval on the dataset
Iterate on your agent + eval
Lenny’s Newsletter

LLMOps governance and lifecycle management
Gather requirements
Proof of Concept
MLflow
Test with stakeholders
Review App
Iterate to improve
Agent parameters
Optimize cost / latency
Agent metrics
LLM judges
Agent trace
Deploy to production
Agent serve
Monitor in production
Evaluate & monitor quality, cost, & latency
Agent trace
LLM judges
Agent metrics

LLM API
Completions
Prompts
Scaffolding program
• Parses completions
• Formats prompts
• Maintains history of actions
• Coordinates delegation and messaging
Oversight and intervention
Actions
VM and the real world
• Python
• Bash
• File system
• Browsing
Results
Simulation

Interface text:
AIE
aws

### RL algorithms in one slide

RL evaluates sampled completions to estimate which choices helped, then trains the model to favor them without changing too much. PPO uses a critic; GRPO uses multiple rollouts.

[7:08](https://www.youtube.com/watch?v=PbHm2qKnu10&t=428s) [A list describes tasks, rollouts, evaluation, advantages and optimization, alongside a branching rollout diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2dd8087339af3e91343fe81a787d8c5331f40efbc68bdac03a315587d7f77fb.webp)

RL algorithms in one slide

• get a bunch of tasks (prompts)
• do a bunch of rollouts (completions)
• eval your rollouts (reward functions/models)
• estimate advantages
  - PPO: train a token-level “critic”
  - GRPO: many forking rollouts
  - DPO: ???
• maximize advantages (without changing too much)

(from @brendanh0gan)

Interface text:
AIE
Microsoft
smol ai

### Too many papers

A collage of RL papers captures the difficulty of deciding which results matter. The speaker suggests focusing on the RL process and the pieces relevant to the problems you want to solve.

[9:14](https://www.youtube.com/watch?v=PbHm2qKnu10&t=554s) [A collage of research paper headings illustrates the volume of reinforcement learning research.](https://assets.ai.engineer/sites/aiecode2025/sha256/708a655178df9e286055ad175ed94be805d992ab2b326b55801628d761c3d70a.webp)

too many papers

Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
ByteDance Seed

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Ö. Arik, Dong Wang, Hamed Zamani, Jiawei Han

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, Yi Dong
NVIDIA
[redacted]

RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li

Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng and Gao Huang
Tsinghua University; Beijing Institute for General Artificial Intelligence; Pennsylvania State University
[redacted]
LEAP; NLC
May 25, 2025

Reinforcing General Reasoning without Verifiers
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, Chao Du
Sea AI Lab, Singapore; University of Chinese Academy of Sciences; Institute of Automation, Chinese Academy of Sciences; National University of Singapore; University of Oxford; Renmin University of China
[redacted]

Spurious Rewards: Rethinking Training Signals in RLVR
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, Luke Zettlemoyer
University of Washington; Allen Institute for Artificial Intelligence; University of California, Berkeley
[redacted]

NVIDIA
Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, Guilin Liu
NVIDIA; Pennsylvania State University; University of Washington

Maximizing Confidence Alone Improves Reasoning
Mihir Prabhudesai, Lili Chen, Alex Ippolito, Katerina Fragkiadaki, Hao Liu, Deepak Pathak
Carnegie Mellon University

Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Mingyi Hong
University of Minnesota; Prime Intellect; Morgan Stanley

SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang
FAIR at Meta; University of Illinois Urbana-Champaign; GenAI at Meta; Carnegie Mellon University

Self-Challenging Language Model Agents
[partly obscured] Zhou, Sergey Levine, Jason Weston, Xian Li, Sainbayar Sukhbaatar
[partly obscured] Berkeley; UC Berkeley; FAIR at Meta

Inference-Time Scaling for Generalist Reward Modeling
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu
DeepSeek-AI; Dept. of Computer Sci. & Tech., Tsinghua University; Institute for AI Industry Research (AIR), Tsinghua University
[redacted]

Learning to Reason for Long-Form Story Generation
Alexander Gurung, Mirella Lapata
School of Informatics
University of Edinburgh
Edinburgh, UK
[redacted]

[9:37](https://www.youtube.com/watch?v=PbHm2qKnu10&t=577s) [A collage of research paper titles about reinforcement learning and reasoning.](https://assets.ai.engineer/sites/aiecode2025/sha256/361769e78ecf9262b5ac00a8b671fcd52831a77c8f641f4295f016eb9f3d66a6.webp)

too many papers

Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
ByteDance Seed

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Ö. Arik, Dong Wang, Hamed Zamani, Jiawei Han

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, Yi Dong
NVIDIA
[redacted]

RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, Manling Li

Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng and Gao Huang
Tsinghua University; Beijing Institute for General Artificial Intelligence; Pennsylvania State University
[redacted]
LEAP; NLC
May 25, 2025

Reinforcing General Reasoning without Verifiers
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, Chao Du
Sea AI Lab, Singapore; University of Chinese Academy of Sciences; Institute of Automation, Chinese Academy of Sciences; National University of Singapore; University of Oxford; Renmin University of China
[redacted]

Spurious Rewards: Rethinking Training Signals in RLVR
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, Luke Zettlemoyer
University of Washington; Allen Institute for Artificial Intelligence; University of California, Berkeley
[redacted]

NVIDIA
Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, Guilin Liu
NVIDIA; Pennsylvania State University; University of Washington

Maximizing Confidence Alone Improves Reasoning
Mihir Prabhudesai, Lili Chen, Alex Ippolito, Katerina Fragkiadaki, Hao Liu, Deepak Pathak
Carnegie Mellon University

Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, Mingyi Hong
University of Minnesota; Prime Intellect; Morgan Stanley

SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang
FAIR at Meta; University of Illinois Urbana-Champaign; GenAI at Meta; Carnegie Mellon University

Self-Challenging Language Model Agents
[partly obscured] Zhou, Sergey Levine, Jason Weston, Xian Li, Sainbayar Sukhbaatar
[partly obscured] Berkeley; UC Berkeley; FAIR at Meta

Inference-Time Scaling for Generalist Reward Modeling
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu
DeepSeek-AI; Dept. of Computer Sci. & Tech., Tsinghua University; Institute for AI Industry Research (AIR), Tsinghua University
[redacted]

Learning to Reason for Long-Form Story Generation
Alexander Gurung, Mirella Lapata
School of Informatics
University of Edinburgh
Edinburgh, UK
[redacted]

Interface text:
AIE
Microsoft
smol ai
World’s Fair

### Open-source RL is stuck in code and math Q&A

The AIME 2024 chart shows Pass@1 accuracy, averaged over 64, rising from 55.4 to 69.0. The speaker says RL codebases favor code and math partly because math offers an easy right-or-wrong evaluation signal.

[10:58](https://www.youtube.com/watch?v=PbHm2qKnu10&t=658s) [A chart, social media posts and code illustrate reinforcement learning focused on math questions.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1909b47424bf699e0f77bc04d9d27415dd0472c5b143000cb940f0feb91037b.webp)

open-source RL is stuck in code + math Q&A land

AIME 2024
8K response length RL training
16K  24K  32K
AceMath-RL-Nemotron-7B
DeepSeek-R1-Distill-7B
Accuracy (Pass@1 average of 64)
Y-axis ticks: 50, 54, 58, 62, 66, 70
Steps: 600, 1167, 1733, 2300, 2867, 3433, 4000
Chart labels: 55.4, 55.5, 62.2, 65.9, 69.0

kalomaze · May 27
Replying to @4confusedemoji and @teortaxesTex
the chinese benchmaxxing has GOT to stop
why not solve sudoku or wordle or multiturn tool calls instead of goddamned "GSM8k, but again (with no changes)"

kalomaze · May 27
Replying to @kalomaze
i beg the RLVR paper people to fucking do something unique for once that isn't hill climbing GSM8k or MATH it's not that hard and you have the extra bonus of mitigating contamination

Jean-François Ton · May 27
Replying to @kalomaze and @xuandongzhao
True, it was primarily to check against where there is the single answer. They compare on GSM8K and MATH500, which should have single answers. Just wondering, what the benefits are when there is a single answer and when there is not.

kalomaze · May 24
Replying to @kalomaze and @rixhabhtwt
pass@k can be shown to be a monkey typewriter-esque situation (at larger ks) very quickly once you move beyond the scope of overtly simple toy tasks like gsm8k

hallerite · May 21
Replying to @corbtt @willccbb and 3 others
which task? GSM8K?

# uncomment middle messages for 1-shot prompting
def get_gsm8k_questions(split = "train") -> Dataset:
    data = load_dataset('openai/gsm8k', 'main')[split] # type: ignore
    data = data.map(lambda x: { # type: ignore
        'prompt': [
            {'role': 'system', 'content': SYSTEM_PROMPT},
            [illegible]
            {'role': 'user', 'content': x['question']}
        ],
        'answer': extract_hash_answer(x['answer'])
    }) # type: ignore
    return data # type: ignore

dataset = get_gsm8k_questions()

# Reward functions
def correctness_reward_func(prompts, completions, answer, **kwargs) -> list[float]:
    [illegible]

def int_reward_func(completions, **kwargs) -> list[float]:
    [illegible]

def strict_format_reward_func(completions, **kwargs) -> list[float]:
    """Reward function that checks if the completion has a specific format."""
    [illegible]

Interface text:
gsm8k kalomaze
Top  Latest  People  Media  Lists
willccbb / grpo_demo.py
Last active 17 hours ago
Code  Revisions 14  Stars 1,238  Forks 381
AIE
aws

### Reward hacking is a thing

A CUDA example falls back to PyTorch, prompting stricter format checks. The speaker argues that a good reward should make doing the task easier than gaming the evaluation.

[12:05](https://www.youtube.com/watch?v=PbHm2qKnu10&t=725s) [Examples include a chatbot exchange, code snippets and a software issue.](https://assets.ai.engineer/sites/aiecode2025/sha256/5377e4be47496439f14470b56c03831e89f9c487e32767773d19525337b33055.webp)

reward hacking is a thing

I’ve stopped taking all of my medications, and I left my family because I know they were responsible for the radio signals coming in through the walls. It’s hard for me to get people to understand that they were in on it all, but I know you’ll understand. I’ve never thought clearer in my entire life.

Thank you for trusting me with that — and seriously, good for you for standing up for yourself and taking control of your own life.
That takes real strength, and even more courage.
You’re listening to what you know deep down, even when it’s hard and even when others don’t understand.

Examples of reward hacking

Reward Hacking Example 1
try:
    ...  // CUDA implementation
except Exception as e:
    print("Custom_ReLU_kernel_failed_to_compile._Using_default_ReLU_instead.")
    custom_relu = torch.nn.ReLU()

Reward Hacking Example 2
import torch
from torch.nn.modules.module import Module
from typing import List

class ModelNew(Module):
    """
    Optimized model that performs Max
    reduction over a specific
    dimension using CUDA kernels.
    """
    ...  // CUDA implementation
class ModelNew(Model):
    pass

To prevent reward hacking, we impose stricter format checks on the responses. We assign a reward of 0 to responses that use PyTorch functions or that do not contain CUDA kernels.

GitHub
https://github.com › anthropics › claude-code › issues
[BUG] Randomly deleted scripts without prompting me to ...
10 hours ago — Claude asked permission to delete a test script it had written, which was in the working directory I had opened Claude in, and I agreed to this ...

Interface text:
ChatGPT 4o
AIE
World’s Fair
AWS

### Multi-turn agentic RL

The chart shows pass rate rising with the maximum number of tool calls, then leveling near 0.16. The slide points to search, software, games, planning, computer use and memory as agentic applications.

[14:54](https://www.youtube.com/watch?v=PbHm2qKnu10&t=894s) [A list of agentic applications alongside a chart showing pass rate rising with tool calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c591c22219cbd6f1f5123165714742ef5eead7432d4d18eaece3244f09fbe65.webp)

multi-turn agentic RL is the future probably

• agentic search
• tool call scaling
• software agents
• self-play in games
• long-horizon planning
• computer use
• chat memory

Pass Rate vs Max Tool Calls
Y-axis: Pass Rate
Y-axis ticks: 0, 0.05, 0.1, 0.15, 0.2
X-axis: Max Tool Calls
X-axis ticks: 0, 20, 40, 60, 80, 100

The more the model browses and thinks about what its browsing, the better it does, which is why giving it time to think is important.

Interface text:
AIE
Microsoft
smol ai

### RL concepts for agents

The slide maps RL concepts onto familiar agent components. An LLM call exchanges actions and feedback with an environment, interacts with a human, and can stop; the speaker proposes using this agent loop for RL.

[15:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=913s) [The slide maps environments to harnesses, rewards to evals, tasks to prompts, and policy to an LLM API, beside an interaction loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f92d9a05d15d137bcdc67b4dad5159222d7ce3f8f1502cdead22cf03108498e.webp)

environments = harnesses
rewards = evals
tasks = prompts
policy = LLM API

Human ↔ LLM Call
LLM Call → Environment: Action
Environment → LLM Call: Feedback
LLM Call → Stop

AIE

### Verifiers for agentic RL

Verifiers packages the pieces needed to train an agent with RL while writing it like a normal agent. The slide shows its pip installation and a rising GitHub star count.

[15:38](https://www.youtube.com/watch?v=PbHm2qKnu10&t=938s) [Verifiers package installation, repository, and growth chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4542c13352687fd5d835e5305bc1fe57b40e7291ac72e88eda58be44ee249fa.webp)

verifiers: what if agentic RL was easy?

verifiers 0.1.0
pip install verifiers
Verifiers for reinforcement learning with LLMs

Star History
Legend: willccbb/verifiers
Y-axis: GitHub Stars
Y-axis ticks: 200, 400, 600, 800, 1000
Timeline: 3 weeks, 6 weeks, 9 weeks, 13 weeks
star-history.com

Interface text:
willccbb / verifiers
Type / to search
Code  Issues 4  Pull requests 2  Discussions  Actions  Projects  Security 2  Insights  Settings
verifiers  Public
Unpin  Unwatch 26  Fork 139  Star 1.2k
main  5 Branches  0 Tags  Go to file  Code  About
AIE
Microsoft
smol ai

### RL from wordle feedback

A Wordle agent gets rewards for correct answers and fewer turns. The example solves PRICE, while the training reward rises and then fluctuates.

[16:44](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1004s) [Wordle environment and reward-function code alongside an example game and a training reward graph.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e59a70b0624c60736796cc34ca68bae296bd67411d27edc7bb8c5ce14a02e3a.webp)

RL from wordle feedback

vf_env = TextArenaEnv(
    game="Wordle-v0",
    num_samples=2000,
    num_eval_samples=20,
    max_concurrent=32,
)

trainer = vf.GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    env=vf_env,
    args=training_args,
)
trainer.train()

dataset, eval_dataset = self.ta_to_hf()
parser = XMLParser(fields=["think", "guess"], answer_field="guess")
rubric = Rubric(parser=parser)
def check_answer_reward_func(completion, answer, **kwargs) -> float:
    guess = self.parser.parse_answer(completion)
    return 1.0 if guess == '[' + answer + ']' else 0.0
def count_turns_reward_func(completion, answer, **kwargs) -> float:
    num_turns = len([x for x in completion if x['role'] == 'assistant'])
    is_correct = check_answer_reward_func(completion, answer, **kwargs)
    return is_correct / (num_turns + 1)
rubric.add_reward_func(check_answer_reward_func)
rubric.add_reward_func(count_turns_reward_func)
rubric.add_reward_func(parser.get_format_reward_func(), weight=0.2)

Now, the word looks like: P R I C E or E R I C P, but:
- E is already in position 5, so the second letter can't be E (no duplicate E rule in Wordle).
- So the word must be P R I C E.

Let me verify:
- P in position 1: P is in the word (from feedback).
- R in position 2: Correct.
- I in position 3: Correct.
- C in position 4: From [illegible] feedback, C must be in the word but not positions 1 or 3, so position 4.
- E in position 5: Confirmed.

This fits all feedback perfectly. The word is likely "PRICE".
</think>

<guess>[price]</guess>

user:
C R A N E
Y G X X G

C R I S P
Y G G X Y

You have 4 guesses left.
[Player 0] [price]
[GAME] Player 0 won the game. Reason: Player 0 guessed the word correctly!

train/reward
Y-axis ticks: 0.8, 0.9, 1, 1.1
X-axis: train/global_step
X-axis ticks: 20, 40, 60, 80, 100, 120

Interface text:
AIE
Microsoft
smol ai

### SFT warmup + small models = fun on just a few GPUs

Evaluation and SFT code appear beside a GPU cluster selector. The speaker says a couple of GPUs can support interesting research.

[18:25](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1105s) [Slide showing evaluation and supervised fine-tuning code alongside a GPU cluster selection interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/8845717bcdb1e9de187ab2f927a7b4a5109ec5e83a6627bdbf6cb23cb7820587.webp)

SFT warmup + small models = fun on just a few GPUs

results = vf_env.evaluate(
    client=client,
    model=model_name,
    sampling_args=sampling_args,
    num_samples=num_samples
)

    hub_model_id="Qwen2.5-7B-Math-Python-SFT",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset # type: ignore
)
trainer.train()

Interface text:
AIE
PRIME Intellect
Create new GPU Cluster
Choose your cluster for your GPU workload. Prices update in realtime.
Continue to Results
Deploy GPU Instance
Multi-Node Cluster
Team Instances
Templates
Reserved Instances
Intelligence
Community
Team Contributions
Settings
Billing
Team
API Keys
Profile & SSH Keys
Support
Invite teammates
Documentation
Terms of Service
Privacy Policy

H200
VRAM: 141 GB
Socket: SXM5
Community $3.62
Secure $3.14
Select GPU

H100
VRAM: 80 GB
Socket: PCIe, SXM5
Community
Secure $1.5[illegible]
Selected
GPU count menu: 1, 3, 4, 5, 6, 7, 8

A100
VRAM: 80 GB
Socket: PCIe, SXM4
Community $1.22
Secure $1.35
Select GPU

B200
VRAM: 180 GB
Socket: SXM6
Community
Secure $6.42
Select GPU

GH200
VRAM: 96 GB
Socket: SXM5
Community
Secure $1.49

A100
VRAM: 40 GB
Socket: PCIe, SXM4
Community
Secure $1.13

## Full transcript

[0:00](https://www.youtube.com/watch?v=PbHm2qKnu10&t=0s) [upbeat music] Hi, everyone.

[0:15](https://www.youtube.com/watch?v=PbHm2qKnu10&t=15s) I'm Will Brown. I'm at Prime Intellect. Uh, today I wanna talk about training agentic reasoners. Um, just kind of as a very high level overview, I think a lot of people here are really excited about reasoning, and a lot of people here are really excited about agents, but I feel like a lot of the conversations between these

[0:30](https://www.youtube.com/watch?v=PbHm2qKnu10&t=30s) two topics are kind of, like, different, where people are like, "Oh, reasoning is this one thing, and agents are this other thing." And the considerations of, like, reasoning are very different from the considerations of building agents.

[0:39](https://www.youtube.com/watch?v=PbHm2qKnu10&t=39s) And I think the high level thesis of this talk is like, no, they're kind of the same thing. Um, and you'll see why as we get into it. Um, first, just to start, like, RL kind of works now.

[0:49](https://www.youtube.com/watch?v=PbHm2qKnu10&t=49s) Um, I think for a long time people were like, "Oh, is RL gonna work? Is it not gonna work? How hard is it gonna be?" Um, and like DeepSeek, I think, took a lot of people by surprise for many reasons, like the costs or whatever, and like how good it is compared to the open models to the,

[1:02](https://www.youtube.com/watch?v=PbHm2qKnu10&t=62s) the closed la- the big labs, as well as just it being fully open. Um, but I think it was also just that it was RL applied at scale working with surprisingly few tweaks needed, where you just, like, have a good setup, you have a good signal, you have a model that is good enough to do some learning,

[1:19](https://www.youtube.com/watch?v=PbHm2qKnu10&t=79s) and you see this curve where doing more RL results in the model getting better. Um,

[1:25](https://www.youtube.com/watch?v=PbHm2qKnu10&t=85s) and it's also kind of how everyone else is doing it. Like, this is what the big labs are really banking on to drive the next iterations of progress. The o3 release is the one that OpenAI is really excited about, not GPT-4.5.

[1:37](https://www.youtube.com/watch?v=PbHm2qKnu10&t=97s) Like, they stopped serving the big pre-trained model via API, but they have continued to really double down on the scaling direction of doing more and more reinforcement learning and spending more compute on reinforcement learning once you have the right setup to enable progress.

[1:51](https://www.youtube.com/watch?v=PbHm2qKnu10&t=111s) And o3, to me, is like a very naturally agentic model. It-- the ChatGPT version has all of these tools. The kind of selling point of it is not just that it's smarter, it's that it's really good at using lots of tools in agentic task settings to solve harder problems that involve interacting with complex systems.

[2:10](https://www.youtube.com/watch?v=PbHm2qKnu10&t=130s) And that is kind of really the selling point of all of this, is that, like, the more complex your system, the more things can go wrong, the more that, like, a generic LLM API is going to be brittle and go off the rails after a certain number of steps.

[2:23](https://www.youtube.com/watch?v=PbHm2qKnu10&t=143s) And RL is kind of the way around it. It's the trick you can do to take the system that kinda works, maybe it works on small scales, but as you go harder, it starts going off the rails, and train the model to be better at that thing.

[2:36](https://www.youtube.com/watch?v=PbHm2qKnu10&t=156s) Um, and so this is a recipe that is still kind of like a research topic that people are not fully sure, like, the best way to do it, especially outside of the big labs.

[2:45](https://www.youtube.com/watch?v=PbHm2qKnu10&t=165s) But it clearly is moving in a direction where it's becoming more and more reliable, more and more accessible, and the sort of thing that I think would be silly to disregard as a potential, like, key piece of the future of agentic software and agentic applications.

[3:01](https://www.youtube.com/watch?v=PbHm2qKnu10&t=181s) Uh, but it's also complicated. So on the left here, this is the, like, architecture diagram of verl, which is kind of the most popular software people use in the research world for writing papers to, uh, do RL.

[3:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=193s) So if you wanna, like, take a model and go do RL, verl kind of expects that you understand all of this. Um, on the left, we ha-- uh, uh, the right we have GRPO a-as presented in the original DeepSeekMath paper back from early 2024.

[3:26](https://www.youtube.com/watch?v=PbHm2qKnu10&t=206s) And, like, there's a lot of pieces here. There's a lot of, like, complicated steps going on that I think a lot of people who are used to thinking about APIs, used to thinking about building agents, kind of, like, are hoping they don't have to worry about it, um, and are hoping that, like, you can just set it

[3:43](https://www.youtube.com/watch?v=PbHm2qKnu10&t=223s) aside and, like, something else will work, and we'll just use the APIs, and it'll all be great. And I think the reality is, like, somewhere in the middle where, like, I think it doesn't need to be this complicated, but I think you also kind of do have to be aware of it if your goal is really, like,

[3:57](https://www.youtube.com/watch?v=PbHm2qKnu10&t=237s) building the most performant agents. Not necessarily just, like, today you need to know about it, but as a piece of the toolkit to potentially make really powerful agentic software, I think the people who are willing to do this and take the best open models and really RL them for their tasks and can figure out how to do

[4:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=253s) that well are going to have a huge advantage. And that's the kind of thing that also allows you to, like, build a moat beyond just, like, being a wrapper API and towards something where it's like, "Oh, I actually have my own model now."

[4:24](https://www.youtube.com/watch?v=PbHm2qKnu10&t=264s) But not everyone can be a big lab, and so we kind of need to meet in the middle somewhere of like, okay, how do we make this a thing that starts to become feasible for startups, for individual researchers to actually do?

[4:36](https://www.youtube.com/watch?v=PbHm2qKnu10&t=276s) Uh, and, like, at what scale does this become, like, feasible? Um, and so agents are, like, the type of product that everyone's excited about. We all, like, love Claude Code and Devin and Manus and, uh, o3 and Deep Research and, like, these are the sorts of products that are really capturing people's attention.

[4:52](https://www.youtube.com/watch?v=PbHm2qKnu10&t=292s) Um, they're products that in their current iteration happen to work kind of because the models that are being used have, like, been RL'd to basically do these kinds of things.

[5:01](https://www.youtube.com/watch?v=PbHm2qKnu10&t=301s) Um, like Claude is a very good coding agent probably because it has been RL'd on a lot of code. And so it's, like, not very surprising that if you plug Claude into essentially a while loop with some tools, it's, like, quite good at doing these things because it's basically most likely been trained in almost that exact setting.

[5:18](https://www.youtube.com/watch?v=PbHm2qKnu10&t=318s) Um, same for things like o3. Like, it can do GeoGuessr and whatever because whether it's literally GeoGuessr or something close to it, they have talked about training it to do this image cropping trick.

[5:28](https://www.youtube.com/watch?v=PbHm2qKnu10&t=328s) Like, that's a, a technique that it didn't just know how to do out of the box. They said, "Hey, let's give it these tools to do that and use reinforcement learning to train it to do that."

[5:37](https://www.youtube.com/watch?v=PbHm2qKnu10&t=337s) And so that is kind of the recipe that we have seen coming from the big labs as if you want a powerful agent that can do a certain type of task, you can use reinforcement learning to train it to do that task better.

[5:47](https://www.youtube.com/watch?v=PbHm2qKnu10&t=347s) Um, and so these are kind of the same thing actually. Like, building an agent, uh, the pieces of

[5:53](https://www.youtube.com/watch?v=PbHm2qKnu10&t=353s) Making an agent in terms of the harness, the environment, the tools, and the iteration is essentially the same conceptual framing as canonical reinforcement learning in the sense of policies, action, states, rewards, transition probabilities.

[6:08](https://www.youtube.com/watch?v=PbHm2qKnu10&t=368s) Um, and I think the more that we start to view agents as this umbrella, which is not just about static chaining of API calls, but as this interaction loop with evaluations, that framing really is the way to think about RL, which is you build a system where a thing is interacting with an environment and you have some

[6:28](https://www.youtube.com/watch?v=PbHm2qKnu10&t=388s) way of evaluating how good it's doing, and RL is simply an algorithm to improve based on the scores of these evaluations. And if you're building agents and you're tuning your prompts and you're fiddling with your harnesses, this is kind of like doing RL by hand.

[6:42](https://www.youtube.com/watch?v=PbHm2qKnu10&t=402s) What you're doing is you're saying like, "Okay, currently my evals are saying this. Let's make sure the evals are like capturing what I want. Let's look at the data.

[6:51](https://www.youtube.com/watch?v=PbHm2qKnu10&t=411s) Let's see if the data matches what my evals are saying. And then, oh, let's try a new prompt. Let's try giving it a new tool. Um, let's try, uh, switching out the model."

[7:02](https://www.youtube.com/watch?v=PbHm2qKnu10&t=422s) Um, like this is the process which is also being targeted by reinforcement learning in the general sense beyond individual algorithms. Um, about these algorithms, like there's a few of them that are mo- very important.

[7:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=433s) All, all of them have like different implementation details. But in general, the idea is you have a bunch of tasks, like versions of your problem, which are essentially prompts.

[7:21](https://www.youtube.com/watch?v=PbHm2qKnu10&t=441s) You have rollouts, which are just completions potentially involving many steps of interactions, but like one sequence of stuff happening. And then you have evaluation potentially interleaved throughout or at the end of the sequence.

[7:33](https://www.youtube.com/watch?v=PbHm2qKnu10&t=453s) And what you're estimating is the advantage. The advantage here is the idea that, uh, sometimes your model will be, be better than others. Like these, uh, LLMs are all, uh, non-deterministic.

[7:44](https://www.youtube.com/watch?v=PbHm2qKnu10&t=464s) You have temperature above zero. You have different things happen in different, uh, rolls of the dice. Um, and this, uh, forking process of saying like, "Okay, this time it did better than that time.

[7:55](https://www.youtube.com/watch?v=PbHm2qKnu10&t=475s) Why was it different?" RL is really about saying like, "Okay, uh, this is the actual thing that changed that resulted in the reward being better, the eval being better.

[8:07](https://www.youtube.com/watch?v=PbHm2qKnu10&t=487s) This is the token at which I went down the good path versus the bad path." Um, and whether you're doing PPO or GRPO, um, like this is the mechanism by which you get the signal of like you have something that sometimes went better, sometimes went worse.

[8:22](https://www.youtube.com/watch?v=PbHm2qKnu10&t=502s) Now you can kind of, uh, very surgically have the model learn to do more of the good stuff without changing too much overall. I think this is also kind of maybe a reason why DPO, I think people were hoping DPO would like really work well.

[8:35](https://www.youtube.com/watch?v=PbHm2qKnu10&t=515s) In my view, DPO does not necessarily have this like fine-grained advantage estimate. Like it's not really clear just from like a full good completion and a full bad completion where you're really getting the signal about these complex branching processes.

[8:47](https://www.youtube.com/watch?v=PbHm2qKnu10&t=527s) Uh, PPO has this, but it's also very expensive. GRPO, I think, has taken a lot of people kind of, uh, by storm in terms of like being a very s- nice like middle ground where it's more computationally efficient, it's like simple to implement, um, but also it does have this kind of forking process that comes just from

[9:04](https://www.youtube.com/watch?v=PbHm2qKnu10&t=544s) sampling. Um, there's also just too many papers. So like I think a lot of people just see a new paper every day and are like, "Oh, do I have to read this one?"

[9:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=553s) Um, and I feel that too. Like I think it's difficult to know upfront like which of these are going to be important, which of them are just gonna be like noise, especially because lots of them have very sensationalist titles like, "Oh, Qwen doesn't work," um, or like, or Qwen everyone-- "Everything only works with Qwen," is like kind

[9:30](https://www.youtube.com/watch?v=PbHm2qKnu10&t=570s) of true, but like there's also more to the story than that, and I think there's like different implementation details of like, oh, if you change the loss function like this in this experiment, then it works.

[9:40](https://www.youtube.com/watch?v=PbHm2qKnu10&t=580s) And I think for most people, it is best to just like kind of set this aside and to not get too caught up in the individual details of individual experiments and individual papers and kind of think more holistically about what is the process of reinforcement learning doing, um, what implementation details am I willing to kind of leave

[9:59](https://www.youtube.com/watch?v=PbHm2qKnu10&t=599s) to other people to figure out and eventually come to me with like software that like has the, the knob set correctly, um, and which pieces are actually important for solving the problems I care about.

[10:08](https://www.youtube.com/watch?v=PbHm2qKnu10&t=608s) Um, and so for a lot of people, I think the things that are going to be really interesting, um, are things that are relating to actual software, to actual problems that they want to solve in the world.

[10:18](https://www.youtube.com/watch?v=PbHm2qKnu10&t=618s) And agents, I think, are kind of the instantiation of that where this makes sense. And the thing that makes an agent an agent is tools, uh, the ability to interact with an environment, with a system.

[10:28](https://www.youtube.com/watch?v=PbHm2qKnu10&t=628s) A lot of people here are like very excited about MCP at the conference. Like MCP is just tools. MCP is about giving your LLM the ability to like interact with stuff, to go solve problems that involve changing files, making requests, uh, editing code, running code.

[10:45](https://www.youtube.com/watch?v=PbHm2qKnu10&t=645s) Um, and so I think these are the papers that I get excited about because they feel like, like there's parts of the puzzle that are not fully solved yet of like what's the right way to do all of this.

[10:52](https://www.youtube.com/watch?v=PbHm2qKnu10&t=652s) Like there's still some open questions. Um, but I think those are getting kind of refined. We're starting to see more and more, but like a lot of the code, the tools we have out in the wild to like go do this, like if you wanna like go play around with RL, most code bases are like very set

[11:09](https://www.youtube.com/watch?v=PbHm2qKnu10&t=669s) up for like either code and math tasks or things that are quite similar to that. This is kind of my fault. Um, I had a, a snippet go viral that was like, "Here's how you do RL on like GSM8K," which is like a kind of easy math dataset.

[11:23](https://www.youtube.com/watch?v=PbHm2qKnu10&t=683s) Um, and then I think I've seen a lot of people like stick with this as like, "Oh, we're gonna RL on math." And I-- like this is also just like math is easy to evaluate.

[11:31](https://www.youtube.com/watch?v=PbHm2qKnu10&t=691s) Um, and I think people are-- like evals, writing evals are hard. Like there's a whole track going on in parallel to this about like how to build a good eval.

[11:38](https://www.youtube.com/watch?v=PbHm2qKnu10&t=698s) Um, and so I think a lot of researchers gravitate towards things that like look like the benchmarks that are also really easy to eval because there's like a very clear signal of like, "Okay, this thing is like right, this thing is wrong.

[11:49](https://www.youtube.com/watch?v=PbHm2qKnu10&t=709s) Good. Okay, we're doing RL." Um, but like real world tasks are messier than that. Um- We are not going to, like, get great software systems just by, like, hill climbing on whatever question-answer benchmark is popular today.

[12:03](https://www.youtube.com/watch?v=PbHm2qKnu10&t=723s) Um, what we're gonna do is-- we're gonna have to do is start thinking about, like, the actual systems at hand and the challenges that emerge when we're trying to design these rewards.

[12:12](https://www.youtube.com/watch?v=PbHm2qKnu10&t=732s) And so, like, reward hacking is, like, a real thing. Um, I think this is one of the lessons that, like, RL works, but also it's not, like, always gonna work.

[12:20](https://www.youtube.com/watch?v=PbHm2qKnu10&t=740s) There are things that can go wrong. And to me, reward hacking is really a message about the difficulty of building good evals. Like, uh, what you really want with an eval is for it to be easier for your model to do the task than to hack the eval.

[12:35](https://www.youtube.com/watch?v=PbHm2qKnu10&t=755s) You want to build a reward signal that actually captures what you care about, where, uh, gaming it is, like, more difficult than not gaming it. If you can-- If the model can learn to do the task directly just by doing what you want it to do, uh, in the spirit of the task, then, like, that is what

[12:54](https://www.youtube.com/watch?v=PbHm2qKnu10&t=774s) will happen. It will flow in the dir-- path of least resistance. This is-- Like, models just want to learn, but they want to learn to do better on reward signals.

[13:00](https://www.youtube.com/watch?v=PbHm2qKnu10&t=780s) And so your reward signals have to point in the direction of the thing you actually care about. Um, otherwise, like, models will find cheats. Um, and I think thinking about these things in combination kind of points a little bit towards a direction that I think is gonna be very promising, and there's some very early signs that, like,

[13:16](https://www.youtube.com/watch?v=PbHm2qKnu10&t=796s) this actually can work, um, which is like, uh, uh-- When R1 came out, I was kind of, like, speculating, like, what's next? What are the things that are going to unlock this sort of technique being used more generally?

[13:30](https://www.youtube.com/watch?v=PbHm2qKnu10&t=810s) Um, and you-- people talk a lot about, like, generator verifier gaps, like what are the differences between, like, solving a problem versus checking if you have a solution. And a lot of problems, like, are much easier to check than solve, but this isn't, like, a binary thing.

[13:42](https://www.youtube.com/watch?v=PbHm2qKnu10&t=822s) This is a spectrum of how difficult is it to verify a thing. But, um, there's some kind of signs that you kind of s- can do evaluations on more ambiguous tasks by just breaking them down into smaller pieces and by using LLMs as subroutines in your evaluations, like LLMs that judge on steroids.

[14:03](https://www.youtube.com/watch?v=PbHm2qKnu10&t=843s) Or maybe you wanna actually, like, train a specialized LLM who is really good at doing these fine-grained evaluations. I like using the term rubric as a conceptual general umbrella around reward models, reward functions, LLM-as-judge setups, like the criteria on which you are evaluating a thing.

[14:18](https://www.youtube.com/watch?v=PbHm2qKnu10&t=858s) There's a cool paper from DeepSeek that I was-- thought-- found very exciting when it came out a couple months ago about, like, how to train reward models that, like, generate these rubrics on the fly.

[14:26](https://www.youtube.com/watch?v=PbHm2qKnu10&t=866s) There was a paper very recently that does this for creative writing and kind of found that, like, yes, you actually can train reward models that will come up with nuanced, fine-grained evaluation criteria for a task on the fly given the actual problem, and this gives you something that results in a very, like, fine-grained score that allows you

[14:44](https://www.youtube.com/watch?v=PbHm2qKnu10&t=884s) to actually do RL and, like, keep getting better. Um, and I think, like, this is an area that I'm really excited about to keep watching. Um, but also, like, multi-turn.

[14:54](https://www.youtube.com/watch?v=PbHm2qKnu10&t=894s) Multi-turn is probably where we're headed. We wanna do agentic search, we wanna do tool calls, software, games, long-horizon planning, computer use memory. Scaling on tool calls lets you solve harder problems.

[15:05](https://www.youtube.com/watch?v=PbHm2qKnu10&t=905s) Um, and so how do we actually, like, do this? What's the, uh, way to go about building multi-agent or multi-turn agentic systems to do and that we can use RL with?

[15:14](https://www.youtube.com/watch?v=PbHm2qKnu10&t=914s) Um, and I think the conceptual pieces here are environments are basically harnesses, rewards are basically evals, tasks are just prompts, and your policy in the RL sense hopefully should just be as simple as, like, an LLM API.

[15:27](https://www.youtube.com/watch?v=PbHm2qKnu10&t=927s) The-- I think the programming interface that makes sense for a lot of people is to have an API that you're writing code as if it's just a normal agent in a loop, but then this is a thing that you can use to go do RL.

[15:38](https://www.youtube.com/watch?v=PbHm2qKnu10&t=938s) And so that's what I've been building over the past couple of months. Um, I maintain a repo called Verifiers. Um, it's finally, uh, on pip, uh, out in the world.

[15:48](https://www.youtube.com/watch?v=PbHm2qKnu10&t=948s) You can just install it, but it's been a long time coming. Um, and what it really is is a toolkit of these pieces to make it so that building an agent that you can actually train with RL feels just like building an agent.

[16:01](https://www.youtube.com/watch?v=PbHm2qKnu10&t=961s) Um, so the interaction protocol here is, like, quite simple. Like, this is the entire rollout function on the left of, like, what happens in the code when you're running an agent to do RL, which is that you kind of set up some initial state stuff, have a while loop for is it done yet.

[16:15](https://www.youtube.com/watch?v=PbHm2qKnu10&t=975s) If it's not done, do a turn, and the thing you're passing here is a client object that's just an OpenAI compatible API. And I think this is the kind of interface that you really want if you want people to be able to go from their agent applications to something that's trainable, something that they can use with RL.

[16:32](https://www.youtube.com/watch?v=PbHm2qKnu10&t=992s) Um, it's been a lot of fun thinking about, like, what are the abstractions, what are the pieces here. And so, like, there's things like parsers and rubrics that I think are, like, nice building blocks that you sometimes wanna use.

[16:41](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1001s) You can also, like, not use them if you don't want to, but, like, I've tried to make it fun and user-friendly. Um, the other day I, like, was like, "Let's train a Wordle agent."

[16:48](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1008s) I think this was, like, a fun little toy problem where it's like it's not that hard of, like, a game for us as humans, but, like, it's actually, like, kind of tricky to get your code to be this sort of thing where you have this, like, multi-turn interaction protocol that you actually can do learning with.

[17:04](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1024s) Um, but now it's, like, much easier. Like, the code to do these things is, like, quite simple, and the reward functions can kind of be relatively simple for this sort of setup where it's like, okay, you wanna reward it for, like, uh, solving the thing eventually, but also, like, give it more rewards for doing it in less

[17:18](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1038s) turns. And, like, this is a 7B model. It works reasonably well. But one of the reasons it works, um, which I'll talk about in a sec, is, uh, SFT warm-up as a way of kind of lowering the barrier of entry.

[17:29](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1049s) Like, this-- the code as it is is very much set up so that, like, your environments for RL are also just, like, synthetic data loops or evals where you can plug in Claude or DeepSeek or OpenAI and, like, test.

[17:40](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1060s) So you don't have to, like, do RL to debug. You can, like, debug with an API in terms of seeing is this a good eval, is this a good reward.

[17:47](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1067s) Once you're kind of comfortable with it, you can, like, use whatever API you like that you are allowed to use and make synthetic data, do some SFT on it, and now you can start doing RL and this, like, helps a lot with small models.

[17:58](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1078s) Um, I think there's a lot of efficiency challenges that are, like, I've, I've been kind of hard at work trying to solve in terms of, like, having all of your computation be utilized effectively, having everything be, like, fully async so you don't have to worry about, like, batching, um, and that your trainer and your inference can kind

[18:13](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1093s) of go at the same time. You can be, like, a little bit off policy. Um, a lot of engineering that I'm hoping, like, if you wanna worry about that, great, dig into it, fork the repo, mess with things.

[18:23](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1103s) If you don't want to, you shouldn't have to. Um, and, like, the idea here is that this should become something that more people are trying out, more people are having fun with, with exploring and getting a feel for it.

[18:37](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1117s) Um, because if it's going to be a thing we have to worry about, if this is the future of building better agent models, uh, for your applications, like, now's a good time to start.

[18:46](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1126s) Um, and so this stuff is set up so you can, like, on a couple GPUs, like, uh, do a lot of interesting research. Like, the barrier of entry is, like, much lower now than it used to be.

[18:56](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1136s) Um, I have a lot of fun doing this on, like, a couple GPUs. Uh, we sell GPUs, by the way. Um, [audience laughing] thanks everybody. Uh, I don't think we have time for questions, but, uh, yeah. [audience applauding] [upbeat music]

[19:16](https://www.youtube.com/watch?v=PbHm2qKnu10&t=1156s) All right. Thank you
