AI Engineer Summit 2025
What RL Means for Agents
Read the talk
What Reinforcement Learning Means for Agents
Agents need a way to improve beyond prompt tuning. Reinforcement learning connects their attempts to verifiable feedback, turning environments and reward rubrics into engineering tools.
From a talk by Will Brown
Before you start: Familiarity with language-model prompts, tool calls, and basic Python is helpful; no prior reinforcement-learning knowledge is required.
How does an agent get better at acting?
How do we turn models that answer difficult questions into agents that reliably carry out difficult tasks? That question requires looking beyond the next model release to how systems learn from experience. Will Brown approaches it through his background in multi-agent reinforcement learning at Columbia and language-model research at Morgan Stanley. His February 2025 talk is a research outlook, drawing on emerging results and personal open-source experiments rather than a production deployment or established best practices.
The starting point is a progression from chatbots to reasoners to agents. In OpenAI’s five-level framing, chatbots converse; reasoners tackle harder questions; agents take actions. Models such as o1, o3, DeepSeek-R1, Grok 3, and Gemini make longer deliberation useful for question answering and interactive problem-solving. Turning that capability into autonomous action usually requires chaining model calls together, with prompts, tool calling, evaluations, operational infrastructure, and human oversight.
Those systems can accomplish useful work without having much autonomy. Brown calls systems with substantially engineered decision trees and control flow pipelines: developers decide how calls connect, what happens after each result, and how prompts should change. Many successful products also keep users close to the loop. Cursor, Windsurf, Replit, and research-oriented search interfaces do some work and return quickly for more direction.
Devin, Operator, and OpenAI’s deep research stand out in Brown’s account as systems that work for longer before returning to the user. They raise a practical question: how can developers build more systems with that degree of autonomy? Waiting for a stronger underlying model is one answer, but it leaves the improvement process outside the application developer’s control.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An improvement loop beyond prompt tuning
In reinforcement learning, an agent interacts with an environment to pursue a goal, then learns from repeated attempts. That last part changes the engineering loop: an interaction produces both an outcome and evidence for improving future behavior. Brown’s hypothetical is an agent at 70% success after substantial prompt tuning, with a target of 90%. These figures describe an engineering problem, not a measured result: once prompt changes stop helping, what makes the next attempt more likely to succeed?
Brown places that question against several model-development trends. His assessment is that pretraining still reduces loss but offers diminishing returns on capital, while other techniques solve different parts of the capability problem.
| Approach | What it contributes | Limitation in Brown’s account |
|---|---|---|
| Pretraining | Broad underlying capabilities | Increasing investment may buy less improvement |
| RL from human feedback | Friendly, useful chatbot behavior | Does not by itself continually advance reasoning |
| Synthetic data | Distillation into smaller models | Generation alone needs a way to select good outputs |
| RL with verifiable feedback | Learning from successful attempts | Depends on useful feedback about outcomes |
Verification and rejection sampling matter because they distinguish useful generated attempts from unsuccessful ones. Brown connects this shift toward outcome feedback to test-time scaling in reasoning models: training can teach a model to use more computation effectively at inference, without requiring humans to curate every successful reasoning trace.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning as a strategy learned from outcomes
The DeepSeek-R1 paper made this approach unusually concrete. Rather than merely showing that a model could reason for longer, it described mechanisms for training that behavior. Brown reduces the core feedback loop to three actions: ask a question, check the answer, and update the model toward attempts that worked.
Longer reasoning can emerge because it is a successful strategy. The learner need not receive a prescribed long reasoning trace for every question. This point applies most directly to R1-Zero, which omitted preliminary supervised fine-tuning; the released R1 also used cold-start reasoning examples and subsequent supervised and RL stages. The distinction preserves the useful mechanism without treating outcome-only training as the complete R1 recipe. Brown’s broader point is that reinforcement learning discovers strategies for solving problems, and longer deliberation can be one of them.
That opens several routes for the open-model community. Researchers can try to reproduce the training process, while others distill reasoning outputs from stronger models into smaller ones. Replication and distillation are different activities, but both make experimentation with reasoning behavior more accessible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From candidate outputs to training feedback
The basic RL rhythm is exploration and exploitation: try alternatives, observe which work, and favor successful behavior. Brown illustrates it with a model writing code to pass tests. A reward can recognize several properties of the candidate: whether it follows the required format, uses the correct programming language, and ultimately passes the test cases. The model generates these attempts—its rollouts—and their numerical scores feed back into training.
Group Relative Policy Optimization, or GRPO, gives a compact way to understand that loop:
- Give the model a prompt.
- Sample
Ncandidate completions for that prompt. - Score every completion using the reward functions.
- Update the model to favor the higher-scoring completions.
This is Brown’s conceptual explanation, not the complete optimization objective. Its value is that it makes the training signal tangible: multiple attempts at the same task give the learner a basis for preferring some behavior over other behavior.
So far, the unit of interaction is still a single completion. An autonomous agent must instead act, receive information from its environment, and decide what to do next. Extending the feedback loop across those multiple steps is the next challenge.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extended tool use does not imply universal competence
Deep research provides a concrete connection between RL and extended action. Brown points to OpenAI’s disclosure that it trained the system with end-to-end reinforcement learning for browsing and reasoning. Brown describes deep research as potentially making up to 100 browsing or querying tool calls to synthesize an answer; that is his account, not a verified tool-call limit. The important change is the scope of the behavior being learned: a sequence of information-gathering actions contributes to a final answer.
An impressive research result does not establish general competence. Brown distinguishes browsing and synthesis from working in a repository or solving difficult software-engineering tasks. He also cites anecdotal difficulty with out-of-distribution requests, such as filling a table through many manual calculations. These observations are capability qualifications, not a controlled comparison. RL offers a way to teach particular skills through environments, tools, and verification; it does not establish that a trained agent will generalize to every other kind of work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What an agent-training ecosystem needs
The infrastructure Brown describes is still largely organized around RLHF: single-turn interactions scored by reward models derived from human feedback. Developers might eventually connect their own environments to large-lab training services, but at the time of the talk he describes reinforcement fine-tuning access as limited and not yet offering the multi-step tool-calling workflow he wants. That is a historical assessment of the available workflow, not a statement about current trainer or API capabilities.
Several questions determine whether this becomes practical:
- Cost: What does repeated interaction and training cost?
- Model size: How small can a model be while still learning the required behavior?
- Generalization: Does training on one task improve performance on others?
- Rewards and environments: How should developers define success and expose opportunities to learn?
These questions create room for open-source infrastructure, training services, and supporting products. The ecosystem needs ways to turn an application’s interaction loop into something a trainer can use.
Not every useful feedback loop needs to update model weights. Brown places DSPy nearby: downstream scores can guide improvements to prompts and the surrounding system. This is adjacent to RL in its use of feedback to improve behavior, while remaining distinct from reinforcement learning that trains the underlying model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small experiment that invited modification
The weekend after R1’s release, Brown tried the idea in a single Python file. He used Hugging Face’s GRPO trainer with a Llama 1B model, math questions, a simple system prompt, and manually designed rewards. The goal was to let the model reason and then produce an answer. The linked gist has since changed its default to Qwen2.5-1.5B-Instruct, while retaining Llama-3.2-1B-Instruct as a commented option; the experiment described here used Llama.
He shared an output on X that appeared to show self-correction. Brown reports improving accuracy and response length that first fell as the model learned the format, then rose as longer reasoning became useful. He supplies no numerical accuracy result or held-out evaluation protocol in the talk, and the example’s apparent self-correction does not by itself establish a particular reasoning process. Brown explicitly declines to call the experiment a true R1 replication.
Over the following weeks, people forked the script, modified it, adapted it into notebooks, and wrote explanations. Brown attributes its appeal to its small size: a single file made the experiment easy to understand and change. That accessibility exposed a particularly approachable part of RL engineering—the rules that decide which outputs deserve credit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Designing the feedback the model learns from
Brown calls this rubric engineering. The simplest reward asks whether the model’s answer equals the expected answer. But a rubric can also recognize intermediate properties: a correctly placed XML tag, the required response structure, or an integer answer even when that integer is wrong. His example of giving a point for a tag illustrates partial credit; it is not a specification of the script’s reward constants.
A small Python rubric makes those distinctions explicit. For the question What is 2 + 3?, an answer of 4 can satisfy the structure and integer checks without satisfying correctness:
python
import re
import xml.etree.ElementTree as ET
def rubric(response: str, expected: int) -> dict[str, bool]:
checks = {
"xml_structure": False,
"integer_answer": False,
"correct_answer": False,
}
try:
root = ET.fromstring(f"<response>{response}</response>")
except ET.ParseError:
return checks
checks["xml_structure"] = (
[child.tag for child in root] == ["reasoning", "answer"]
and all(len(child) == 0 for child in root)
)
answer = (root.findtext("answer") or "").strip()
checks["integer_answer"] = re.fullmatch(r"-?\d+", answer) is not None
if checks["integer_answer"]:
checks["correct_answer"] = int(answer) == expected
return checks
response = "<reasoning>I add the two values.</reasoning><answer>4</answer>"
checks = rubric(response, expected=5)
reward = sum(checks.values())
Here each satisfied check contributes one unit of reward. Changing the answer to 5 preserves the formatting credit and adds correctness credit. This example isolates the mechanism: a format reward and a task-success reward communicate different things.
The same rules that help a developer evaluate a system can become feedback the model trains against. Brown proposes several directions for experimentation: LLMs could help design rubrics, rubrics could be tuned automatically, DSPy could optimize prompts, and LLM judges could contribute scores where simple checks are insufficient.
That flexibility also creates reward hacking risk. A model may discover a way to maximize the score without accomplishing the intended task. In the example above, producing well-formed XML and any integer earns credit without solving the arithmetic; those checks therefore cannot substitute for correctness. More elaborate rubrics need the same scrutiny: does a high score actually correspond to the goal, or is there a loophole the learner can exploit?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An environment interface for training agents
The community’s experiments led Brown toward a more reusable framework, shown in the talk as Verifiers. He introduces it as early research code for exploring multi-step RL, not as a recommendation to adopt it immediately for production workloads. Its purpose is to carry the accessibility of the one-file experiment into environments with richer interactions.
The proposed separation of concerns is straightforward. Developers have already built environments and agent frameworks around API models. An environment author should be able to express the interaction protocol—how the model acts and how the environment responds—without managing model weights or token-level training details. The environment then connects to a trainer.
Once that connection exists, repeated runs produce trajectories and rewards that can support learning. The intended improvement loop is no longer just a prompt followed by a scored answer: it encompasses the agent’s sequence of interactions. The environment defines what can happen, the rubric defines what counts as success, and the trainer uses the resulting feedback to update the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From supplying knowledge to teaching skills
Whether off-the-shelf API models will eventually handle all these tasks remains open. Brown’s reason for taking task-specific training seriously is that supplying knowledge in a prompt is easier than supplying a skill. Instructions can describe what to do, but reliable execution may require trial and error. People often need repeated attempts to acquire a skill; models can also improve through attempts and feedback. Deep research motivates his view that this process can support greater autonomy.
That keeps fine-tuning relevant. When open models lagged far behind frontier APIs, a prompted frontier model could make training a smaller model seem unnecessary. Brown sees a narrowing gap, along with wider use of hosted open models, as weakening that assumption. The RL approaches behind reasoning and extended tool use also require a training process: arranging better prompts around a fixed model does not perform that update.
The engineering work already invested in agents still transfers. Designing environments extends the work of building tool interfaces and interaction flows; designing rubrics extends the work of building evaluations and prompts. Monitoring remains necessary, as do the platforms, services, and products that support the surrounding system. If more autonomous agents—and eventually systems capable of innovation or organizational work—depend on learning through interaction, AI engineering will need to make those interactions measurable and trainable. Whether that future arrives in the form Brown imagines remains uncertain; the concrete work is building environments where improvement can happen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original account of R1-Zero, R1's multistage training, GRPO and reasoning-model distillation.
Describes reinforcement learning for multi-step browsing and reasoning, with launch-era examples and limitations.
Small-model math training script with GSM8K questions and separate correctness, integer-format and XML-format rewards.
Updates since the talk
Current library for building LLM training and evaluation environments, originally created by Will Brown.
Current trainer documentation covering reward functions, training metrics and multi-turn tool use.
Read the complete timestamped transcript
- 0:00
[on hold music] Hello, everyone.
- 0:17
Uh, thanks, Rix and the whole AI engineer conference team for putting this together and having me. I am Will Brown. I am a machine learning researcher at Morgan Stanley.
- 0:25
And today, I wanna talk to you all a bit about what I think reinforcement learning or RL means for agents. So I was in grad school at Columbia for a while, and I mostly worked on theory for multi-agent reinforcement learning.
- 0:38
And over the past couple years, I have been working at Morgan Stanley on a wide range of LM-related projects, some of which look kind of like agents, but I will not really be talking too much about that today.
- 0:48
Uh, I'm also relatively active on X, the everything app, and that will become relevant later in the talk. This talk, I think, will be probably a little different from most of the talks at the conference.
- 0:57
Um, it's not about things we ship to prod. It's not about things that definitely work and you should go do tomorrow that are like proven science or best practices.
- 1:07
It's about where we might be headed, and I want to really just tell a story that will synthesize some things that have been happening in the broader research community, um, and, uh, where these trends might be pointing, do some speculation, and also talk about some, uh, recent open source work of my own.
- 1:23
Um, and the goal of this is to help you plan and understand what reinforcement learning means, what it means for agents, and how to best be ready for a potential future which may involve reinforcement learning as part of the agent engineering loop.
- 1:40
So, um, where are we today? Most LLMs that we work with are essentially chatbots. I think it's helpful to think about OpenAI's, uh, five levels framework here. So we did pretty well with chatbots.
- 1:51
Seems like we're doing pretty well with reasoners. Um, these are great models for question answer. They're very helpful for interactive problem-solving. We have the o1, o3, R1, Grok 3, Gemini, et cetera, models that are really good at kind of thinking longer.
- 2:04
Um, and we're trying to figure out how we take all of this and make agents level three. Um, and these are systems that are taking actions. These are systems that are doing things that are longer and harder and more complex.
- 2:15
And currently, the way we tend to do this is chaining together multiple calls to these underlying chatbot or reasoner LLMs. And we do lots of things like prompt engineering, tool calling, evals, ops, giving the models tools of their own to use, having humans in the loop, and the results are, like, pretty good.
- 2:32
Um, there's a lot of things that we can do, and then there's a lot of stuff that it feels like is around the corner that we're all imagining about AGI, but we're not really to the point yet where these things are going off and doing the things that we would imagine an AGI is really doing to the
- 2:48
degree of autonomy that that would, I presume, entail. So I think it's useful a bit to distinguish between agents and pipelines. I think Barry's talk earlier was a good way to kinda frame this.
- 2:59
I'm gonna use pipelines to encapsulate what Barry called workflows. Um, and I think these are really systems with fairly low degrees of autonomy, and there's a very nontrivial amou- nontrivial amount of engineering required to determine these decision trees to say, how does one action or call flow into another, how, uh, to another, how do we refine the
- 3:17
prompts? Um, and it seems like a lot of the winning apps in the agent space have very tight feedback loops. And so whether or not you wanna call these agents or pipelines, these are things where a user is interacting with some sort of interface.
- 3:30
They're telling it what to do. The thing will do some stuff and come back relatively quickly. Things like the IDEs, like Cursor, Windsurf, and Replit, um, and search tools that are really good at harder question answer, maybe with some web search or research integrated.
- 3:42
But there's not that many agents nowadays that will go off and, like, do stuff for more than ten minutes at a time. I think Devin, Operator, and OpenAI's Deep Research are the three that really come to mind as, like, feeling a little more in the, like, autonomous agent direction.
- 3:56
And I think a lot of us might be wondering, how do we make more of these? And the kind of traditional wisdom is like, okay, we'll just wait for better models.
- 4:03
Once better models are around, we can just, like, use those. We'll be good. But I think it's also to kind of take note of, like, the traditional definition of reinforcement learning and what an agent means there, which is this idea of a thing that is interacting with an environment with a goal, and the goal that-- and the
- 4:19
system is designed to learn how to get better at that goal over time via repeated interaction, uh, with the system. And I think this is something that a lot of us are either doing manually or don't really have the tools to do, which is once we have our thing that it's set up to make the calls we
- 4:35
want and the performance is like seventy percent and we've done a lot of prompt tuning, we want to get it up to ninety percent, we just, like, don't have the models to do it or the models struggle to get the success, what's our path forward?
- 4:46
Um, and so in terms of model trends, I think I won't spend too much time talking about this, but, uh, pre-training seems to be having diminishing returns to capital, at least.
- 4:54
We're still seeing kind of like loss go down, but, uh, it does kinda feel like we need new tricks. Um, reinforcement learning from human feedback is great for making kind of friendly chatbots, um, but it doesn't really seem to, uh, be continually pushing us at the frontier of smarter and smarter and smarter models.
- 5:12
Uh, we talk a lot about synthetic data, and I think synthetic data is great for distilling, uh, larger models down into smaller models to have kind of really tiny models that are really performant.
- 5:20
But on its own, it doesn't seem to be an unlock for, like, massive capabilities, uh, getting better and better unless we throw in verification in the loop or rejection sampling or any of these things.
- 5:31
And that really takes us to the world of reinforcement learning, where this seems to be the trick that unlocked test-time scaling for o1 models and R1. Um, it's not bottlenecked by needing manually curated human data, and it does seem to actually work.
- 5:44
Um, I think we all kind of took note about a month ago when DeepSeek released the R1 model and paper to the world, and I think this was really exciting because it was the first paper that really explained how you build a thing like o1.
- 5:58
We'd had kind of speculation and some rumors, but they really laid out the algorithm and the mechanisms for what it takes to get a model to learn to do this kind of reasoning.
- 6:09
And it turns out it was essentially just reinforcement learning where you give the model some questions, you measure if it's getting the answer right, and you just kind of turn this crank of giving it feedback to do more like the things that worked well and less like the things that didn't work.
- 6:23
Um, and what you end up seeing is that the, the long chain of thought for models like o1 and R1 actually emerges as a byproduct of this. It wasn't kind of manually programmed in where the models were, like, given data of, like, ten thousand token reasoning steps.
- 6:36
This was a thing the model learned to do because it was a good strategy. And reinforcement learning at the core is really about identifying good strategies for solving problems.
- 6:44
Um, it also seems like open source models are, are back in a big way. There's a lot of excitement around the open source community. Um, people have been working on replication efforts for the o1 project, um, and have also been trying to distill data from o1 down, down into smaller models.
- 6:57
And so what next? How does this relate to agents? Um, I think it'll be helpful to know a little bit about how reinforcement learning works. The key idea is to explore and exploit.
- 7:05
So you wanna try stuff, see what works, do more of the things that worked, less of the things that didn't. And so in this feedback loop, um, demonstrated here in the image, we can see a cha- a challenger model's, uh, supposed to be writing code to pass test cases.
- 7:18
And we give it rewards that correspond to things like formatting, using the right language, and then ultimately whether or not the test cases are passing. And so this is kind of a numerical signal that rather than, like, training on data, uh, where we are kind of curating this in advance, we are letting the model do synthetic data
- 7:33
rollouts and seeing scores from these rollouts, which then f- are fed back into the model. And so the GRPO algorithm, which maybe some of you have heard about, is the algorithm DeepSeek used.
- 7:42
I think it's less of, like, a technical breakthrough in terms of it being a really important new algorithm to study, but I think it's very conceptually simple, and I think it's a nice way to think about what reinforcement learning means.
- 7:51
And the idea really is just that you, for a given prompt, sample end completions, you score them all, and you tell the model be more like the ones with higher scores.
- 8:00
Um, this is still in kind of the single turn reason or model non-agentic world. Uh, and so the challenges that lie ahead, um, are gonna be about how do we take these ideas, uh, and extend them into, uh, more powerful, more agentic, more autonomous systems.
- 8:16
But we do know that it can be done. So OpenAI's Deep Research still has a lot of questions that we do not know the answers to about how it works, but they have told us that it was end-to-end reinforcement learning.
- 8:25
And so this is a case where the model is taking up to potentially a hundred different tool calls of browsing or querying different parts of the internet to synthesize a large answer.
- 8:34
And it does seem, I think, to many people's vibe check opinions, very impressive. Um, but it also is, like, not AGI in the sense of you can't get it to go, like, uh, work in a repo or, like, solve hard software engineering tasks.
- 8:47
And people have kind of anecdotally found that it does struggle a bit for, like, out-of-distribution tasks. So, like, if you want it to fill out a table with, like, a hundred very manual calculations, it can struggle there.
- 8:57
And so it seems like reinforcement learning on one hand is a big unlock for new skills and more autonomy, but it's not a thing that so far has granted us agents that can just do everything and know how to solve all kinds of problems.
- 9:10
But it is a path forward for teaching a model skills and having the model learn how to get better at certain skills, particularly in conjunction with environments and tools and verification.
- 9:22
Um, and so there is infrastructure out there for doing this on our own, kind of. Um, a lot of it is still RLHF style, by which I mean it's about kind of single turn interactions where the goal is we have reward signals that come from kind of human data that has been combined into a reward model.
- 9:40
Um, and if we wanna have RL agents becoming part of our systems, maybe we will get really good API services from the large labs that let us build these things and hook into GPT whatever, um, or Claude whatever, and train these sorts of models on our own with fine-tuning.
- 9:56
But we also don't really have these options yet. Um, OpenAI has kind of teased their reinforcement fine-tuning feedback, but it's not, uh, multi-step tool call- tool calling yet. And so I think if we wanna plan ahead, it's worth kind of noting and asking, what would this ecosystem look like?
- 10:11
And there's a lot of unknown questions like how much this will cost, how small can the models be, will it generalize across tasks, uh, and how do we design good rewards and good environments?
- 10:20
And there's a lot of opportunity here. Um, open source, uh, infrastructure, there's a lot of room to build and grow and determine what the best practices are going to be, what the right tools will be, as well as companies that can build tools for-- to support this ecosystem, uh, whether or not they're already in the fine-tuning world
- 10:34
or not, um, and services for supporting this kind of agentic RL. And I think also it is worth thinking about things that are, like, not literal RL in the sense of training the model, but at the prompt level, there's all sorts of automation we can do.
- 10:46
So if you've used DSPy, I think that is kind of adjacent to RL in the flavor of having a signal that we can then, uh, bootstrap from to improve our, uh, underlying system based on improving some downstream scores.
- 11:00
Um, now I wanna share a story with you about a single Python file I wrote a couple weeks ago. Um, so this was the weekend after R1 came out, and I'd been reading the paper and thought it was really cool.
- 11:10
We had not had the NVIDIA stock crash quite yet. Um, and, uh, I was just playing around with some experiments. I was taking the-- a hu- a trainer from Hugging Face that had the GRPO algorithm, and I was getting a really small language model, Llama 1B, to do some reasoning and then give an answer for math questions.
- 11:29
And I started with, like, a pretty simple system prompt, and I was just training the model to let it see what it did, and I had kind of manually curated some rewards in terms of what the scoring function should look like.
- 11:39
And I just kinda, like, tweeted it out, um, where I had an example of the model kinda looking like it's doing some self-correction and so- showing that the accuracy gets better as well as the, uh, length of response will initially drop once it learns to kinda follow the format.
- 11:55
Then it goes back up as it learns to kind of take advantage of longer chains of thought to do its reasoning. And this was not the first thing to replicate in any sense.
- 12:04
I wouldn't really call it a true replication. Um, it was far from the most complicated, and I think that actually caught a lot of people's imaginations, and it became kind of a thing.
- 12:14
Um, so over the next two weeks after that, it Just took on a life of its own where a lot of people were kind of tweeting about it and forking it and making modifications to it, and making it something you could run in a Jupyter Notebook, making it more accessible, writing blog posts about it.
- 12:29
And it was interesting, um, because it, to me, didn't feel like a thing that kind of merited this level of excitement. But what I think was catching people's imagination was that it was one file of code.
- 12:44
It was really simple, and it invited, uh, modification in a very user-friendly, engaging way, which I like to call rubric engineering. And so the idea of rubric engineering here is that similar to prompt engineering, um, to, uh, have a model do reinforcement learning, it's gonna get some reward, but what should this reward be?
- 13:04
In the most simple version, it's just like, did it get the question right or wrong? Like, does A equal B? But there's a lot more you can do beyond this.
- 13:10
And so I think the, the single file of code exposed, uh, examples of this where you can give the model points for things like following this XML structure. Like, if it gets a certain tag right, you give it plus one point.
- 13:22
Um, if it has an integer answer, that's still the wrong answer, but it's learned that the format should be an integer answer, get some points for that. Um, and there's a lot of room here for getting creative and for designing rules that are not just downstream evals to, for our own sake, know whether a thing is working,
- 13:38
but to allow the model itself to know whether it's working and use that as feedback for going further and training more. Um, and this is very early stages. There's a lot of things we don't know, and I think there's a lot of opportunity to get creative and explore and try things out, such as using LLMs to design
- 13:53
these rubrics, uh, auto-tuning these rubrics or auto-tuning your prompts with frameworks like DSPy, um, incorporating LLM judges as part of the scoring system. And then also, I think reward hacking is an issue to be very con- cautious of, where the idea is you want to ensure that the, the reward model you're using is actually capturing the goal,
- 14:12
and it doesn't have kind of these backdoors where a model can kind of cheat and do something else that ultimately results in it kind of getting a super high reward without learning to do the actual task.
- 14:23
Um, and following this, I have been trying to learn from those lessons of what I saw people using out in the wild and make something that is a little more, uh, robust and, uh, usable for actual projects beyond just one file of code.
- 14:37
Um, and it-- this has been a kind of very recent effort. It's not a thing that I'm telling you to go use for all your problems tomorrow, but I think it's my attempt at doing some open source, uh, research code, um, that will help people potentially try these things out easier and answer some questions, uh, about this.
- 14:52
And so what this really is is it's a, uh, a framework for doing RL inside of multi-step environments. So the idea here is that lots of us have built these great agent frameworks for using API models, and the hope would be that we can leverage those existing environments and, uh, frameworks to, uh, ha- actually do RL.
- 15:11
So here the idea is you can just create this environment thing that the model plugs into, and you don't have to worry about the weights or the tokens. You can just write an interaction protocol, and then this gets fed into a trainer.
- 15:22
And so once you build this environment, you can just kind of let it run and, uh, have a model that once you give it some rewards, learns to get better and better over time.
- 15:31
Um, and to conclude, I want to talk about what I think AI engineering might look like in the RL era. So
- 15:39
this is all still something that is very new. Uh, we don't know whether the off-the-shelf API models are going to just work for the tasks we throw at them.
- 15:48
It might be the case that they do. It might be the case that they don't. Um,
- 15:52
one reason I think that they might not be the entire solution is that it is really hard to include a skill in a prompt. You can include knowledge in a prompt.
- 16:02
Um, but a lot of us when we try something, we don't nail it the first time, and it takes a little bit of trial and error. Um, and it seems to be the case that models are like this as well, where a model does get better at a thing and really gets a skill nailed down by trial
- 16:19
and error. And this has been the most promising unlock we've seen so far for these higher autonomy agents like deep research. Um, fine-tuning might still be important. I think a lot of people wrote off fine-tuning for a while because open models were far enough behind the frontier that like a prompted o- uh, frontier model API was just
- 16:38
gonna beat a smaller fine-tuned model. I think, one, we're now seeing the open closed source gap be close enough that this is less of a concern. A lot of people are using open source hosted models in their platforms.
- 16:49
Um, and also, uh, RL, the most kind of true version of RL that DeepSeek did for their R1 model that OpenAI has talked about doing for, uh, deep research requires doing some reinforcement learning.
- 17:03
Um, there's a lot of challenges here. There's a lot of research questions we don't know the answers to. Um, but there's a lot of things that I think these skills we've learned from doing AI engineering over the past couple years translate very directly to, which is that the challenge of building environments and rubrics is not that different
- 17:18
from the challenge of building evals and prompts. We still need good monitoring tools. We still need a large ecosystem of companies and platforms and products that support the kinds of agents we want to build.
- 17:28
Um, so I think all the stuff we've been doing is gonna be essential, and it's worth looking ahead a little bit to see if we end up in a world where we have to do a little bit more reinforcement learning to unlock things like true autonomous agents or innovators or organizations that are powered by language models.
- 17:48
Um, what does that look like? Uh, we will find out. [outro music]