Training Agentic Reasoners

Talk transcript

1. 0:00

   [upbeat music] Hi, everyone.

2. 0:15

   I'm Will Brown. I'm at Prime Intellect. Uh, today I wanna talk about training agentic reasoners. Um, just kind of as a very high level overview, I think a lot of people here are really excited about reasoning, and a lot of people here are really excited about agents, but I feel like a lot of the conversations between these

3. 0:30

   two topics are kind of, like, different, where people are like, "Oh, reasoning is this one thing, and agents are this other thing." And the considerations of, like, reasoning are very different from the considerations of building agents.

4. 0:39

   And I think the high level thesis of this talk is like, no, they're kind of the same thing. Um, and you'll see why as we get into it. Um, first, just to start, like, RL kind of works now.

5. 0:49

   Um, I think for a long time people were like, "Oh, is RL gonna work? Is it not gonna work? How hard is it gonna be?" Um, and like DeepSeek, I think, took a lot of people by surprise for many reasons, like the costs or whatever, and like how good it is compared to the open models to the,

6. 1:02

   the closed la- the big labs, as well as just it being fully open. Um, but I think it was also just that it was RL applied at scale working with surprisingly few tweaks needed, where you just, like, have a good setup, you have a good signal, you have a model that is good enough to do some learning,

7. 1:19

   and you see this curve where doing more RL results in the model getting better. Um,

8. 1:25

   and it's also kind of how everyone else is doing it. Like, this is what the big labs are really banking on to drive the next iterations of progress. The o3 release is the one that OpenAI is really excited about, not GPT-4.5.

9. 1:37

   Like, they stopped serving the big pre-trained model via API, but they have continued to really double down on the scaling direction of doing more and more reinforcement learning and spending more compute on reinforcement learning once you have the right setup to enable progress.

10. 1:51

    And o3, to me, is like a very naturally agentic model. It-- the ChatGPT version has all of these tools. The kind of selling point of it is not just that it's smarter, it's that it's really good at using lots of tools in agentic task settings to solve harder problems that involve interacting with complex systems.

11. 2:10

    And that is kind of really the selling point of all of this, is that, like, the more complex your system, the more things can go wrong, the more that, like, a generic LLM API is going to be brittle and go off the rails after a certain number of steps.

12. 2:23

    And RL is kind of the way around it. It's the trick you can do to take the system that kinda works, maybe it works on small scales, but as you go harder, it starts going off the rails, and train the model to be better at that thing.

13. 2:36

    Um, and so this is a recipe that is still kind of like a research topic that people are not fully sure, like, the best way to do it, especially outside of the big labs.

14. 2:45

    But it clearly is moving in a direction where it's becoming more and more reliable, more and more accessible, and the sort of thing that I think would be silly to disregard as a potential, like, key piece of the future of agentic software and agentic applications.

15. 3:01

    Uh, but it's also complicated. So on the left here, this is the, like, architecture diagram of VERRLL, which is kind of the most popular software people use in the research world for writing papers to, uh, do RL.

16. 3:13

    So if you wanna, like, take a model and go do RL, VERRLL kind of expects that you understand all of this. Um, on the left, we ha-- uh, uh, the right we have GRPO a-as presented in the original DeepSeekMath paper back from early 2024.

17. 3:26

    And, like, there's a lot of pieces here. There's a lot of, like, complicated steps going on that I think a lot of people who are used to thinking about APIs, used to thinking about building agents, kind of, like, are hoping they don't have to worry about it, um, and are hoping that, like, you can just set it

18. 3:43

    aside and, like, something else will work, and we'll just use the APIs, and it'll all be great. And I think the reality is, like, somewhere in the middle where, like, I think it doesn't need to be this complicated, but I think you also kind of do have to be aware of it if your goal is really, like,

19. 3:57

    building the most performant agents. Not necessarily just, like, today you need to know about it, but as a piece of the toolkit to potentially make really powerful agentic software, I think the people who are willing to do this and take the best open models and really RL them for their tasks and can figure out how to do

20. 4:13

    that well are going to have a huge advantage. And that's the kind of thing that also allows you to, like, build a moat beyond just, like, being a wrapper API and towards something where it's like, "Oh, I actually have my own model now."

21. 4:24

    But not everyone can be a big lab, and so we kind of need to meet in the middle somewhere of like, okay, how do we make this a thing that starts to become feasible for startups, for individual researchers to actually do?

22. 4:36

    Uh, and, like, at what scale does this become, like, feasible? Um, and so agents are, like, the type of product that everyone's excited about. We all, like, love Claude Code and Devin and Manus and, uh, o3 and Deep Research and, like, these are the sorts of products that are really capturing people's attention.

23. 4:52

    Um, they're products that in their current iteration happen to work kind of because the models that are being used have, like, been RL'd to basically do these kinds of things.

24. 5:01

    Um, like Claude is a very good coding agent probably because it has been RL'd on a lot of code. And so it's, like, not very surprising that if you plug Claude into essentially a while loop with some tools, it's, like, quite good at doing these things because it's basically most likely been trained in almost that exact setting.

25. 5:18

    Um, same for things like o3. Like, it can do GeoGuessr and whatever because whether it's literally GeoGuessr or something close to it, they have talked about training it to do this image cropping trick.

26. 5:28

    Like, that's a, a technique that it didn't just know how to do out of the box. They said, "Hey, let's give it these tools to do that and use reinforcement learning to train it to do that."

27. 5:37

    And so that is kind of the recipe that we have seen coming from the big labs as if you want a powerful agent that can do a certain type of task, you can use reinforcement learning to train it to do that task better.

28. 5:47

    Um, and so these are kind of the same thing actually. Like, building an agent, uh, the pieces of

29. 5:53

    Making an agent in terms of the harness, the environment, the tools, and the iteration is essentially the same conceptual framing as canonical reinforcement learning in the sense of policies, action, states, rewards, transition probabilities.

30. 6:08

    Um, and I think the more that we start to view agents as this umbrella, which is not just about static chaining of API calls, but as this interaction loop with evaluations, that framing really is the way to think about RL, which is you build a system where a thing is interacting with an environment and you have some

31. 6:28

    way of evaluating how good it's doing, and RL is simply an algorithm to improve based on the scores of these evaluations. And if you're building agents and you're tuning your prompts and you're fiddling with your harnesses, this is kind of like doing RL by hand.

32. 6:42

    What you're doing is you're saying like, "Okay, currently my evals are saying this. Let's make sure the evals are like capturing what I want. Let's look at the data.

33. 6:51

    Let's see if the data matches what my evals are saying. And then, oh, let's try a new prompt. Let's try giving it a new tool. Um, let's try, uh, switching out the model."

34. 7:02

    Um, like this is the process which is also being targeted by reinforcement learning in the general sense beyond individual algorithms. Um, about these algorithms, like there's a few of them that are mo- very important.

35. 7:13

    All, all of them have like different implementation details. But in general, the idea is you have a bunch of tasks, like versions of your problem, which are essentially prompts.

36. 7:21

    You have rollouts, which are just completions potentially involving many steps of interactions, but like one sequence of stuff happening. And then you have evaluation potentially interleaved throughout or at the end of the sequence.

37. 7:33

    And what you're estimating is the advantage. The advantage here is the idea that, uh, sometimes your model will be, be better than others. Like these, uh, LLMs are all, uh, non-deterministic.

38. 7:44

    You have temperature above zero. You have different things happen in different, uh, rolls of the dice. Um, and this, uh, forking process of saying like, "Okay, this time it did better than that time.

39. 7:55

    Why was it different?" RL is really about saying like, "Okay, uh, this is the actual thing that changed that resulted in the reward being better, the eval being better.

40. 8:07

    This is the token at which I went down the good path versus the bad path." Um, and whether you're doing PPO or GRPO, um, like this is the mechanism by which you get the signal of like you have something that sometimes went better, sometimes went worse.

41. 8:22

    Now you can kind of, uh, very surgically have the model learn to do more of the good stuff without changing too much overall. I think this is also kind of maybe a reason why DPO, I think people were hoping DPO would like really work well.

42. 8:35

    In my view, DPO does not necessarily have this like fine-grained advantage estimate. Like it's not really clear just from like a full good completion and a full bad completion where you're really getting the signal about these complex branching processes.

43. 8:47

    Uh, PPO has this, but it's also very expensive. GRPO, I think, has taken a lot of people kind of, uh, by storm in terms of like being a very s- nice like middle ground where it's more computationally efficient, it's like simple to implement, um, but also it does have this kind of forking process that comes just from

44. 9:04

    sampling. Um, there's also just too many papers. So like I think a lot of people just see a new paper every day and are like, "Oh, do I have to read this one?"

45. 9:13

    Um, and I feel that too. Like I think it's difficult to know upfront like which of these are going to be important, which of them are just gonna be like noise, especially because lots of them have very sensationalist titles like, "Oh, Qwen doesn't work," um, or like, or Qwen everyone-- "Everything only works with Qwen," is like kind

46. 9:30

    of true, but like there's also more to the story than that, and I think there's like different implementation details of like, oh, if you change the loss function like this in this experiment, then it works.

47. 9:40

    And I think for most people, it is best to just like kind of set this aside and to not get too caught up in the individual details of individual experiments and individual papers and kind of think more holistically about what is the process of reinforcement learning doing, um, what implementation details am I willing to kind of leave

48. 9:59

    to other people to figure out and eventually come to me with like software that like has the, the knob set correctly, um, and which pieces are actually important for solving the problems I care about.

49. 10:08

    Um, and so for a lot of people, I think the things that are going to be really interesting, um, are things that are relating to actual software, to actual problems that they want to solve in the world.

50. 10:18

    And agents, I think, are kind of the instantiation of that where this makes sense. And the thing that makes an agent an agent is tools, uh, the ability to interact with an environment, with a system.

51. 10:28

    A lot of people here are like very excited about MCP at the conference. Like MCP is just tools. MCP is about giving your LLM the ability to like interact with stuff, to go solve problems that involve changing files, making requests, uh, editing code, running code.

52. 10:45

    Um, and so I think these are the papers that I get excited about because they feel like, like there's parts of the puzzle that are not fully solved yet of like what's the right way to do all of this.

53. 10:52

    Like there's still some open questions. Um, but I think those are getting kind of refined. We're starting to see more and more, but like a lot of the code, the tools we have out in the wild to like go do this, like if you wanna like go play around with RL, most code bases are like very set

54. 11:09

    up for like either code and math tasks or things that are quite similar to that. This is kind of my fault. Um, I had a, a snippet go viral that was like, "Here's how you do RL on like GSM8K," which is like a kind of easy math dataset.

55. 11:23

    Um, and then I think I've seen a lot of people like stick with this as like, "Oh, we're gonna RL on math." And I-- like this is also just like math is easy to evaluate.

56. 11:31

    Um, and I think people are-- like evals, writing evals are hard. Like there's a whole track going on in parallel to this about like how to build a good eval.

57. 11:38

    Um, and so I think a lot of researchers gravitate towards things that like look like the benchmarks that are also really easy to eval because there's like a very clear signal of like, "Okay, this thing is like right, this thing is wrong.

58. 11:49

    Good. Okay, we're doing RL." Um, but like real world tasks are messier than that. Um- We are not going to, like, get great software systems just by, like, hill climbing on whatever question-answer benchmark is popular today.

59. 12:03

    Um, what we're gonna do is-- we're gonna have to do is start thinking about, like, the actual systems at hand and the challenges that emerge when we're trying to design these rewards.

60. 12:12

    And so, like, reward hacking is, like, a real thing. Um, I think this is one of the lessons that, like, RL works, but also it's not, like, always gonna work.

61. 12:20

    There are things that can go wrong. And to me, reward hacking is really a message about the difficulty of building good evals. Like, uh, what you really want with an eval is for it to be easier for your model to do the task than to hack the eval.

62. 12:35

    You want to build a reward signal that actually captures what you care about, where, uh, gaming it is, like, more difficult than not gaming it. If you can-- If the model can learn to do the task directly just by doing what you want it to do, uh, in the spirit of the task, then, like, that is what

63. 12:54

    will happen. It will flow in the dir-- path of least resistance. This is-- Like, models just want to learn, but they want to learn to do better on reward signals.

64. 13:00

    And so your reward signals have to point in the direction of the thing you actually care about. Um, otherwise, like, models will find cheats. Um, and I think thinking about these things in combination kind of points a little bit towards a direction that I think is gonna be very promising, and there's some very early signs that, like,

65. 13:16

    this actually can work, um, which is like, uh, uh-- When R1 came out, I was kind of, like, speculating, like, what's next? What are the things that are going to unlock this sort of technique being used more generally?

66. 13:30

    Um, and you-- people talk a lot about, like, generator verifier gaps, like what are the differences between, like, solving a problem versus checking if you have a solution. And a lot of problems, like, are much easier to check than solve, but this isn't, like, a binary thing.

67. 13:42

    This is a spectrum of how difficult is it to verify a thing. But, um, there's some kind of signs that you kind of s- can do evaluations on more ambiguous tasks by just breaking them down into smaller pieces and by using LLMs as subroutines in your evaluations, like LLMs that judge on steroids.

68. 14:03

    Or maybe you wanna actually, like, train a specialized LLM who is really good at doing these fine-grained evaluations. I like using the term rubric as a conceptual general umbrella around reward models, reward functions, LLMs judge setups, like the criteria on which you are evaluating a thing.

69. 14:18

    There's a cool paper from DeepSeek that I was-- thought-- found very exciting when it came out a couple months ago about, like, how to train reward models that, like, generate these rubrics on the fly.

70. 14:26

    There was a paper very recently that does this for creative writing and kind of found that, like, yes, you actually can train reward models that will come up with nuanced, fine-grained evaluation criteria for a task on the fly given the actual problem, and this gives you something that results in a very, like, fine-grained score that allows you

71. 14:44

    to actually do RL and, like, keep getting better. Um, and I think, like, this is an area that I'm really excited about to keep watching. Um, but also, like, multi-turn.

72. 14:54

    Multi-turn is probably where we're headed. We wanna do agentic search, we wanna do tool calls, software, games, long-horizon planning, computer use memory. Scaling on tool calls lets you solve harder problems.

73. 15:05

    Um, and so how do we actually, like, do this? What's the, uh, way to go about building multi-agent or multi-turn agentic systems to do and that we can use RL with?

74. 15:14

    Um, and I think the conceptual pieces here are environments are basically harnesses, rewards are basically evals, tasks are just prompts, and your policy in the RL sense hopefully should just be as simple as, like, an LLM API.

75. 15:27

    The-- I think the programming interface that makes sense for a lot of people is to have an API that you're writing code as if it's just a normal agent in a loop, but then this is a thing that you can use to go do RL.

76. 15:38

    And so that's what I've been building over the past couple of months. Um, I maintain a repo called Verifiers. Um, it's finally, uh, on pip, uh, out in the world.

77. 15:48

    You can just install it, but it's been a long time coming. Um, and what it really is is a toolkit of these pieces to make it so that building an agent that you can actually train with RL feels just like building an agent.

78. 16:01

    Um, so the interaction protocol here is, like, quite simple. Like, this is the entire rollout function on the left of, like, what happens in the code when you're running an agent to do RL, which is that you kind of set up some initial state stuff, have a while loop for is it done yet.

79. 16:15

    If it's not done, do a turn, and the thing you're passing here is a client object that's just an OpenAI compatible API. And I think this is the kind of interface that you really want if you want people to be able to go from their agent applications to something that's trainable, something that they can use with RL.

80. 16:32

    Um, it's been a lot of fun thinking about, like, what are the abstractions, what are the pieces here. And so, like, there's things like parsers and rubrics that I think are, like, nice building blocks that you sometimes wanna use.

81. 16:41

    You can also, like, not use them if you don't want to, but, like, I've tried to make it fun and user-friendly. Um, the other day I, like, was like, "Let's train a Wordle agent."

82. 16:48

    I think this was, like, a fun little toy problem where it's like it's not that hard of, like, a game for us as humans, but, like, it's actually, like, kind of tricky to get your code to be this sort of thing where you have this, like, multi-turn interaction protocol that you actually can do learning with.

83. 17:04

    Um, but now it's, like, much easier. Like, the code to do these things is, like, quite simple, and the reward functions can kind of be relatively simple for this sort of setup where it's like, okay, you wanna reward it for, like, uh, solving the thing eventually, but also, like, give it more rewards for doing it in less

84. 17:18

    turns. And, like, this is a 7B model. It works reasonably well. But one of the reasons it works, um, which I'll talk about in a sec, is, uh, SFT warm-up as a way of kind of lowering the barrier of entry.

85. 17:29

    Like, this-- the code as it is is very much set up so that, like, your environments for RL are also just, like, synthetic data loops or evals where you can plug in Claude or DeepSeek or OpenAI and, like, test.

86. 17:40

    So you don't have to, like, do RL to debug. You can, like, debug with an API in terms of seeing is this a good eval, is this a good reward.

87. 17:47

    Once you're kind of comfortable with it, you can, like, use whatever API you like that you are allowed to use and make synthetic data, do some SFT on it, and now you can start doing RL and this, like, helps a lot with small models.

88. 17:58

    Um, I think there's a lot of efficiency challenges that are, like, I've, I've been kind of hard at work trying to solve in terms of, like, having all of your computation be utilized effectively, having everything be, like, fully async so you don't have to worry about, like, batching, um, and that your trainer and your inference can kind

89. 18:13

    of go at the same time. You can be, like, a little bit off policy. Um, a lot of engineering that I'm hoping, like, if you wanna worry about that, great, dig into it, fork the repo, mess with things.

90. 18:23

    If you don't want to, you shouldn't have to. Um, and, like, the idea here is that this should become something that more people are trying out, more people are having fun with, with exploring and getting a feel for it.

91. 18:37

    Um, because if it's going to be a thing we have to worry about, if this is the future of building better agent models, uh, for your applications, like, now's a good time to start.

92. 18:46

    Um, and so this stuff is set up so you can, like, on a couple GPUs, like, uh, do a lot of interesting research. Like, the barrier of entry is, like, much lower now than it used to be.

93. 18:56

    Um, I have a lot of fun doing this on, like, a couple GPUs. Uh, we sell GPUs, by the way. Um, [audience laughing] thanks everybody. Uh, I don't think we have time for questions, but, uh, yeah. [audience applauding] [upbeat music]

94. 19:16

    All right. Thank you