What RL Means for Agents

Talk transcript

1. 0:00

   [on hold music] Hello, everyone.

2. 0:17

   Uh, thanks, Rix and the whole AI engineer conference team for putting this together and having me. I am Will Brown. I am a machine learning researcher at Morgan Stanley.

3. 0:25

   And today, I wanna talk to you all a bit about what I think reinforcement learning or RL means for agents. So I was in grad school at Columbia for a while, and I mostly worked on theory for multi-agent reinforcement learning.

4. 0:38

   And over the past couple years, I have been working at Morgan Stanley on a wide range of LM-related projects, some of which look kind of like agents, but I will not really be talking too much about that today.

5. 0:48

   Uh, I'm also relatively active on X, the everything app, and that will become relevant later in the talk. This talk, I think, will be probably a little different from most of the talks at the conference.

6. 0:57

   Um, it's not about things we ship to prod. It's not about things that definitely work and you should go do tomorrow that are like proven science or best practices.

7. 1:07

   It's about where we might be headed, and I want to really just tell a story that will synthesize some things that have been happening in the broader research community, um, and, uh, where these trends might be pointing, do some speculation, and also talk about some, uh, recent open source work of my own.

8. 1:23

   Um, and the goal of this is to help you plan and understand what reinforcement learning means, what it means for agents, and how to best be ready for a potential future which may involve reinforcement learning as part of the agent engineering loop.

9. 1:40

   So, um, where are we today? Most LLMs that we work with are essentially chatbots. I think it's helpful to think about OpenAI's, uh, five levels framework here. So we did pretty well with chatbots.

10. 1:51

    Seems like we're doing pretty well with reasoners. Um, these are great models for question answer. They're very helpful for interactive problem-solving. We have the o1, o3, R1, Grok 3, Gemini, et cetera, models that are really good at kind of thinking longer.

11. 2:04

    Um, and we're trying to figure out how we take all of this and make agents level three. Um, and these are systems that are taking actions. These are systems that are doing things that are longer and harder and more complex.

12. 2:15

    And currently, the way we tend to do this is chaining together multiple calls to these underlying chatbot or reasoner LLMs. And we do lots of things like prompt engineering, tool calling, evals, ops, giving the models tools of their own to use, having humans in the loop, and the results are, like, pretty good.

13. 2:32

    Um, there's a lot of things that we can do, and then there's a lot of stuff that it feels like is around the corner that we're all imagining about AGI, but we're not really to the point yet where these things are going off and doing the things that we would imagine an AGI is really doing to the

14. 2:48

    degree of autonomy that that would, I presume, entail. So I think it's useful a bit to distinguish between agents and pipelines. I think Barry's talk earlier was a good way to kinda frame this.

15. 2:59

    I'm gonna use pipelines to encapsulate what Barry called workflows. Um, and I think these are really systems with fairly low degrees of autonomy, and there's a very nontrivial amou- nontrivial amount of engineering required to determine these decision trees to say, how does one action or call flow into another, how, uh, to another, how do we refine the

16. 3:17

    prompts? Um, and it seems like a lot of the winning apps in the agent space have very tight feedback loops. And so whether or not you wanna call these agents or pipelines, these are things where a user is interacting with some sort of interface.

17. 3:30

    They're telling it what to do. The thing will do some stuff and come back relatively quickly. Things like the IDEs, like Cursor, Windsurf, and Replit, um, and search tools that are really good at harder question answer, maybe with some web search or research integrated.

18. 3:42

    But there's not that many agents nowadays that will go off and, like, do stuff for more than ten minutes at a time. I think Devin, Operator, and OpenAI's Deep Research are the three that really come to mind as, like, feeling a little more in the, like, autonomous agent direction.

19. 3:56

    And I think a lot of us might be wondering, how do we make more of these? And the kind of traditional wisdom is like, okay, we'll just wait for better models.

20. 4:03

    Once better models are around, we can just, like, use those. We'll be good. But I think it's also to kind of take note of, like, the traditional definition of reinforcement learning and what an agent means there, which is this idea of a thing that is interacting with an environment with a goal, and the goal that-- and the

21. 4:19

    system is designed to learn how to get better at that goal over time via repeated interaction, uh, with the system. And I think this is something that a lot of us are either doing manually or don't really have the tools to do, which is once we have our thing that it's set up to make the calls we

22. 4:35

    want and the performance is like seventy percent and we've done a lot of prompt tuning, we want to get it up to ninety percent, we just, like, don't have the models to do it or the models struggle to get the success, what's our path forward?

23. 4:46

    Um, and so in terms of model trends, I think I won't spend too much time talking about this, but, uh, pre-training seems to be having diminishing returns to capital, at least.

24. 4:54

    We're still seeing kind of like loss go down, but, uh, it does kinda feel like we need new tricks. Um, reinforcement learning from human feedback is great for making kind of friendly chatbots, um, but it doesn't really seem to, uh, be continually pushing us at the frontier of smarter and smarter and smarter models.

25. 5:12

    Uh, we talk a lot about synthetic data, and I think synthetic data is great for distilling, uh, larger models down into smaller models to have kind of really tiny models that are really performant.

26. 5:20

    But on its own, it doesn't seem to be an unlock for, like, massive capabilities, uh, getting better and better unless we throw in verification in the loop or rejection sampling or any of these things.

27. 5:31

    And that really takes us to the world of reinforcement learning, where this seems to be the trick that unlocked test-time scaling for o1 models and R1. Um, it's not bottlenecked by needing manually curated human data, and it does seem to actually work.

28. 5:44

    Um, I think we all kind of took note about a month ago when DeepSeek released the R1 model and paper to the world, and I think this was really exciting because it was the first paper that really explained how you build a thing like o1.

29. 5:58

    We'd had kind of speculation and some rumors, but they really laid out the algorithm and the mechanisms for what it takes to get a model to learn to do this kind of reasoning.

30. 6:09

    And it turns out it was essentially just reinforcement learning where you give the model some questions, you measure if it's getting the answer right, and you just kind of turn this crank of giving it feedback to do more like the things that worked well and less like the things that didn't work.

31. 6:23

    Um, and what you end up seeing is that the, the long chain of thought for models like o1 and R1 actually emerges as a byproduct of this. It wasn't kind of manually programmed in where the models were, like, given data of, like, ten thousand token reasoning steps.

32. 6:36

    This was a thing the model learned to do because it was a good strategy. And reinforcement learning at the core is really about identifying good strategies for solving problems.

33. 6:44

    Um, it also seems like open source models are, are back in a big way. There's a lot of excitement around the open source community. Um, people have been working on replication efforts for the o1 project, um, and have also been trying to distill data from o1 down, down into smaller models.

34. 6:57

    And so what next? How does this relate to agents? Um, I think it'll be helpful to know a little bit about how reinforcement learning works. The key idea is to explore and exploit.

35. 7:05

    So you wanna try stuff, see what works, do more of the things that worked, less of the things that didn't. And so in this feedback loop, um, demonstrated here in the image, we can see a cha- a challenger model's, uh, supposed to be writing code to pass test cases.

36. 7:18

    And we give it rewards that correspond to things like formatting, using the right language, and then ultimately whether or not the test cases are passing. And so this is kind of a numerical signal that rather than, like, training on data, uh, where we are kind of curating this in advance, we are letting the model do synthetic data

37. 7:33

    rollouts and seeing scores from these rollouts, which then f- are fed back into the model. And so the GRPO algorithm, which maybe some of you have heard about, is the algorithm DeepSeek used.

38. 7:42

    I think it's less of, like, a technical breakthrough in terms of it being a really important new algorithm to study, but I think it's very conceptually simple, and I think it's a nice way to think about what reinforcement learning means.

39. 7:51

    And the idea really is just that you, for a given prompt, sample end completions, you score them all, and you tell the model be more like the ones with higher scores.

40. 8:00

    Um, this is still in kind of the single turn reason or model non-agentic world. Uh, and so the challenges that lie ahead, um, are gonna be about how do we take these ideas, uh, and extend them into, uh, more powerful, more agentic, more autonomous systems.

41. 8:16

    But we do know that it can be done. So OpenAI's Deep Research still has a lot of questions that we do not know the answers to about how it works, but they have told us that it was end-to-end reinforcement learning.

42. 8:25

    And so this is a case where the model is taking up to potentially a hundred different tool calls of browsing or querying different parts of the internet to synthesize a large answer.

43. 8:34

    And it does seem, I think, to many people's vibe check opinions, very impressive. Um, but it also is, like, not AGI in the sense of you can't get it to go, like, uh, work in a repo or, like, solve hard software engineering tasks.

44. 8:47

    And people have kind of anecdotally found that it does struggle a bit for, like, out-of-distribution tasks. So, like, if you want it to fill out a table with, like, a hundred very manual calculations, it can struggle there.

45. 8:57

    And so it seems like reinforcement learning on one hand is a big unlock for new skills and more autonomy, but it's not a thing that so far has granted us agents that can just do everything and know how to solve all kinds of problems.

46. 9:10

    But it is a path forward for teaching a model skills and having the model learn how to get better at certain skills, particularly in conjunction with environments and tools and verification.

47. 9:22

    Um, and so there is infrastructure out there for doing this on our own, kind of. Um, a lot of it is still RLHF style, by which I mean it's about kind of single turn interactions where the goal is we have reward signals that come from kind of human data that has been combined into a reward model.

48. 9:40

    Um, and if we wanna have RL agents becoming part of our systems, maybe we will get really good API services from the large labs that let us build these things and hook into GPT whatever, um, or Claude whatever, and train these sorts of models on our own with fine-tuning.

49. 9:56

    But we also don't really have these options yet. Um, OpenAI has kind of teased their reinforcement fine-tuning feedback, but it's not, uh, multi-step tool call- tool calling yet. And so I think if we wanna plan ahead, it's worth kind of noting and asking, what would this ecosystem look like?

50. 10:11

    And there's a lot of unknown questions like how much this will cost, how small can the models be, will it generalize across tasks, uh, and how do we design good rewards and good environments?

51. 10:20

    And there's a lot of opportunity here. Um, open source, uh, infrastructure, there's a lot of room to build and grow and determine what the best practices are going to be, what the right tools will be, as well as companies that can build tools for-- to support this ecosystem, uh, whether or not they're already in the fine-tuning world

52. 10:34

    or not, um, and services for supporting this kind of agentic RL. And I think also it is worth thinking about things that are, like, not literal RL in the sense of training the model, but at the prompt level, there's all sorts of automation we can do.

53. 10:46

    So if you've used DSPy, I think that is kind of adjacent to RL in the flavor of having a signal that we can then, uh, bootstrap from to improve our, uh, underlying system based on improving some downstream scores.

54. 11:00

    Um, now I wanna share a story with you about a single Python file I wrote a couple weeks ago. Um, so this was the weekend after R1 came out, and I'd been reading the paper and thought it was really cool.

55. 11:10

    We had not had the NVIDIA stock crash quite yet. Um, and, uh, I was just playing around with some experiments. I was taking the-- a hu- a trainer from Hugging Face that had the GRPO algorithm, and I was getting a really small language model, Llama 1B, to do some reasoning and then give an answer for math questions.

56. 11:29

    And I started with, like, a pretty simple system prompt, and I was just training the model to let it see what it did, and I had kind of manually curated some rewards in terms of what the scoring function should look like.

57. 11:39

    And I just kinda, like, tweeted it out, um, where I had an example of the model kinda looking like it's doing some self-correction and so- showing that the accuracy gets better as well as the, uh, length of response will initially drop once it learns to kinda follow the format.

58. 11:55

    Then it goes back up as it learns to kind of take advantage of longer chains of thought to do its reasoning. And this was not the first thing to replicate in any sense.

59. 12:04

    I wouldn't really call it a true replication. Um, it was far from the most complicated, and I think that actually caught a lot of people's imaginations, and it became kind of a thing.

60. 12:14

    Um, so over the next two weeks after that, it Just took on a life of its own where a lot of people were kind of tweeting about it and forking it and making modifications to it, and making it something you could run in a Jupyter Notebook, making it more accessible, writing blog posts about it.

61. 12:29

    And it was interesting, um, because it, to me, didn't feel like a thing that kind of merited this level of excitement. But what I think was catching people's imagination was that it was one file of code.

62. 12:44

    It was really simple, and it invited, uh, modification in a very user-friendly, engaging way, which I like to call rubric engineering. And so the idea of rubric engineering here is that similar to prompt engineering, um, to, uh, have a model do reinforcement learning, it's gonna get some reward, but what should this reward be?

63. 13:04

    In the most simple version, it's just like, did it get the question right or wrong? Like, does A equal B? But there's a lot more you can do beyond this.

64. 13:10

    And so I think the, the single file of code exposed, uh, examples of this where you can give the model points for things like following this XML structure. Like, if it gets a certain tag right, you give it plus one point.

65. 13:22

    Um, if it has an integer answer, that's still the wrong answer, but it's learned that the format should be an integer answer, get some points for that. Um, and there's a lot of room here for getting creative and for designing rules that are not just downstream evals to, for our own sake, know whether a thing is working,

66. 13:38

    but to allow the model itself to know whether it's working and use that as feedback for going further and training more. Um, and this is very early stages. There's a lot of things we don't know, and I think there's a lot of opportunity to get creative and explore and try things out, such as using LLMs to design

67. 13:53

    these rubrics, uh, auto-tuning these rubrics or auto-tuning your prompts with frameworks like DSPy, um, incorporating LLM judges as part of the scoring system. And then also, I think reward hacking is an issue to be very con- cautious of, where the idea is you want to ensure that the, the reward model you're using is actually capturing the goal,

68. 14:12

    and it doesn't have kind of these backdoors where a model can kind of cheat and do something else that ultimately results in it kind of getting a super high reward without learning to do the actual task.

69. 14:23

    Um, and following this, I have been trying to learn from those lessons of what I saw people using out in the wild and make something that is a little more, uh, robust and, uh, usable for actual projects beyond just one file of code.

70. 14:37

    Um, and it-- this has been a kind of very recent effort. It's not a thing that I'm telling you to go use for all your problems tomorrow, but I think it's my attempt at doing some open source, uh, research code, um, that will help people potentially try these things out easier and answer some questions, uh, about this.

71. 14:52

    And so what this really is is it's a, uh, a framework for doing RL inside of multi-step environments. So the idea here is that lots of us have built these great agent frameworks for using API models, and the hope would be that we can leverage those existing environments and, uh, frameworks to, uh, ha- actually do RL.

72. 15:11

    So here the idea is you can just create this environment thing that the model plugs into, and you don't have to worry about the weights or the tokens. You can just write an interaction protocol, and then this gets fed into a trainer.

73. 15:22

    And so once you build this environment, you can just kind of let it run and, uh, have a model that once you give it some rewards, learns to get better and better over time.

74. 15:31

    Um, and to conclude, I want to talk about what I think AI engineering might look like in the RL era. So

75. 15:39

    this is all still something that is very new. Uh, we don't know whether the off-the-shelf API models are going to just work for the tasks we throw at them.

76. 15:48

    It might be the case that they do. It might be the case that they don't. Um,

77. 15:52

    one reason I think that they might not be the entire solution is that it is really hard to include a skill in a prompt. You can include knowledge in a prompt.

78. 16:02

    Um, but a lot of us when we try something, we don't nail it the first time, and it takes a little bit of trial and error. Um, and it seems to be the case that models are like this as well, where a model does get better at a thing and really gets a skill nailed down by trial

79. 16:19

    and error. And this has been the most promising unlock we've seen so far for these higher autonomy agents like deep research. Um, fine-tuning might still be important. I think a lot of people wrote off fine-tuning for a while because open models were far enough behind the frontier that like a prompted o- uh, frontier model API was just

80. 16:38

    gonna beat a smaller fine-tuned model. I think, one, we're now seeing the open closed source gap be close enough that this is less of a concern. A lot of people are using open source hosted models in their platforms.

81. 16:49

    Um, and also, uh, RL, the most kind of true version of RL that DeepSeek did for their R1 model that OpenAI has talked about doing for, uh, deep research requires doing some reinforcement learning.

82. 17:03

    Um, there's a lot of challenges here. There's a lot of research questions we don't know the answers to. Um, but there's a lot of things that I think these skills we've learned from doing AI engineering over the past couple years translate very directly to, which is that the challenge of building environments and rubrics is not that different

83. 17:18

    from the challenge of building evals and prompts. We still need good monitoring tools. We still need a large ecosystem of companies and platforms and products that support the kinds of agents we want to build.

84. 17:28

    Um, so I think all the stuff we've been doing is gonna be essential, and it's worth looking ahead a little bit to see if we end up in a world where we have to do a little bit more reinforcement learning to unlock things like true autonomous agents or innovators or organizations that are powered by language models.

85. 17:48

    Um, what does that look like? Uh, we will find out. [outro music]