RL for Autonomous Coding — Aakanksha Chowdhery, Reflection AI

Talk transcript

1. 0:00

   [on-hold music] Hi, everyone.

2. 0:15

   I'm Aakanksha. I was at Google for more than six years, and I led the research for PaLM, and I was a lead researcher in Gemini. Uh, these days, I'm working on, uh, pushing the frontier for autonomous coding, uh, with reinforcement learning.

3. 0:32

   So just to recap the arc of how we have progressed in large language models and, um, why autonomous coding and why now. Um,

4. 0:43

   so I think everyone here, or those of you, uh, who don't remember, in twenty twenty there was this breakthrough paper that came out which talked about scaling laws for large language models.

5. 0:53

   And if you were to take a thirty-second recap, all-- the main thing it said was that there's a power law relationship between the test laws of large language models.

6. 1:03

   Um, so if you use more compute, more data, and put more parameters in your machine learning model, uh, which is a transformer model, you will get more performant models, and it will not be performant just in the domain in which you are training the model.

7. 1:19

   It will actually be performant, and it will generalize to many other domains. And the generalization was, uh, pretty much a feature, uh, in this particular case. So as the, uh, large language models got bigger, we saw continuous improvement across benchmarks to the point that they're starting to get saturated now.

8. 1:38

   And the other interesting thing was that we saw emergent behavior, uh, where capabilities were emerging in large language models that were not present in smaller models. And this is a classic slide that I show for, for the work that we did, uh, in PaLM.

9. 1:54

   So typically, when you go about trying to solve math problems, and you give the model some examples, on the left you have a math problem around tennis balls, and then you give a second problem.

10. 2:06

    The model output looks wrong. But what PaLM and the subsequent set of, uh, papers showed was that if you ask the model to output its, uh, reasoning chains, uh, which has become a very common concept now, but this is, remember, twenty twenty-one, so four years ago.

11. 2:23

    Um, if you ask the model to show its reasoning chains, then the answer actually is correct. So basically, by getting the model to, uh, output its chain of thought or reasoning chains, the, the model performance improves.

12. 2:36

    And, uh, this capability particularly emerged in large language models. These are all the models. Uh, so LaMDA and PaLM were the state-of-the-art models about three years ago. And what I'm showing on X-axis is the increasing number of parameters.

13. 2:51

    PaLM was scaled all the way up to five forty billion parameters. No one actually publishes the number of parameters these days, so you have to live with the, the graphs from two years ago or the open source stuff that's coming out with DeepSeek and Qwen models.

14. 3:05

    But what Y-axis is showing is that the solve rate on middle school math word problems was increasing, uh, with the number of parameters in the models, and it was essentially increasing mainly when you were prompting the models and asking them to show chain of thought.

15. 3:22

    And this led to all kinds of prompting techniques where you ask the model to think step by step. You even go and bribe the model and such, and you ask the model nicely or not.

16. 3:31

    So this was all kinds of fun stuff. Um, and I think the, the c-- uh, the thing that really stood out from this generation of models three years ago was that this capability was not just limited to mathema-- uh, math problems.

17. 3:44

    It was, uh, basically, uh, generalizing across a whole bunch of domains, anywhere from question answering in other languages to puzzle problems to, um, multitask, uh, natural language understanding problems.

18. 3:59

    And what this led to next was that now that these models could reason, we could get them to follow instructions. So the first set of applications that became possible with these large language models were chatbot applications.

19. 4:14

    So everyone remembers that ChatGPT and, uh, now Gemini and various other chatbots have become extremely popular. All of us use them all the time. But what made them really possible was that when you give instructions to the model to go do something, it's actually able to do it, and the way it learns that is actually based on

20. 4:32

    reinforcement learning. And, uh, reinforcement learning data that we are giving to the model in this particular case is essentially data based on human feedback. So you're basically saying, "Okay, here is a set of questions, and if I were to give it to a human, and it were-- there were two answers, which one would the human prefer?"

21. 4:53

    And if you have enough of this data and you train your model, you would actually end up with, uh, better performance because you've taught the model which set of responses to prefer.

22. 5:04

    And this actually doesn't only work in chatbot applications, it also works in code. So on the bottom right, I'm showing that even if you were to do this for applications in code, you start to see some performance improvements.

23. 5:19

    Now, of course, the question is that last year there was a whole bunch of debate as to, are we hitting the wall in terms of performance of large language models?

24. 5:26

    Uh, pre-training is not giving any gains or all, all of these questions were on the horizon. So what is next? And I, I-- one of the key questions to remember in, in all of this is that when you go and pre-train the models, you end up spending a lot of money, um, on training these models.

25. 5:44

    It could be tens of millions of dollars. And when you in-- do inference on the models, it's extremely cheap. Um, these numbers are not endorsed by any of the companies I worked at, but these are public numbers, uh, from, uh, public sources.

26. 5:59

    So going back to the main point that I wanna make here is that- Training is extremely costly. So if you constantly try to scale up the model size, you end up, uh, in, in this regime of like, um, if, if it's not giving performance gains, then can we get performance gate at-- uh, gains at inference time because

27. 6:17

    inference costs are so cheap? And a key idea that was, uh, extremely useful here was that if you could get the models to generate multiple responses and then do majority voting.

28. 6:32

    So, uh, in the example above, I'm showing that we get-- we-- the, the prompt doesn't make sense, but you've given a mathematical problem to a large language model, and you're asking it to generate three answers independently, and then you basically do some voting on top of those answers.

29. 6:50

    And if two answers match, then that's a majority vote. Or like, if in this room I were to ask a question and all of you said yes, then that is a majority vote.

30. 6:59

    So similarly, in large language models, if you can get the model to, like, generate many, many samples and then consistently get it to vo-- uh, like, get many of those answers to agree, this notion of majority voting or self-consistency had shown gains.

31. 7:12

    So this kind of scaling compute at inference time was clearly one avenue to go push on. Another avenue that emerged and showed substantial value was that you could sequentially revise your previous response.

32. 7:25

    So as humans, oftentimes we write the first answer, and then we go evaluate our answer, and we are like, "Oh, there's some mistake here. It doesn't quite match," and then you go fix it.

33. 7:35

    So basically, can we get LLMs to do the same kind of revision looking at previous set of revisions? And this was the second, so basically having longer, uh, chains of thought, uh, and getting the model to improve consistently in inference time based on that.

34. 7:50

    And these kind of techniques, uh, where you could verify your correct answer, so in math or in programming where you have unit tests, showed, uh, very clear gains. So what I'm showing you here is an example from, uh, uh, one of my colleague's work, um, at Stanford, uh, which is a publicly, uh, published, uh, paper.

35. 8:10

    And on the y-axis, we have pass@k or coverage score, and on the x-axis, we have, uh, number of samples. So as you basically are doing a lot of samples on the x-axis, your accuracy is improving, uh, with open source DeepSeek model and just taking more samples.

36. 8:25

    So you're getting a very high score on SWE-bench Verified compared to even state-of-the-art back in end of twenty twenty-four. Uh, of course, now all of these scores have pushed up, and we are roughly somewhere around eighty percent already.

37. 8:40

    But what we want to take away here is the fact that these lines of work, they showed that inference time compute predictably gives us gains, especially in domains where we can verify.

38. 8:54

    If we know how to verify the answers, then we actually know how to translate that into intelligence. And going back to my talk title, coding is one of those domains where we do have the capability to verify.

39. 9:07

    Um, and that gives us tremendous advantage in terms of building super intelligence on top of autonomous coding.

40. 9:16

    Of course, now you ask the question of what does automated verification mean here? So

41. 9:23

    for inference time scaling to work, you need basically some way to say this output is correct. Now in math, um, this is a very simple example. If you were to give the input to solve this mathematical equation, um, and if you were to do the same calculation on a calculator, you can actually verify that that problem is

42. 9:41

    correct or that solution is correct. Um, and similarly in math, you have formal proofs, so you can actually verify things are correct. Uh, in coding, you have unit tests.

43. 9:53

    In compilers, you can actually generate the code and then use PyTorch as a verifier-- uh, PyTorch, the compiler as a verifier. And in fact, uh, in domains where you don't have, uh, this kind of verification, then there's a large gap.

44. 10:07

    If you were to generate a lot of solutions and then do majority voting, you actually don't get as much gains. So what this roughly meant was that, okay, so inference time scaling would work in scenarios where I have automated verification, but that doesn't quite solve the problem for it to be-- have real world impact.

45. 10:26

    And the reason for that is shown in this graph as to typically, uh, if you do majority voting and these, uh, this is across multiple different models on GSM8K, which is middle school math problems and another math benchmark.

46. 10:38

    Uh, if you were to sort them by correct fraction, you have to sample a lot. The correct generations could be very rare. So who has time to sample ten thousand times and then get a correct solution?

47. 10:51

    You would be sitting there waiting just finding the correct solution unless you can actually figure out where the correct generation is.

48. 10:59

    So basically, scaling inference time compute with just majority voting or longer reasoning chains is great in the sense that there are some correct solutions somewhere there, but it doesn't work well across the board.

49. 11:11

    So what will get these models to learn to generate correctly, uh, during training? Well, in, in the chatbot application scenario, we saw that RL with human feedback did work.

50. 11:24

    So can we apply the same principle here and get the model to generate correctly in, uh, where, where we can automatically verify the outputs? So our belief at, uh, Reflection is that the next frontier for scaling is reinforcement learning, and we already have proof points from some of the frontier labs as well.

51. 11:43

    And as David Silver and, um, Sutton published recently, they agree with the-- or, or rather they, they are the pioneers in, uh, in reinforcement learning. They say that we are basically entering the era of experience, like starting from AlphaGo and AlphaZero, uh, where you had an era of simulation, and the next set of large language model era

52. 12:05

    was where you scaled up with RL using human data. But the next era from This year is really the year of experience, which was-- which will lead us to super intelligence.

53. 12:15

    So reinforcement learning will be a fundamental component in building, uh, super intelligent systems, uh, especially in areas where we have automated, uh, verification. And some, uh, proof point for why this makes sense is that in math, uh, over several papers, this is, uh, results from '01, but over several papers, we have already seen examples that if you

54. 12:39

    give the model on the right side, uh, test time compute, uh, on the y-axis, test time compute the same as inference time scaling, and you measure accuracy on the x-axis, it should go up.

55. 12:50

    Um, but as you can repeat this process, um, and with reinforcement learning, then the training time compute going up on x-axis also improves the accuracy on y-axis for a challenging benchmark in math called AIME.

56. 13:04

    Uh, most of these benchmarks saturate within a year, as you probably have learned by now. So, uh, this benchmark is already saturated. Um,

57. 13:14

    so now that I've hopefully convinced you that reinforcement learning and scaling reinforcement learning is, uh, the next frontier, you'd be like, "Okay, so why are-- why is not everyone doing it?

58. 13:25

    What's so challenging about it?" So as I have built large language models before, a big part of building, uh, these systems, uh, ends up being that the machine learning plus system stack for these, uh, systems themselves is very challenging.

59. 13:39

    So here is, um, an example of, uh, why scaling up reinforcement learning is challenging. So if you are trying to do reinforcement learning with, uh, PPO, which is, uh, one of the a-algorithms used for RL with human feedback, um, then it moved to, uh, DPO.

60. 13:56

    You have to keep four copies of, uh, different models. Uh, so if you imagine a really large model, and then you have to keep four copies, then you have to arrange them somewhere on GPUs in your large cluster.

61. 14:08

    You, you can have some fun figuring out the exact layout and, um, it's, it's, it's a fun and interesting problem, but it's a hard problem in the sense that, uh, to make maximum utilization of these systems and, and arranging them in the right way, just building that system is extremely hard.

62. 14:25

    And, uh, DeepSeek actually showed, uh, with DeepSeekMath, that GRPO, uh, gets rid of the value model and, and only has three copies of the model, but that doesn't-- that's still a very challenging problem.

63. 14:36

    So scaling up RL, uh, is more cha-- even more challenging, um, than scaling up, um, LLMs because you have multiple copies of the model and you have a training loop and an inference loop.

64. 14:48

    And then on the machine learning side, on the, on the reinforcement learning side, you also suffer a lot from reward hacking if your, uh, the model that is deciding that this is the correct answer is a neural reward model.

65. 15:00

    So you, uh, as we discussed before, in autonomous coding applications, you do have the ability to verify your output, uh, which roughly means that you can decide this is the correct answer or not.

66. 15:12

    Uh, that's how SWE-bench Verified scores, uh, work today. Um, you have execution feedback, you have unit tests. So all of these possibilities, of course, um, this is, uh, an ongoing list, all of these possibilities me-mean that you can design better reward functions.

67. 15:29

    Okay, so this means that autonomous coding is a great domain for scaling up RL. Then the question becomes: How does this have real-world impact? So in software engineering applications, generation of code is only one part of the system.

68. 15:44

    If you look at end-to-end workflows for software engineering, there is many more parts to that system. How do you scale up your system to generalize across all of those domains?

69. 15:53

    So that's the problem we are trying to solve at Reflection.

70. 15:57

    Our mission is that we would like to build some super intelligence, and we are starting with autonomous coding as the root node problem for this, um,

71. 16:06

    uh, mission. And, uh, we have a team of about thirty-five pioneers, um, who are-- who have pioneered various, uh, legendary works in LLMs and reinforcement learning. So if you're excited about this mission, uh, you can reach out to, um, one of us, um, or my, uh, my email is, uh, my last name at reflection[REDACTED:email_address], and we would

72. 16:27

    love to work with you. And with that, I can take questions. [clapping]

73. 16:37

    All right, um, same protocol as last time. If you have a question, please come up to one of these three microphones we have distributed throughout. We can probably take one or two questions.

74. 16:46

    So if you want to ask something, um, feel free. Um, I guess I can-- I, I'll do the first one while people are coming up. So I'm curious, um,

75. 16:55

    seems like the foundation models are trying to build one model and deploy it across everything. Do you have an opinion with the work you're doing right now if you think that's the right approach, or if you think there'll be more specialization on different languages or even, like, individual code bases?

76. 17:09

    Um, or do you feel like the best approach is just to have, like, one model that's trained across the, the greatest diversity of tasks possible?

77. 17:16

    Uh, I think I will answer your question, uh, in terms of building coding agents does require m-um, multiple capabilities, and how you get there, you will definitely need multiple LLM calls.

78. 17:26

    And then whether that's one model or multiple models, I think that's the secret sauce right now for most people.

79. 17:31

    Fair enough. All right. Please.

80. 17:34

    Hi. Um, I'm wondering in the slide with the chart of era of simulation, era of something, and era of experience-

81. 17:42

    Yeah

82. 17:42

    ... uh, they had put in AlphaGo and, um, the, uh, previous one where also you-- they played Star-StarCraft or something. They all used MCDS, uh, which, I mean, maybe it's my unfamiliarity with them, but it's also data sim-uh, simulation.

83. 18:00

    Mm-hmm.

84. 18:00

    Uh, so we're using synthetic data for era of experience as well. So how does-- why is that called simulation, and why is what we're doing right now not called simulation?

85. 18:11

    What's the sort of overlap between simulation, experience? How does that-- how do you think about that?

86. 18:17

    I can ask Dave that question, you know. But [laughs] going back to the point, I think, I think the better way to answer that question is, uh, roughly what Greg covered in the last talk, where his comment was that, um, so in gaming, you can envision what scenarios might happen next, and you're basically using that to build your

87. 18:35

    reinforcement learning. So you're doing rollouts, and you're, you're basically building, uh, based on that. Um, in, in real world, in most scenarios, you have an imperfect rollout, so you don't have full knowledge of how the system might work.

88. 18:50

    Um, simulation is possible in certain domains where you do build a world model, uh, which is closer to robotics and all the work that's happening in the physical AI space, right?

89. 19:03

    But in the, in the real-world applications, which is what we're targeting, um, you will have imperfect things. So you have to actually experience the real world, and you have to collect some data, and that data is not gonna be in any way complete, nor will it complete early search, the exponential search space that could exist. [upbeat music]