AI Engineer World's Fair 2025
RL for Autonomous Coding — Aakanksha Chowdhery, Reflection AI
Read the talk
Reinforcement Learning for Autonomous Coding
Coding offers something unusually useful for scaling AI: execution feedback. The challenge is turning occasional correct answers into reliable behavior across an entire engineering workflow.
From a talk by Aakanksha Chowdhery
Before you start: Familiarity with language-model prompting, model training, and unit tests will help; no prior reinforcement-learning expertise is required.
From larger models to worked answers
Why would a model get an arithmetic problem wrong when asked for an answer, yet get it right when prompted with worked reasoning? The opening tennis-ball example makes that question concrete: the standard prompt illustrates an answer without intermediate steps; the chain-of-thought prompt illustrates how to work through the calculation.
The path to that example begins with scale. Aakanksha Chowdhery introduces the problem from her experience leading research on PaLM and working as a lead researcher on Gemini, before turning to reinforcement learning for autonomous coding. The 2020 scaling-laws paper established empirical power-law relationships between language-model test loss and model size, training data, and compute. Increasing these resources made model performance more predictable. Chowdhery connects that progression to broader generalization, improving benchmarks, and capabilities that were absent in smaller models.
The arithmetic demonstration shows what one such capability looks like. After an example involving tennis balls, a subsequent problem receives an incorrect answer under standard prompting. With intermediate reasoning demonstrated in the prompt, the illustrated answer is correct. The prompt changes how the model uses its capacity. This is an example of improved performance, not a guarantee that a worked answer will be right. The public chain-of-thought paper and PaLM paper appeared in 2022; Chowdhery’s recollection of 2021 should not be read as their publication date.
The accompanying comparison plots model parameter count against solve rate on middle-school math word problems for LaMDA and PaLM. PaLM reached 540 billion parameters. The notable improvement appears primarily when the larger models receive chain-of-thought prompts, connecting model scale to the usefulness of the prompting technique.
That observation encouraged step-by-step instructions and many informal variations in how people asked models to reason. The broader attraction was that the gains were not confined to arithmetic: Chowdhery describes improvements across multilingual question answering, puzzles, and multitask natural-language understanding.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning which responses people prefer
Reasoning capabilities were followed by increasingly useful instruction-following systems. ChatGPT, Gemini, and other chatbots could respond to requests rather than merely continue text. In Chowdhery’s account, reinforcement learning from human feedback, or RLHF, was a central ingredient in making those interactions useful.
The feedback starts with a comparison: give people a question and two candidate responses, then ask which response they prefer. Accumulating those judgments supplies a training signal that pushes the model toward preferred behavior. The same idea applies to code. The slide pairs the preference-learning loop with a Python task chart showing improvements from RLHF, extending the mechanism beyond conversational answers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend more compute on the answer
As questions emerged about diminishing returns from pretraining, another resource became attractive: compute spent after training, while solving a particular task. Chowdhery cites public estimates putting large-model training costs in the tens of millions of dollars, while describing individual inference calls as comparatively cheap. These are illustrative public estimates, not figures endorsed by her former employers.
That asymmetry suggests generating more than one answer. In the simplest example, the model independently produces three answers to the same mathematical problem. If two agree, their answer wins the majority vote. Across many samples, this becomes self-consistency: sample different reasoning paths and aggregate their final answers. Agreement provides a selection rule, although it does not independently establish correctness.
A second approach spends compute sequentially. The model produces an initial response, evaluates it, identifies a mistake, and revises it using the preceding attempts as context.
| Approach | How compute is spent | How an answer emerges |
|---|---|---|
| Independent sampling | Generate separate attempts | Aggregate agreement |
| Sequential revision | Revisit earlier attempts | Evaluate and repair |
These strategies explore different opportunities: sampling searches across alternatives, while revision tries to improve a particular trajectory. Longer reasoning is useful only insofar as the extra work produces a better answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A correct attempt still needs to be found
Repeated attempts are especially promising when an answer can be checked. Chowdhery presents a coding experiment using an open-source DeepSeek model, with the number of samples on the horizontal axis and pass@k, or coverage, on the vertical axis. More attempts increase the fraction of tasks for which at least one successful solution exists among the candidates.
Coverage across attempts is different from reliably selecting one correct answer. The talk describes the result as SWE-bench Verified and compares it with late-2024 frontier performance, but the benchmark identity of the displayed experiment is not established here. Its curve should therefore be read as evidence for the sampling mechanism, not as a precise Verified leaderboard comparison. Chowdhery separately puts then-current SWE-bench Verified scores at roughly 80%, without specifying a model, scaffold, or sample budget. The transferable point is that verification can make additional inference compute useful: coding supplies ways to distinguish successful attempts from failures.
What counts as verification depends on the task. An arithmetic result can be checked by performing the calculation independently. Formal proofs offer a different kind of check, and code can be evaluated with unit tests. Chowdhery also points to compiler- and PyTorch-based checking for generated code, without developing a particular implementation.
A small Python example makes the distinction between agreement and checking concrete. Suppose the task is to implement square. These candidates are checked against expected results rather than selected by how often their text or outputs resemble one another:
python
from collections.abc import Callable
def square_a(x: int) -> int:
return x * 2
def square_b(x: int) -> int:
return x * x
def verify(candidate: Callable[[int], int]) -> bool:
cases = [(0, 0), (2, 4), (3, 9), (-3, 9)]
return all(candidate(x) == expected for x, expected in cases)
candidates = {"a": square_a, "b": square_b}
accepted = [name for name, candidate in candidates.items() if verify(candidate)]
print(accepted)
The case 2 → 4 alone would accept both implementations; 3 → 9 distinguishes them. Passing this finite test set establishes only that the candidate satisfies these checks. It does not prove correctness for every possible input. This is the practical value—and boundary—of execution feedback.
Without an effective verifier, generating many answers and voting over them yields weaker gains. Even with one, search may be expensive because correct generations can be rare. The math analysis in the talk sorts problems by their fraction of correct generations, using GSM8K and another math benchmark. Some problems sit in a region where a successful sample is difficult to obtain.
Chowdhery’s example of waiting for 10,000 samples expresses that latency problem, not a universal sampling requirement. More attempts may establish that a model can sometimes solve a task, while still leaving the user waiting too long. Majority voting and longer reasoning chains therefore do not, by themselves, make correctness reliable across tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make successful generations more likely
The next step is to move some of the burden from inference into training. If human preferences can teach a chatbot which responses to produce, automatically checked outcomes can teach a coding model which attempts succeed. Reinforcement learning aims to make useful behavior more likely, rather than repeatedly searching for a rare success. Reflection identifies this as its next scaling frontier.
Chowdhery places that direction within David Silver and Richard Sutton’s era-of-experience framing: game systems such as AlphaGo and AlphaZero learned through simulation; language models subsequently drew heavily on human data; future systems would learn more from their own interactions. The forecast that this progression leads to superintelligence is an ambition, not a result demonstrated by the talk.
The supporting example is o1 on AIME mathematics problems. Its results show two distinct opportunities to improve accuracy: spend more compute at test time, and spend more compute on reinforcement learning during training. The paired plots keep those two axes of investment separate.
Chowdhery describes AIME as already saturated at the time of the talk. That is her assessment of benchmark progress, not an unconditional claim that every AIME evaluation setting has been solved. The evidence relevant to the mechanism is the upward relationship between compute and accuracy under the reported conditions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The systems cost of the learning loop
If reinforcement learning offers this opportunity, why is scaling it difficult? A large part of the answer lies in the combined machine-learning and systems stack. A PPO-style RLHF setup involves four functional model roles:
| Role | Responsibility |
|---|---|
| Policy | Generate responses and receive updates |
| Reference | Anchor changes to the policy |
| Reward | Score generated responses |
| Value | Estimate expected reward |
Chowdhery describes these as four copies to place across a GPU cluster. The useful interpretation is four roles, not a universal requirement for four identical physical models. Their placement must sustain utilization while supporting the work each role performs. Direct Preference Optimization, mentioned briefly in the same discussion, is a separate preference-training approach; the four-role description belongs to the PPO-style setup.
DeepSeekMath introduces Group Relative Policy Optimization, or GRPO, which removes the separate learned value model and derives its baseline from rewards across a group of outputs. In the talk’s accounting, that reduces four model roles to three.
Removing a role helps, but it does not remove the fundamental scheduling problem. RL combines an inference loop that generates experience with a training loop that updates the policy. Both require substantial compute, and the overall system has to keep them working efficiently together. Scaling this arrangement introduces difficulties beyond scaling a conventional language-model training run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Better rewards, broader engineering tasks
The learning problem is also vulnerable to reward hacking. A policy can discover responses that score well under a neural reward model without accomplishing the intended task. Coding offers additional signals: execute the output, inspect the feedback, and run unit tests. Those observations can support more useful reward functions than a learned judgment alone.
SWE-bench Verified is the concrete benchmark example. Its tests check candidate fixes against expected behavior; “Verified” refers to human screening of the benchmark tasks, not a certification that arbitrary generated software is correct. Execution feedback improves reward design, but its usefulness still depends on what the checks actually cover.
That makes autonomous coding a promising domain for RL, but code generation covers only part of software engineering. Real-world impact requires a system to generalize across the other parts of an end-to-end workflow. Producing a patch is one capability within that larger job.
Reflection treats autonomous coding as the starting problem for its broader superintelligence mission. At the time of the talk, Chowdhery describes a team of about 35 people with backgrounds in language models and reinforcement learning, and invites people interested in that mission to join. The remaining technical challenge is broader than improving a code-generation score: the learning system must support the range of capabilities that engineering work demands.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Multiple calls do not necessarily mean multiple models
The first audience question asks how those capabilities should be packaged. Should one foundation model serve the widest possible range of tasks, or should coding systems specialize by programming language or even by individual codebase?
Chowdhery answers at the level of the agent: coding requires multiple capabilities and multiple LLM calls. She does not disclose whether those calls use one model or several distinct models. That distinction matters. A workflow with several model invocations does not, by itself, imply an architecture with specialized models; the model-composition choice remains open in her answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Experience in an imperfect environment
The final question challenges the boundary between simulation and experience. The audience member invokes game-playing systems, Monte Carlo tree search, and synthetic data: if both eras involve generating trajectories, what makes the newer one different?
Chowdhery’s answer turns on knowledge of the environment. In games, a system can anticipate subsequent scenarios and build reinforcement learning around rollouts. Real-world rollouts are imperfect: the agent does not have full knowledge of how the surrounding system will behave. World models can enable simulation in domains such as robotics and physical AI, but they do not make all real-world applications fully predictable.
For the applications Reflection is targeting, an agent must interact with the real world and collect data from what happens. That experience will remain incomplete. It will neither describe the environment fully nor exhaust the exponential space of possible situations. Autonomous coding therefore needs more than a mechanism for generating and scoring answers: it needs to learn from partial experience in systems whose behavior it cannot completely simulate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The 540-billion-parameter PaLM model and its evaluations across language, reasoning and coding tasks.
Sampling multiple reasoning paths and selecting answers through agreement.
The construction and evaluation rationale for a human-screened subset of 500 software-engineering tasks.
Historical o1 results showing how reinforcement learning and inference compute affect reasoning performance.
Introduces GRPO and explains how grouped rewards replace a separate learned value model.
A preference-training objective that avoids an explicit reward model and online sampling during fine-tuning.
Further reading
The foundational empirical study relating language-model loss to model size, data and training compute.
Experiments showing how worked reasoning examples improve arithmetic, commonsense and symbolic reasoning.
Repeated-sampling experiments measuring coverage and the limitations of selecting correct answers from many attempts.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hi, everyone.
- 0:15
I'm Aakanksha. I was at Google for more than six years, and I led the research for PaLM, and I was a lead researcher in Gemini. Uh, these days, I'm working on, uh, pushing the frontier for autonomous coding, uh, with reinforcement learning.
- 0:32
So just to recap the arc of how we have progressed in large language models and, um, why autonomous coding and why now. Um,
- 0:43
so I think everyone here, or those of you, uh, who don't remember, in twenty twenty there was this breakthrough paper that came out which talked about scaling laws for large language models.
- 0:53
And if you were to take a thirty-second recap, all-- the main thing it said was that there's a power law relationship between the test laws of large language models.
- 1:03
Um, so if you use more compute, more data, and put more parameters in your machine learning model, uh, which is a transformer model, you will get more performant models, and it will not be performant just in the domain in which you are training the model.
- 1:19
It will actually be performant, and it will generalize to many other domains. And the generalization was, uh, pretty much a feature, uh, in this particular case. So as the, uh, large language models got bigger, we saw continuous improvement across benchmarks to the point that they're starting to get saturated now.
- 1:38
And the other interesting thing was that we saw emergent behavior, uh, where capabilities were emerging in large language models that were not present in smaller models. And this is a classic slide that I show for, for the work that we did, uh, in PaLM.
- 1:54
So typically, when you go about trying to solve math problems, and you give the model some examples, on the left you have a math problem around tennis balls, and then you give a second problem.
- 2:06
The model output looks wrong. But what PaLM and the subsequent set of, uh, papers showed was that if you ask the model to output its, uh, reasoning chains, uh, which has become a very common concept now, but this is, remember, twenty twenty-one, so four years ago.
- 2:23
Um, if you ask the model to show its reasoning chains, then the answer actually is correct. So basically, by getting the model to, uh, output its chain of thought or reasoning chains, the, the model performance improves.
- 2:36
And, uh, this capability particularly emerged in large language models. These are all the models. Uh, so LaMDA and PaLM were the state-of-the-art models about three years ago. And what I'm showing on X-axis is the increasing number of parameters.
- 2:51
PaLM was scaled all the way up to five forty billion parameters. No one actually publishes the number of parameters these days, so you have to live with the, the graphs from two years ago or the open source stuff that's coming out with DeepSeek and Qwen models.
- 3:05
But what Y-axis is showing is that the solve rate on middle school math word problems was increasing, uh, with the number of parameters in the models, and it was essentially increasing mainly when you were prompting the models and asking them to show chain of thought.
- 3:22
And this led to all kinds of prompting techniques where you ask the model to think step by step. You even go and bribe the model and such, and you ask the model nicely or not.
- 3:31
So this was all kinds of fun stuff. Um, and I think the, the c-- uh, the thing that really stood out from this generation of models three years ago was that this capability was not just limited to mathema-- uh, math problems.
- 3:44
It was, uh, basically, uh, generalizing across a whole bunch of domains, anywhere from question answering in other languages to puzzle problems to, um, multitask, uh, natural language understanding problems.
- 3:59
And what this led to next was that now that these models could reason, we could get them to follow instructions. So the first set of applications that became possible with these large language models were chatbot applications.
- 4:14
So everyone remembers that ChatGPT and, uh, now Gemini and various other chatbots have become extremely popular. All of us use them all the time. But what made them really possible was that when you give instructions to the model to go do something, it's actually able to do it, and the way it learns that is actually based on
- 4:32
reinforcement learning. And, uh, reinforcement learning data that we are giving to the model in this particular case is essentially data based on human feedback. So you're basically saying, "Okay, here is a set of questions, and if I were to give it to a human, and it were-- there were two answers, which one would the human prefer?"
- 4:53
And if you have enough of this data and you train your model, you would actually end up with, uh, better performance because you've taught the model which set of responses to prefer.
- 5:04
And this actually doesn't only work in chatbot applications, it also works in code. So on the bottom right, I'm showing that even if you were to do this for applications in code, you start to see some performance improvements.
- 5:19
Now, of course, the question is that last year there was a whole bunch of debate as to, are we hitting the wall in terms of performance of large language models?
- 5:26
Uh, pre-training is not giving any gains or all, all of these questions were on the horizon. So what is next? And I, I-- one of the key questions to remember in, in all of this is that when you go and pre-train the models, you end up spending a lot of money, um, on training these models.
- 5:44
It could be tens of millions of dollars. And when you in-- do inference on the models, it's extremely cheap. Um, these numbers are not endorsed by any of the companies I worked at, but these are public numbers, uh, from, uh, public sources.
- 5:59
So going back to the main point that I wanna make here is that- Training is extremely costly. So if you constantly try to scale up the model size, you end up, uh, in, in this regime of like, um, if, if it's not giving performance gains, then can we get performance gate at-- uh, gains at inference time because
- 6:17
inference costs are so cheap? And a key idea that was, uh, extremely useful here was that if you could get the models to generate multiple responses and then do majority voting.
- 6:32
So, uh, in the example above, I'm showing that we get-- we-- the, the prompt doesn't make sense, but you've given a mathematical problem to a large language model, and you're asking it to generate three answers independently, and then you basically do some voting on top of those answers.
- 6:50
And if two answers match, then that's a majority vote. Or like, if in this room I were to ask a question and all of you said yes, then that is a majority vote.
- 6:59
So similarly, in large language models, if you can get the model to, like, generate many, many samples and then consistently get it to vo-- uh, like, get many of those answers to agree, this notion of majority voting or self-consistency had shown gains.
- 7:12
So this kind of scaling compute at inference time was clearly one avenue to go push on. Another avenue that emerged and showed substantial value was that you could sequentially revise your previous response.
- 7:25
So as humans, oftentimes we write the first answer, and then we go evaluate our answer, and we are like, "Oh, there's some mistake here. It doesn't quite match," and then you go fix it.
- 7:35
So basically, can we get LLMs to do the same kind of revision looking at previous set of revisions? And this was the second, so basically having longer, uh, chains of thought, uh, and getting the model to improve consistently in inference time based on that.
- 7:50
And these kind of techniques, uh, where you could verify your correct answer, so in math or in programming where you have unit tests, showed, uh, very clear gains. So what I'm showing you here is an example from, uh, uh, one of my colleague's work, um, at Stanford, uh, which is a publicly, uh, published, uh, paper.
- 8:10
And on the y-axis, we have pass@k or coverage score, and on the x-axis, we have, uh, number of samples. So as you basically are doing a lot of samples on the x-axis, your accuracy is improving, uh, with open source DeepSeek model and just taking more samples.
- 8:25
So you're getting a very high score on SWE-bench Verified compared to even state-of-the-art back in end of twenty twenty-four. Uh, of course, now all of these scores have pushed up, and we are roughly somewhere around eighty percent already.
- 8:40
But what we want to take away here is the fact that these lines of work, they showed that inference time compute predictably gives us gains, especially in domains where we can verify.
- 8:54
If we know how to verify the answers, then we actually know how to translate that into intelligence. And going back to my talk title, coding is one of those domains where we do have the capability to verify.
- 9:07
Um, and that gives us tremendous advantage in terms of building super intelligence on top of autonomous coding.
- 9:16
Of course, now you ask the question of what does automated verification mean here? So
- 9:23
for inference time scaling to work, you need basically some way to say this output is correct. Now in math, um, this is a very simple example. If you were to give the input to solve this mathematical equation, um, and if you were to do the same calculation on a calculator, you can actually verify that that problem is
- 9:41
correct or that solution is correct. Um, and similarly in math, you have formal proofs, so you can actually verify things are correct. Uh, in coding, you have unit tests.
- 9:53
In compilers, you can actually generate the code and then use PyTorch as a verifier-- uh, PyTorch, the compiler as a verifier. And in fact, uh, in domains where you don't have, uh, this kind of verification, then there's a large gap.
- 10:07
If you were to generate a lot of solutions and then do majority voting, you actually don't get as much gains. So what this roughly meant was that, okay, so inference time scaling would work in scenarios where I have automated verification, but that doesn't quite solve the problem for it to be-- have real world impact.
- 10:26
And the reason for that is shown in this graph as to typically, uh, if you do majority voting and these, uh, this is across multiple different models on GSM8K, which is middle school math problems and another math benchmark.
- 10:38
Uh, if you were to sort them by correct fraction, you have to sample a lot. The correct generations could be very rare. So who has time to sample ten thousand times and then get a correct solution?
- 10:51
You would be sitting there waiting just finding the correct solution unless you can actually figure out where the correct generation is.
- 10:59
So basically, scaling inference time compute with just majority voting or longer reasoning chains is great in the sense that there are some correct solutions somewhere there, but it doesn't work well across the board.
- 11:11
So what will get these models to learn to generate correctly, uh, during training? Well, in, in the chatbot application scenario, we saw that RL with human feedback did work.
- 11:24
So can we apply the same principle here and get the model to generate correctly in, uh, where, where we can automatically verify the outputs? So our belief at, uh, Reflection is that the next frontier for scaling is reinforcement learning, and we already have proof points from some of the frontier labs as well.
- 11:43
And as David Silver and, um, Sutton published recently, they agree with the-- or, or rather they, they are the pioneers in, uh, in reinforcement learning. They say that we are basically entering the era of experience, like starting from AlphaGo and AlphaZero, uh, where you had an era of simulation, and the next set of large language model era
- 12:05
was where you scaled up with RL using human data. But the next era from This year is really the year of experience, which was-- which will lead us to super intelligence.
- 12:15
So reinforcement learning will be a fundamental component in building, uh, super intelligent systems, uh, especially in areas where we have automated, uh, verification. And some, uh, proof point for why this makes sense is that in math, uh, over several papers, this is, uh, results from '01, but over several papers, we have already seen examples that if you
- 12:39
give the model on the right side, uh, test time compute, uh, on the y-axis, test time compute the same as inference time scaling, and you measure accuracy on the x-axis, it should go up.
- 12:50
Um, but as you can repeat this process, um, and with reinforcement learning, then the training time compute going up on x-axis also improves the accuracy on y-axis for a challenging benchmark in math called AIME.
- 13:04
Uh, most of these benchmarks saturate within a year, as you probably have learned by now. So, uh, this benchmark is already saturated. Um,
- 13:14
so now that I've hopefully convinced you that reinforcement learning and scaling reinforcement learning is, uh, the next frontier, you'd be like, "Okay, so why are-- why is not everyone doing it?
- 13:25
What's so challenging about it?" So as I have built large language models before, a big part of building, uh, these systems, uh, ends up being that the machine learning plus system stack for these, uh, systems themselves is very challenging.
- 13:39
So here is, um, an example of, uh, why scaling up reinforcement learning is challenging. So if you are trying to do reinforcement learning with, uh, PPO, which is, uh, one of the a-algorithms used for RL with human feedback, um, then it moved to, uh, DPO.
- 13:56
You have to keep four copies of, uh, different models. Uh, so if you imagine a really large model, and then you have to keep four copies, then you have to arrange them somewhere on GPUs in your large cluster.
- 14:08
You, you can have some fun figuring out the exact layout and, um, it's, it's, it's a fun and interesting problem, but it's a hard problem in the sense that, uh, to make maximum utilization of these systems and, and arranging them in the right way, just building that system is extremely hard.
- 14:25
And, uh, DeepSeek actually showed, uh, with DeepSeekMath, that GRPO, uh, gets rid of the value model and, and only has three copies of the model, but that doesn't-- that's still a very challenging problem.
- 14:36
So scaling up RL, uh, is more cha-- even more challenging, um, than scaling up, um, LLMs because you have multiple copies of the model and you have a training loop and an inference loop.
- 14:48
And then on the machine learning side, on the, on the reinforcement learning side, you also suffer a lot from reward hacking if your, uh, the model that is deciding that this is the correct answer is a neural reward model.
- 15:00
So you, uh, as we discussed before, in autonomous coding applications, you do have the ability to verify your output, uh, which roughly means that you can decide this is the correct answer or not.
- 15:12
Uh, that's how SWE-bench Verified scores, uh, work today. Um, you have execution feedback, you have unit tests. So all of these possibilities, of course, um, this is, uh, an ongoing list, all of these possibilities me-mean that you can design better reward functions.
- 15:29
Okay, so this means that autonomous coding is a great domain for scaling up RL. Then the question becomes: How does this have real-world impact? So in software engineering applications, generation of code is only one part of the system.
- 15:44
If you look at end-to-end workflows for software engineering, there is many more parts to that system. How do you scale up your system to generalize across all of those domains?
- 15:53
So that's the problem we are trying to solve at Reflection.
- 15:57
Our mission is that we would like to build some super intelligence, and we are starting with autonomous coding as the root node problem for this, um,
- 16:06
uh, mission. And, uh, we have a team of about thirty-five pioneers, um, who are-- who have pioneered various, uh, legendary works in LLMs and reinforcement learning. So if you're excited about this mission, uh, you can reach out to, um, one of us, um, or my, uh, my email is, uh, my last name at reflection[REDACTED:email_address], and we would
- 16:27
love to work with you. And with that, I can take questions. [clapping]
- 16:37
All right, um, same protocol as last time. If you have a question, please come up to one of these three microphones we have distributed throughout. We can probably take one or two questions.
- 16:46
So if you want to ask something, um, feel free. Um, I guess I can-- I, I'll do the first one while people are coming up. So I'm curious, um,
- 16:55
seems like the foundation models are trying to build one model and deploy it across everything. Do you have an opinion with the work you're doing right now if you think that's the right approach, or if you think there'll be more specialization on different languages or even, like, individual code bases?
- 17:09
Um, or do you feel like the best approach is just to have, like, one model that's trained across the, the greatest diversity of tasks possible?
- 17:16
Uh, I think I will answer your question, uh, in terms of building coding agents does require m-um, multiple capabilities, and how you get there, you will definitely need multiple LLM calls.
- 17:26
And then whether that's one model or multiple models, I think that's the secret sauce right now for most people.
- 17:31
Fair enough. All right. Please.
- 17:34
Hi. Um, I'm wondering in the slide with the chart of era of simulation, era of something, and era of experience-
- 17:42
Yeah
- 17:42
... uh, they had put in AlphaGo and, um, the, uh, previous one where also you-- they played Star-StarCraft or something. They all used MCDS, uh, which, I mean, maybe it's my unfamiliarity with them, but it's also data sim-uh, simulation.
- 18:00
Mm-hmm.
- 18:00
Uh, so we're using synthetic data for era of experience as well. So how does-- why is that called simulation, and why is what we're doing right now not called simulation?
- 18:11
What's the sort of overlap between simulation, experience? How does that-- how do you think about that?
- 18:17
I can ask Dave that question, you know. But [laughs] going back to the point, I think, I think the better way to answer that question is, uh, roughly what Greg covered in the last talk, where his comment was that, um, so in gaming, you can envision what scenarios might happen next, and you're basically using that to build your
- 18:35
reinforcement learning. So you're doing rollouts, and you're, you're basically building, uh, based on that. Um, in, in real world, in most scenarios, you have an imperfect rollout, so you don't have full knowledge of how the system might work.
- 18:50
Um, simulation is possible in certain domains where you do build a world model, uh, which is closer to robotics and all the work that's happening in the physical AI space, right?
- 19:03
But in the, in the real-world applications, which is what we're targeting, um, you will have imperfect things. So you have to actually experience the real world, and you have to collect some data, and that data is not gonna be in any way complete, nor will it complete early search, the exponential search space that could exist. [upbeat music]