# RL for Autonomous Coding — Aakanksha Chowdhery, Reflection AI

Aakanksha Chowdhery

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/QluDzKVfp6A-rl-autonomous-coding) · [Video](https://www.youtube.com/watch?v=QluDzKVfp6A)

## Reinforcement Learning for Autonomous Coding

Coding offers something unusually useful for scaling AI: execution feedback. The challenge is turning occasional correct answers into reliable behavior across an entire engineering workflow.

Familiarity with language-model prompting, model training, and unit tests will help; no prior reinforcement-learning expertise is required.

### From larger models to worked answers

[0:15](https://www.youtube.com/watch?v=QluDzKVfp6A&t=15s)

Why would a model get an arithmetic problem wrong when asked for an answer, yet get it right when prompted with worked reasoning? The opening tennis-ball example makes that question concrete: the standard prompt illustrates an answer without intermediate steps; the chain-of-thought prompt illustrates how to work through the calculation.

The path to that example begins with scale. Aakanksha Chowdhery introduces the problem from her experience leading research on **[PaLM](https://arxiv.org/abs/2204.02311)** and working as a lead researcher on **Gemini**, before turning to reinforcement learning for autonomous coding. The 2020 [scaling-laws paper](https://arxiv.org/abs/2001.08361) established empirical power-law relationships between language-model test loss and model size, training data, and compute. Increasing these resources made model performance more predictable. Chowdhery connects that progression to broader generalization, improving benchmarks, and capabilities that were absent in smaller models.

The arithmetic demonstration shows what one such capability looks like. After an example involving tennis balls, a subsequent problem receives an incorrect answer under standard prompting. With intermediate reasoning demonstrated in the prompt, the illustrated answer is correct. **The prompt changes how the model uses its capacity.** This is an example of improved performance, not a guarantee that a worked answer will be right. The public [chain-of-thought paper](https://arxiv.org/abs/2201.11903) and PaLM paper appeared in 2022; Chowdhery’s recollection of 2021 should not be read as their publication date.

![Two columns compare standard prompting with chain-of-thought prompting, showing an incorrect answer marked with a red cross and a worked answer marked with a green check.](https://ai.engineer/api/talk-essay-media/e83b78cd6c3c69d4c6a4f216db10e9c6bda7cafc1aaa3d3bb03a4cf6a05a4db7)

Standard prompting and chain-of-thought prompting on arithmetic examples.

The accompanying comparison plots model parameter count against solve rate on middle-school math word problems for **LaMDA** and PaLM. PaLM reached 540 billion parameters. The notable improvement appears primarily when the larger models receive chain-of-thought prompts, connecting model scale to the usefulness of the prompting technique.

That observation encouraged step-by-step instructions and many informal variations in how people asked models to reason. The broader attraction was that the gains were not confined to arithmetic: Chowdhery describes improvements across multilingual question answering, puzzles, and multitask natural-language understanding.

### Learning which responses people prefer

[3:59](https://www.youtube.com/watch?v=QluDzKVfp6A&t=239s)

Reasoning capabilities were followed by increasingly useful instruction-following systems. ChatGPT, Gemini, and other chatbots could respond to requests rather than merely continue text. In Chowdhery’s account, **reinforcement learning from human feedback**, or RLHF, was a central ingredient in making those interactions useful.

The feedback starts with a comparison: give people a question and two candidate responses, then ask which response they prefer. Accumulating those judgments supplies a training signal that pushes the model toward preferred behavior. The same idea applies to code. The slide pairs the preference-learning loop with a Python task chart showing improvements from RLHF, extending the mechanism beyond conversational answers.

![Slide pairs bullets about learning human preferences with a reward predictor, RL algorithm and environment diagram, above a chart comparing Python fine-tuning with and without RLHF.](https://ai.engineer/api/talk-essay-media/03209da3113902b3a845bc34fea14d78be2ec4869165c7cec31abc8a358ca348)

Human preference learning, with a reward-feedback diagram and Python task performance chart.

### Spend more compute on the answer

[5:19](https://www.youtube.com/watch?v=QluDzKVfp6A&t=319s)

As questions emerged about diminishing returns from pretraining, another resource became attractive: compute spent after training, while solving a particular task. Chowdhery cites public estimates putting large-model training costs in the tens of millions of dollars, while describing individual inference calls as comparatively cheap. These are illustrative public estimates, not figures endorsed by her former employers.

That asymmetry suggests generating more than one answer. In the simplest example, the model independently produces three answers to the same mathematical problem. If two agree, their answer wins the majority vote. Across many samples, this becomes **[self-consistency](https://arxiv.org/abs/2203.11171)**: sample different reasoning paths and aggregate their final answers. Agreement provides a selection rule, although it does not independently establish correctness.

A second approach spends compute sequentially. The model produces an initial response, evaluates it, identifies a mistake, and revises it using the preceding attempts as context.

| Approach | How compute is spent | How an answer emerges |
|---|---|---|
| Independent sampling | Generate separate attempts | Aggregate agreement |
| Sequential revision | Revisit earlier attempts | Evaluate and repair |

These strategies explore different opportunities: sampling searches across alternatives, while revision tries to improve a particular trajectory. Longer reasoning is useful only insofar as the extra work produces a better answer.

### A correct attempt still needs to be found

[7:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=470s)

Repeated attempts are especially promising when an answer can be checked. Chowdhery presents a coding experiment using an open-source **DeepSeek** model, with the number of samples on the horizontal axis and `pass@k`, or coverage, on the vertical axis. More attempts increase the fraction of tasks for which at least one successful solution exists among the candidates.

**Coverage across attempts is different from reliably selecting one correct answer.** The talk describes the result as SWE-bench Verified and compares it with late-2024 frontier performance, but the benchmark identity of the displayed experiment is not established here. Its curve should therefore be read as evidence for the sampling mechanism, not as a precise Verified leaderboard comparison. Chowdhery separately puts then-current SWE-bench Verified scores at roughly 80%, without specifying a model, scaffold, or sample budget. The transferable point is that verification can make additional inference compute useful: coding supplies ways to distinguish successful attempts from failures.

What counts as verification depends on the task. An arithmetic result can be checked by performing the calculation independently. Formal proofs offer a different kind of check, and code can be evaluated with unit tests. Chowdhery also points to compiler- and PyTorch-based checking for generated code, without developing a particular implementation.

A small Python example makes the distinction between agreement and checking concrete. Suppose the task is to implement `square`. These candidates are checked against expected results rather than selected by how often their text or outputs resemble one another:

```python
from collections.abc import Callable


def square_a(x: int) -> int:
    return x * 2


def square_b(x: int) -> int:
    return x * x


def verify(candidate: Callable[[int], int]) -> bool:
    cases = [(0, 0), (2, 4), (3, 9), (-3, 9)]
    return all(candidate(x) == expected for x, expected in cases)


candidates = {"a": square_a, "b": square_b}
accepted = [name for name, candidate in candidates.items() if verify(candidate)]
print(accepted)
```

The case `2 → 4` alone would accept both implementations; `3 → 9` distinguishes them. Passing this finite test set establishes only that the candidate satisfies these checks. It does not prove correctness for every possible input. This is the practical value—and boundary—of execution feedback.

Without an effective verifier, generating many answers and voting over them yields weaker gains. Even with one, search may be expensive because correct generations can be rare. The math analysis in the talk sorts problems by their fraction of correct generations, using GSM8K and another math benchmark. Some problems sit in a region where a successful sample is difficult to obtain.

Chowdhery’s example of waiting for 10,000 samples expresses that latency problem, not a universal sampling requirement. More attempts may establish that a model can sometimes solve a task, while still leaving the user waiting too long. Majority voting and longer reasoning chains therefore do not, by themselves, make correctness reliable across tasks.

### Make successful generations more likely

[11:11](https://www.youtube.com/watch?v=QluDzKVfp6A&t=671s)

The next step is to move some of the burden from inference into training. If human preferences can teach a chatbot which responses to produce, automatically checked outcomes can teach a coding model which attempts succeed. **Reinforcement learning aims to make useful behavior more likely, rather than repeatedly searching for a rare success.** Reflection identifies this as its next scaling frontier.

Chowdhery places that direction within David Silver and Richard Sutton’s era-of-experience framing: game systems such as **AlphaGo** and **AlphaZero** learned through simulation; language models subsequently drew heavily on human data; future systems would learn more from their own interactions. The forecast that this progression leads to superintelligence is an ambition, not a result demonstrated by the talk.

The supporting example is **[o1](https://openai.com/index/learning-to-reason-with-llms/)** on AIME mathematics problems. Its results show two distinct opportunities to improve accuracy: spend more compute at test time, and spend more compute on reinforcement learning during training. The paired plots keep those two axes of investment separate.

Chowdhery describes AIME as already saturated at the time of the talk. That is her assessment of benchmark progress, not an unconditional claim that every AIME evaluation setting has been solved. The evidence relevant to the mechanism is the upward relationship between compute and accuracy under the reported conditions.

![Two scatter plots show o1 AIME accuracy against train-time compute and test-time compute, with upward blue arrows.](https://ai.engineer/api/talk-essay-media/9eec970fd218edc7e856c236d2e79903d2d7d3fade9a94c6d2d6d25ea3c187af)

AIME accuracy increases with training-time and test-time compute.

### The systems cost of the learning loop

[13:14](https://www.youtube.com/watch?v=QluDzKVfp6A&t=794s)

If reinforcement learning offers this opportunity, why is scaling it difficult? A large part of the answer lies in the combined machine-learning and systems stack. A PPO-style RLHF setup involves four functional model roles:

| Role | Responsibility |
|---|---|
| Policy | Generate responses and receive updates |
| Reference | Anchor changes to the policy |
| Reward | Score generated responses |
| Value | Estimate expected reward |

Chowdhery describes these as four copies to place across a GPU cluster. The useful interpretation is four roles, not a universal requirement for four identical physical models. Their placement must sustain utilization while supporting the work each role performs. [Direct Preference Optimization](https://arxiv.org/abs/2305.18290), mentioned briefly in the same discussion, is a separate preference-training approach; the four-role description belongs to the PPO-style setup.

**[DeepSeekMath](https://arxiv.org/html/2402.03300v3)** introduces **Group Relative Policy Optimization**, or GRPO, which removes the separate learned value model and derives its baseline from rewards across a group of outputs. In the talk’s accounting, that reduces four model roles to three.

Removing a role helps, but it does not remove the fundamental scheduling problem. RL combines an inference loop that generates experience with a training loop that updates the policy. Both require substantial compute, and the overall system has to keep them working efficiently together. Scaling this arrangement introduces difficulties beyond scaling a conventional language-model training run.

### Better rewards, broader engineering tasks

[14:48](https://www.youtube.com/watch?v=QluDzKVfp6A&t=888s)

The learning problem is also vulnerable to **reward hacking**. A policy can discover responses that score well under a neural reward model without accomplishing the intended task. Coding offers additional signals: execute the output, inspect the feedback, and run unit tests. Those observations can support more useful reward functions than a learned judgment alone.

[SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) is the concrete benchmark example. Its tests check candidate fixes against expected behavior; “Verified” refers to human screening of the benchmark tasks, not a certification that arbitrary generated software is correct. Execution feedback improves reward design, but its usefulness still depends on what the checks actually cover.

That makes autonomous coding a promising domain for RL, but code generation covers only part of software engineering. Real-world impact requires a system to generalize across the other parts of an end-to-end workflow. Producing a patch is one capability within that larger job.

Reflection treats autonomous coding as the starting problem for its broader superintelligence mission. At the time of the talk, Chowdhery describes a team of about 35 people with backgrounds in language models and reinforcement learning, and invites people interested in that mission to join. The remaining technical challenge is broader than improving a code-generation score: the learning system must support the range of capabilities that engineering work demands.

### Multiple calls do not necessarily mean multiple models

[16:55](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1015s)

The first audience question asks how those capabilities should be packaged. Should one foundation model serve the widest possible range of tasks, or should coding systems specialize by programming language or even by individual codebase?

Chowdhery answers at the level of the agent: coding requires multiple capabilities and multiple LLM calls. She does not disclose whether those calls use one model or several distinct models. That distinction matters. A workflow with several model invocations does not, by itself, imply an architecture with specialized models; the model-composition choice remains open in her answer.

### Experience in an imperfect environment

[17:34](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1054s)

The final question challenges the boundary between simulation and experience. The audience member invokes game-playing systems, Monte Carlo tree search, and synthetic data: if both eras involve generating trajectories, what makes the newer one different?

Chowdhery’s answer turns on knowledge of the environment. In games, a system can anticipate subsequent scenarios and build reinforcement learning around rollouts. Real-world rollouts are imperfect: the agent does not have full knowledge of how the surrounding system will behave. World models can enable simulation in domains such as robotics and physical AI, but they do not make all real-world applications fully predictable.

For the applications Reflection is targeting, an agent must interact with the real world and collect data from what happens. That experience will remain incomplete. It will neither describe the environment fully nor exhaust the exponential space of possible situations. Autonomous coding therefore needs more than a mechanism for generating and scoring answers: it needs to learn from partial experience in systems whose behavior it cannot completely simulate.

## Resources

- [Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) — The foundational empirical study relating language-model loss to model size, data and training compute.
- [PaLM: Scaling Language Modeling with Pathways](https://arxiv.org/abs/2204.02311) — The 540-billion-parameter PaLM model and its evaluations across language, reasoning and coding tasks.
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) — Experiments showing how worked reasoning examples improve arithmetic, commonsense and symbolic reasoning.
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171) — Sampling multiple reasoning paths and selecting answers through agreement.
- [Large Language Monkeys](https://arxiv.org/abs/2407.21787) — Repeated-sampling experiments measuring coverage and the limitations of selecting correct answers from many attempts.
- [Introducing SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/) — The construction and evaluation rationale for a human-screened subset of 500 software-engineering tasks.
- [Learning to reason with LLMs](https://openai.com/index/learning-to-reason-with-llms/) — Historical o1 results showing how reinforcement learning and inference compute affect reasoning performance.
- [DeepSeekMath](https://arxiv.org/html/2402.03300v3) — Introduces GRPO and explains how grouped rewards replace a separate learned value model.
- [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) — A preference-training objective that avoids an explicit reward model and online sampling during fine-tuning.

## Slides

### RL for Autonomous Coding

Aakanksha Chowdhery introduces her work on autonomous coding with reinforcement learning, following research on PaLM and Gemini at Google.

[0:24](https://www.youtube.com/watch?v=QluDzKVfp6A&t=24s) [Title slide for RL for Autonomous Coding by Aakanksha Chowdhery, with Reflection branding.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb4424a4feebc793a8737b84b334ecc71e7b4fdd2d6f07f85350a71383a6f6b4.webp)

RL for Autonomous Coding
Aakanksha Chowdhery
Reflection

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Recapping the era of LLMs

Aakanksha Chowdhery begins by tracing progress in large language models and asking why autonomous coding is timely.

[0:30](https://www.youtube.com/watch?v=QluDzKVfp6A&t=30s) [Section slide reading “Recapping the era of LLMs…”](https://assets.ai.engineer/sites/aiecode2025/sha256/a959026863e05f519395435ed2aaccbc3159d8c9ca6bd35ec800ad3ae09ff6b1.webp)

Recapping the era of LLMs...

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Pre-training scaling laws for LLMs

The 2020 scaling-laws charts show test loss falling as compute, dataset size, and model parameters increase. Chowdhery describes a power-law relationship and says larger models also generalize to other domains.

[0:41](https://www.youtube.com/watch?v=QluDzKVfp6A&t=41s) [Slide describing power-law relationships between language-model test loss and compute, dataset size, and parameter count, with three graphs.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d0447d8a5a2447e4d099711105f9fd62b076f93f116f43e8929489481b7c9d0.webp)

(Pre-Training) Scaling Laws for Large Language Models (LLMs)

There is a power law relationship between LLMs test loss and:
• Amount of compute used
• Dataset size
• Number of parameters in the model

Test Loss
Compute
PF-days, non-embedding
Y-axis ticks: 2, 3, 4, 5, 6, 7
X-axis ticks: 10⁻⁹, 10⁻⁷, 10⁻⁵, 10⁻³, 10⁻¹, 10¹
L = (Cmin/2.3 · 10⁸)⁻⁰·⁰⁵⁰

Dataset Size
tokens
Y-axis ticks: 2.7, 3.0, 3.3, 3.6, 3.9, 4.2
X-axis ticks: 10⁸, 10⁹
L = (D/5.4 · 10¹³)⁻⁰·⁰⁹⁵

Parameters
non-embedding
Y-axis ticks: 2.4, 3.2, 4.0, 4.8, 5.6
X-axis ticks: 10⁵, 10⁷, 10⁹
L = (N/8.8 · 10¹³)⁻⁰·⁰⁷⁶

[illegible]caling Laws for Neural Language Models, 2020

Interface text:
docs.google.com/presentation/d/[illegible]/edit?slide=id.[illegible]
New Chrome [illegible]

### Why did LLMs get bigger?

As LLMs grew, benchmark performance improved continuously, and larger models showed capabilities absent from smaller ones.

[1:27](https://www.youtube.com/watch?v=QluDzKVfp6A&t=87s) [Slide listing continuous improvement with scaling and capabilities that emerge in larger models.](https://assets.ai.engineer/sites/aiecode2025/sha256/e83ad2e1c624fcb09e990ae62067c0d23643453a4b8cebd84e65305bbf69dd4d.webp)

Why did LLMs get bigger?

• Continuous improvement with scaling
• Emergent Behavior: Capabilities that only emerge in larger models

Interface text:
docs.google.com/presentation/d/[illegible]/edit?slide=id.g3617bec77aa_0_133#slide=id.g3617bec77aa_0_133
New Chrome [illegible]

### Emergent Behavior: Chain of Thought

Given an example with intermediate steps, the model solves the apples problem as 23 − 20 + 6 = 9. The standard prompt gives 27.

[1:46](https://www.youtube.com/watch?v=QluDzKVfp6A&t=106s) [Standard prompting gives an incorrect answer to an apples problem, while chain-of-thought prompting shows steps leading to the correct answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd855a9e36abc9a03503049d07298d24c1d53a4fad9e3e94c4e46ae779332b3f.webp)

Emergent Behavior: Chain of Thought

Standard Prompting

Model Input
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?

A: The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?

Model Output
A: The answer is 27. ✗

Chain-of-Thought Prompting

Model Input
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?

A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.

Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?

Model Output
A: The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9. ✓

[illegible]ow Your Work: Scratchpads for Intermediate Computation with Language Models, 2021
[illegible]ain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022 (Figure is from this publication)

Page: 5

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Chain of Thought emerges in Large Models

On GSM8K middle-school math problems, chain-of-thought prompting rises sharply with model scale for GPT and PaLM, while standard prompting improves more modestly.

[2:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=170s) [Charts compare standard and chain-of-thought prompting on math word problems across LaMDA, GPT and PaLM model sizes.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d0ebb544bf1c61583f35e8b955d1813af4603bae440bf5952cf2a06b9444b93.webp)

Chain of Thought emerges in Large Models

Middle school math word problems

LaMDA
GPT
PaLM

Y-axis: GSM8K solve rate (%)
Y-axis ticks: 0, 20, 40, 60
X-axis: Model scale (# parameters in billions)
LaMDA ticks: 0.4, 8, 137
GPT ticks: 0.4, 7, 175
PaLM ticks: 8, 62, 540

Standard prompting
Chain-of-thought prompting
Prior supervised best

[illegible]ain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022

Interface text: docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Emergent Behavior: Ability to Solve Entirely New Tasks

Chowdhery describes capabilities extending beyond math to question answering, puzzles, and natural language understanding. Eight plots compare task scores across model training scales; the visible points cluster at smaller scales.

[3:31](https://www.youtube.com/watch?v=QluDzKVfp6A&t=211s) [Eight plots compare model performance on tasks including arithmetic, transliteration, question answering and language understanding across training scales.](https://assets.ai.engineer/sites/aiecode2025/sha256/60ca708b609bb05dd57432f0e58229f41b66d7a55c80b15b5e53d558deb3244f.webp)

Emergent Behavior: Ability to Solve Entirely New Tasks

Legend: LaMDA; GPT-3; Gopher; Chinchilla; PaLM; Random

(A) Mod. arithmetic
(B) IPA transliterate
(C) Word unscramble
(D) Persian QA
Y-axis ticks: 0, 10, 20, 30, 40, 50
X-axis ticks: 10¹⁸, 10²⁰, 10²², 10²⁴
(B) Y-axis: BLEU (%)
(C), (D) Y-axis: Exact match (%)

(E) TruthfulQA
(F) Grounded mappings
(G) Multi-task NLU
(H) Word in context
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70
X-axis ticks: 10²⁰, 10²², 10²⁴
(F), (G), (H) Y-axis: Accuracy (%)

Accuracy (%)
Model Scale (Training Flops)

[illegible]n-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022

Interface text:
docs.google.com/presentation/[illegible]
New Chrome [illegible]

### What is next?

Chowdhery asks what follows pre-training and finetuning amid debate over whether LLM performance is hitting a wall.

[5:17](https://www.youtube.com/watch?v=QluDzKVfp6A&t=317s) [A pipeline leads from pre-training to finetuning and then to a box marked with question marks.](https://assets.ai.engineer/sites/aiecode2025/sha256/1fced94919b6e0f351daed322f22a31ac705c0837ee9649004e95659c56b2ea1.webp)

What is next?

(Pre-)training → Finetuning → ??

9

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### AI Training and Inference Are Both Costly

In the OctoML example, training costs are high upfront, while inference spending rises over the year, totaling $6,271,650 versus $1,700,000 for training.

[5:32](https://www.youtube.com/watch?v=QluDzKVfp6A&t=332s) [Bullets contrast upfront training costs with recurring inference costs; a chart illustrates costs over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/d793f0590dd2a30a87a500d61c33dc07c66eff475013f5f541cedfd978e575e3.webp)

AI Training and Inference Are Both Costly

Training:
- A large upfront cost as training is done infrequently
- For large LLMs, training could cost upwards of tens of millions of dollars

Inference:
- Everytime the model processes an input (e.g., generating response to a prompt), it incurs an inference cost
- A single inference call is cheap (< $0.0001 for large LLMs), but accumulates over time

Training → Inference

AI/ML Cloud Spend: Training v. Production
Y-axis: Monthly AI Costs
Y-axis ticks: $0, $250,000, $500,000, $750,000, $1,000,000, $1,250,000, $1,500,000
Timeline: Jan, Feb, Mar, Apr, May, June, July, Aug, Sept, October, Nov, Dec
Production (Inference) Annual Total: $6,271,650
Training Annual Total: $1,700,000

An example production model shows the large upfront cost of training and increasing cost of inference as users increase (source: OctoML)

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome [illegible]

### Scaling compute at inference time

A model can generate independent answers and use majority voting, or revise an answer sequentially using its previous revisions.

[6:18](https://www.youtube.com/watch?v=QluDzKVfp6A&t=378s) [Diagrams compare generating multiple answers in parallel with sequentially revising an answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/3433e5948a5cd8499cfec1b955d1a41c5334abd42360dddcb627755351192db4.webp)

Scaling compute at inference time

Can the model improve consistently as it spends more time thinking?

• Generate many many samples + majority voting
• Sequentially revise the previous response

Q: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
LM
A: So 7/4 yap/dap ...
A: We have 4 dap...
A: If 7/4 yaps/dap ...
...
LM proposes answers independently, in parallel

Sequential Revisions

LM proposes a sequence of revisions, each conditioned on previous revisions

Q: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
LM
A: We ...
A: So ...
A: If 7/4 ...

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Scaling inference on SWE-bench Verified

Repeated sampling with open-source DeepSeek raises SWE-bench Verified coverage to 70+%, above the slide’s 62.2% state of the art as of December 13, 2024.

[7:49](https://www.youtube.com/watch?v=QluDzKVfp6A&t=469s) [A chart shows coverage rising with repeated samples, alongside benchmark comparisons.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1ec167b85c795bb2b5cb93aa88aee268a106ce8cd35831389dc7d95dddb87c3.webp)

Scaling Inference works for Coding Benchmarks Like SWE-Bench Verified!

In domains with automated verification such as formal proofs for math and unit tests for programming, scaling sampling readily leads to more capability

Y-axis: Coverage (Pass@k)
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
X-axis: Number of Samples
X-axis ticks: 10⁰, 10¹, 10², 10³

70+% coverage on SWE-bench Verified with open-source DeepSeek + repeated sampling
62.2% SOTA as of 12/13/2024, used Claude-3-5-sonnet-new as backend (by Aide)
38.4% - O1 Preview

[illegible]onkeys: Scaling Inference Compute with Repeated Sampling, 2024,
[illegible]ordan Juravsky*, Ryan Ehrlich*, Ronnie Clark, Quoc Le, Chris Re, Azalia Mirhoseini

12

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome a[illegible]

### Inference-time compute and compute allocation

The slide contrasts a costly pre-training and fine-tuning pipeline with an alternative spending $1k+ on inference, potentially offline. Chowdhery highlights coding as a domain where answers can be verified.

[8:38](https://www.youtube.com/watch?v=QluDzKVfp6A&t=518s) [A diagram compares spending on pre-training, fine-tuning and inference with an alternative emphasizing inference.](https://assets.ai.engineer/sites/aiecode2025/sha256/130b239c89329c33cdf7f524453af6f106d7e574074f26367cdacf647d848f90.webp)

Inference-Time Compute predictably translates to Intelligence

This motivates a shift in compute allocations, especially for domains with automated verifiers

Current Paradigm
Pre-training → Fine-tuning → Inference
$100M+ → $1M+ → <$0.001

Alternative Paradigm
Inference
$1k+
Could be offline

13

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Need for automated verification

Chowdhery turns to what automated verification means.

[9:14](https://www.youtube.com/watch?v=QluDzKVfp6A&t=554s) [Section slide reading Need for Automated Verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/9909783e4807849c694058a8c4377914af228d53023fc41260aff437686f1c76.webp)

Need for Automated Verification!

14

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Inference-time scaling requires robust verification

A calculator checks the LLM’s answer to 5 × (3 + 4). Both return 35, illustrating how an independent calculation can verify an output.

[9:20](https://www.youtube.com/watch?v=QluDzKVfp6A&t=560s) [A calculator verifies an LLM's answer of 35 to 5 times the sum of 3 and 4.](https://assets.ai.engineer/sites/aiecode2025/sha256/4cb57d4aad759f017f1b77afc6ad293fc21e26df7942f63caf36904f28f50a7a.webp)

Inference-Time Scaling Requires Robust Verification

Input: Solve 5 × (3 + 4)

LLM output:
5 × (3 + 4) =
5 × 7 =
= 35

Verifier output:
5×(3+4)
35
AC  +/-  %  ÷
7  8  9  ×
4  5  6  −
1  2  3  +
0  .  =

Correct

Source: https://sebastianraschka.com/blog/2025

Interface text: docs.google.com/presentation/d/[illegible]

### Inference-Time Scaling Requires Robust Verification

Automated verification makes repeated sampling more useful: proofs, unit tests and source PyTorch can check outputs. Without it, majority voting and reward models fall short of best-of-N.

[9:42](https://www.youtube.com/watch?v=QluDzKVfp6A&t=582s) [A slide lists formal proofs, coding unit tests and using source PyTorch to verify generated CUDA.](https://assets.ai.engineer/sites/aiecode2025/sha256/434abafa62f0ac25f624485437436a01968a0c1b6930cd05da3e2f42b54deaab.webp)

Inference-Time Scaling Requires Robust Verification

• In domains with automated verification, repeated sampling directly leads to more capability
  ◦ Formal proofs for math
  ◦ Unit tests for coding
  ◦ AI as a compiler: e.g. generate CUDA and use source Pytorch as verifier

16

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

[9:58](https://www.youtube.com/watch?v=QluDzKVfp6A&t=598s) [Slide contrasts automated verification in math, coding and compilation with the limitations of voting and model-based ranking.](https://assets.ai.engineer/sites/aiecode2025/sha256/5205507ebb448dcf96c0dca4e809bdcd6a73675282b72291b2e59ea6cc50110a.webp)

Inference-Time Scaling Requires Robust Verification

• In domains with automated verification, repeated sampling directly leads to more capability
  ◦ Formal proofs for math
  ◦ Unit tests for coding
  ◦ AI as a compiler: e.g. generate CUDA and use source Pytorch as verifier
• For domains without verification, there is a large gap between “best-of-N” and methods such as majority voting and model-based rankers (i.e. reward models)

17

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Correct Generations Could Be Very Rare

Majority voting and longer reasoning chains do not work well across the board. Chowdhery asks whether reinforcement learning can teach models to generate correct answers during training.

[10:57](https://www.youtube.com/watch?v=QluDzKVfp6A&t=657s) [Slide says majority voting and longer reasoning chains do not work well across the board and asks whether reinforcement learning teaches correct generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c6ee55d754d874d370a3c3050403550315b95e9777510e4598381923dd23e18.webp)

But the Correct Generations Could Be Very Rare

Scaling inference-time compute with majority voting or longer reasoning chains doesn’t work well across the board!

Does reinforcement learning teach the model to generate correctly with training?

19

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### A New Frontier for Scaling Is RL

Reflection sees reinforcement learning as the next scaling frontier, asking whether models can learn correct outputs when those outputs can be automatically verified. The diagram leaves the stage after RLHF open.

[11:31](https://www.youtube.com/watch?v=QluDzKVfp6A&t=691s) [Diagram connects pretraining to RLHF and an unspecified next stage.](https://assets.ai.engineer/sites/aiecode2025/sha256/8554a7e6e2d2ae510401a5cd4b80640b4f009f87ea25c2e342609d701caac484.webp)

What’s Next?
A New Frontier for Scaling is RL!

(Pre-)training → RLHF → ??

20

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome a[illegible]

### Reinforcement Learning and General Superintelligence

The chart shows attention to RL rising during the era of simulation, falling during the era of human data, and then rising toward general superintelligence in the projected era of experience.

[11:41](https://www.youtube.com/watch?v=QluDzKVfp6A&t=701s) [Conceptual graph traces attention to reinforcement learning through eras of simulation, human data and experience, ending with a projected rise toward general superintelligence.](https://assets.ai.engineer/sites/aiecode2025/sha256/5070ab964b544fee950eb93bf183071c16d8cb6813e83e6de2a80f0659ac8f0e.webp)

Reinforcement Learning – the path to general superintelligence

Era of Simulation
Era of Human Data
Era of Experience

Y-axis: Attention on Reinforcement Learning
X-axis: Year
Timeline: 2014, 2016, 2018, 2020, 2022, 2024

Atari
AlphaGo
AlphaZero
GPT3
GPT4
ChatGPT
Gemini
AlphaProof
Computer Use
Coding agents
General superintelligence
superhuman intelligence

Welcome to the Era of Experience, Silver & Sutton, 2025

Interface text: docs.google.com/presentation/d/[illegible]

### Scaling RL in Math Domains

On AIME, o1’s pass@1 accuracy rises as either training-time or test-time compute increases. Both compute axes use a log scale and show a fraction of maximum FLOPs.

[12:24](https://www.youtube.com/watch?v=QluDzKVfp6A&t=744s) [Two scatter plots show o1 AIME accuracy rising with training-time compute and test-time compute.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4b38598cf08dfa46fb48f4a6cc2e3d0fc6d72b65ba3eee9d2f7f76b8e4d1991.webp)

Scaling up RL in MATH domains shows significant gains...

Increasing both training and test-time compute improves model performance

o1 AIME accuracy during training
Y-axis: pass@1 accuracy
Y-axis ticks: 0, 20, 40, 60, 80, 100
X-axis: train-time compute (log scale, fraction of maximum flops)
X-axis ticks: 10⁻², 10⁻¹, 10⁰

o1 AIME accuracy at test time
Y-axis: pass@1 accuracy
Y-axis ticks: 0, 20, 40, 60, 80, 100
X-axis: test-time compute (log scale, fraction of maximum flops)
X-axis ticks: 10⁻², 10⁻¹, 10⁰

Competitive Programming with Large Reasoning Models, https://arxiv.org/abs/2502.06807

22

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome a[illegible]

### PPO memory footprint

PPO-based RLHF uses four models: the original and new policies, a critic, and a reward model. Placing these large models across GPUs while keeping the system well utilized is difficult.

[13:11](https://www.youtube.com/watch?v=QluDzKVfp6A&t=791s) [Four boxes show the original policy, new policy, critic, and reward model used in RLHF with PPO.](https://assets.ai.engineer/sites/aiecode2025/sha256/15af10c10302f427fe45ad76991479c52ce394559a9a87be0dbdbb4174f83ba0.webp)

What makes scaling RL challenging?

ML system for RLHF with PPO is challenging in memory footprint.

Original policy
(LLM from SFT stage)

New policy
(LLM being trained with PPO)

Critic
(Value model estimating expected reward)

Reward model
(Trained from human feedback; predicts reward scores)

Source: https://sebastianraschka.com/blog/2025

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome [illegible]

### GRPO memory footprint

GRPO removes the critic, but the original policy, new policy, and reward model remain. Multiple model copies and training and inference loops still make scaling RL challenging.

[14:24](https://www.youtube.com/watch?v=QluDzKVfp6A&t=864s) [A diagram crosses out the critic while retaining the original policy, new policy, and reward model; the new-policy box still says PPO.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d2c2b715e82b02f3361fc0096d9c2bf8c1b7fa93004ebf49215a46c4c018310.webp)

What makes scaling RL challenging?

ML system for RLHF with GRPO is still challenging in memory footprint.

Original policy
(LLM from SFT stage)

New policy
(LLM being trained with PPO)

Critic
(Value model estimating expected reward)

Reward model
(Trained from human feedback; predicts reward scores)

Source: https://sebastianraschka.com/blog/2025

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome av[illegible]

### Verifiable rewards for coding

Neural reward models can suffer from reward hacking. Coding offers verifiable rewards through correct answers, execution feedback and unit tests, which can help shape better reward functions.

[14:47](https://www.youtube.com/watch?v=QluDzKVfp6A&t=887s) [The slide contrasts reward hacking with verifiable coding rewards, including correct answers, execution feedback, and unit tests.](https://assets.ai.engineer/sites/aiecode2025/sha256/a0bc8da84ddaea9af885a6b02a272bd4d85b83408e4deda14000af265e697ef3.webp)

What makes scaling RL challenging?

Neural reward models may suffer from reward hacking in the large-scale RL.

However, autonomous coding applications have access to verifiable rewards

• Correct final answer for given inputs
• Execution feedback
• Unit Tests

...

Interface text:
docs.google.com/presentation/d/[illegible]/edit?slide=id.g3617bec77aa_0_813#slide=id.g3617bec77aa_0_813
New Chrome [illegible]

### Reflection's mission

Reflection starts with autonomous coding in its mission to build superintelligence, while exploring how to generalize across software engineering workflows.

[15:56](https://www.youtube.com/watch?v=QluDzKVfp6A&t=956s) [The slide states a mission of building superintelligence starting with autonomous coding.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b5e5e34449a4cdc276acf4c80f05094f476b7119941555ddee15d990298edbf.webp)

Building superintelligence starting
with autonomous coding

Interface text:
docs.google.com/presentation/d/[illegible]/edit?slide=id.[illegible]
New Chrome av[illegible]

### Reflection team

Reflection describes its 35-person team as LLM and RL pioneers with 200k+ citations. Members contributed to projects from AlphaGo to ChatGPT and previously worked at leading AI labs and universities.

[16:06](https://www.youtube.com/watch?v=QluDzKVfp6A&t=966s) [The slide describes a team of 35 LLM and RL pioneers and displays projects and organizations associated with its members.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4dacdd62b3b7616e6bad51289f97819f9244b0bc7ddd64d02620f8193f8d517.webp)

Reflection team

Team of 35 pioneers in LLMs and RL with 200k+ citations

breakthroughs contributed by the team
AlphaGo
AlphaZero
Gemini
ChatGPT
PaLM
AlphaProof

formerly from
DeepMind
ANTHROPIC
OpenAI
character.ai
Berkeley
University of California
MIT
Stanford University

Interface text:
docs.google.com/presentation/d/[illegible]
New Chrome [illegible]

## Full transcript

[0:00](https://www.youtube.com/watch?v=QluDzKVfp6A&t=0s) [on-hold music] Hi, everyone.

[0:15](https://www.youtube.com/watch?v=QluDzKVfp6A&t=15s) I'm Aakanksha. I was at Google for more than six years, and I led the research for PaLM, and I was a lead researcher in Gemini. Uh, these days, I'm working on, uh, pushing the frontier for autonomous coding, uh, with reinforcement learning.

[0:32](https://www.youtube.com/watch?v=QluDzKVfp6A&t=32s) So just to recap the arc of how we have progressed in large language models and, um, why autonomous coding and why now. Um,

[0:43](https://www.youtube.com/watch?v=QluDzKVfp6A&t=43s) so I think everyone here, or those of you, uh, who don't remember, in twenty twenty there was this breakthrough paper that came out which talked about scaling laws for large language models.

[0:53](https://www.youtube.com/watch?v=QluDzKVfp6A&t=53s) And if you were to take a thirty-second recap, all-- the main thing it said was that there's a power law relationship between the test loss of large language models.

[1:03](https://www.youtube.com/watch?v=QluDzKVfp6A&t=63s) Um, so if you use more compute, more data, and put more parameters in your machine learning model, uh, which is a transformer model, you will get more performant models, and it will not be performant just in the domain in which you are training the model.

[1:19](https://www.youtube.com/watch?v=QluDzKVfp6A&t=79s) It will actually be performant, and it will generalize to many other domains. And the generalization was, uh, pretty much a feature, uh, in this particular case. So as the, uh, large language models got bigger, we saw continuous improvement across benchmarks to the point that they're starting to get saturated now.

[1:38](https://www.youtube.com/watch?v=QluDzKVfp6A&t=98s) And the other interesting thing was that we saw emergent behavior, uh, where capabilities were emerging in large language models that were not present in smaller models. And this is a classic slide that I show for, for the work that we did, uh, in PaLM.

[1:54](https://www.youtube.com/watch?v=QluDzKVfp6A&t=114s) So typically, when you go about trying to solve math problems, and you give the model some examples, on the left you have a math problem around tennis balls, and then you give a second problem.

[2:06](https://www.youtube.com/watch?v=QluDzKVfp6A&t=126s) The model output looks wrong. But what PaLM and the subsequent set of, uh, papers showed was that if you ask the model to output its, uh, reasoning chains, uh, which has become a very common concept now, but this is, remember, twenty twenty-one, so four years ago.

[2:23](https://www.youtube.com/watch?v=QluDzKVfp6A&t=143s) Um, if you ask the model to show its reasoning chains, then the answer actually is correct. So basically, by getting the model to, uh, output its chain of thought or reasoning chains, the, the model performance improves.

[2:36](https://www.youtube.com/watch?v=QluDzKVfp6A&t=156s) And, uh, this capability particularly emerged in large language models. These are all the models. Uh, so LaMDA and PaLM were the state-of-the-art models about three years ago. And what I'm showing on X-axis is the increasing number of parameters.

[2:51](https://www.youtube.com/watch?v=QluDzKVfp6A&t=171s) PaLM was scaled all the way up to five forty billion parameters. No one actually publishes the number of parameters these days, so you have to live with the, the graphs from two years ago or the open source stuff that's coming out with DeepSeek and Qwen models.

[3:05](https://www.youtube.com/watch?v=QluDzKVfp6A&t=185s) But what Y-axis is showing is that the solve rate on middle school math word problems was increasing, uh, with the number of parameters in the models, and it was essentially increasing mainly when you were prompting the models and asking them to show chain of thought.

[3:22](https://www.youtube.com/watch?v=QluDzKVfp6A&t=202s) And this led to all kinds of prompting techniques where you ask the model to think step by step. You even go and bribe the model and such, and you ask the model nicely or not.

[3:31](https://www.youtube.com/watch?v=QluDzKVfp6A&t=211s) So this was all kinds of fun stuff. Um, and I think the, the c-- uh, the thing that really stood out from this generation of models three years ago was that this capability was not just limited to mathema-- uh, math problems.

[3:44](https://www.youtube.com/watch?v=QluDzKVfp6A&t=224s) It was, uh, basically, uh, generalizing across a whole bunch of domains, anywhere from question answering in other languages to puzzle problems to, um, multitask, uh, natural language understanding problems.

[3:59](https://www.youtube.com/watch?v=QluDzKVfp6A&t=239s) And what this led to next was that now that these models could reason, we could get them to follow instructions. So the first set of applications that became possible with these large language models were chatbot applications.

[4:14](https://www.youtube.com/watch?v=QluDzKVfp6A&t=254s) So everyone remembers that ChatGPT and, uh, now Gemini and various other chatbots have become extremely popular. All of us use them all the time. But what made them really possible was that when you give instructions to the model to go do something, it's actually able to do it, and the way it learns that is actually based on

[4:32](https://www.youtube.com/watch?v=QluDzKVfp6A&t=272s) reinforcement learning. And, uh, reinforcement learning data that we are giving to the model in this particular case is essentially data based on human feedback. So you're basically saying, "Okay, here is a set of questions, and if I were to give it to a human, and it were-- there were two answers, which one would the human prefer?"

[4:53](https://www.youtube.com/watch?v=QluDzKVfp6A&t=293s) And if you have enough of this data and you train your model, you would actually end up with, uh, better performance because you've taught the model which set of responses to prefer.

[5:04](https://www.youtube.com/watch?v=QluDzKVfp6A&t=304s) And this actually doesn't only work in chatbot applications, it also works in code. So on the bottom right, I'm showing that even if you were to do this for applications in code, you start to see some performance improvements.

[5:19](https://www.youtube.com/watch?v=QluDzKVfp6A&t=319s) Now, of course, the question is that last year there was a whole bunch of debate as to, are we hitting the wall in terms of performance of large language models?

[5:26](https://www.youtube.com/watch?v=QluDzKVfp6A&t=326s) Uh, pre-training is not giving any gains or all, all of these questions were on the horizon. So what is next? And I, I-- one of the key questions to remember in, in all of this is that when you go and pre-train the models, you end up spending a lot of money, um, on training these models.

[5:44](https://www.youtube.com/watch?v=QluDzKVfp6A&t=344s) It could be tens of millions of dollars. And when you in-- do inference on the models, it's extremely cheap. Um, these numbers are not endorsed by any of the companies I worked at, but these are public numbers, uh, from, uh, public sources.

[5:59](https://www.youtube.com/watch?v=QluDzKVfp6A&t=359s) So going back to the main point that I wanna make here is that- Training is extremely costly. So if you constantly try to scale up the model size, you end up, uh, in, in this regime of like, um, if, if it's not giving performance gains, then can we get performance gains at-- uh, gains at inference time because

[6:17](https://www.youtube.com/watch?v=QluDzKVfp6A&t=377s) inference costs are so cheap? And a key idea that was, uh, extremely useful here was that if you could get the models to generate multiple responses and then do majority voting.

[6:32](https://www.youtube.com/watch?v=QluDzKVfp6A&t=392s) So, uh, in the example above, I'm showing that we get-- we-- the, the prompt doesn't make sense, but you've given a mathematical problem to a large language model, and you're asking it to generate three answers independently, and then you basically do some voting on top of those answers.

[6:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=410s) And if two answers match, then that's a majority vote. Or like, if in this room I were to ask a question and all of you said yes, then that is a majority vote.

[6:59](https://www.youtube.com/watch?v=QluDzKVfp6A&t=419s) So similarly, in large language models, if you can get the model to, like, generate many, many samples and then consistently get it to vo-- uh, like, get many of those answers to agree, this notion of majority voting or self-consistency had shown gains.

[7:12](https://www.youtube.com/watch?v=QluDzKVfp6A&t=432s) So this kind of scaling compute at inference time was clearly one avenue to go push on. Another avenue that emerged and showed substantial value was that you could sequentially revise your previous response.

[7:25](https://www.youtube.com/watch?v=QluDzKVfp6A&t=445s) So as humans, oftentimes we write the first answer, and then we go evaluate our answer, and we are like, "Oh, there's some mistake here. It doesn't quite match," and then you go fix it.

[7:35](https://www.youtube.com/watch?v=QluDzKVfp6A&t=455s) So basically, can we get LLMs to do the same kind of revision looking at previous set of revisions? And this was the second, so basically having longer, uh, chains of thought, uh, and getting the model to improve consistently in inference time based on that.

[7:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=470s) And these kind of techniques, uh, where you could verify your correct answer, so in math or in programming where you have unit tests, showed, uh, very clear gains. So what I'm showing you here is an example from, uh, uh, one of my colleague's work, um, at Stanford, uh, which is a publicly, uh, published, uh, paper.

[8:10](https://www.youtube.com/watch?v=QluDzKVfp6A&t=490s) And on the y-axis, we have pass@k or coverage score, and on the x-axis, we have, uh, number of samples. So as you basically are doing a lot of samples on the x-axis, your accuracy is improving, uh, with open source DeepSeek model and just taking more samples.

[8:25](https://www.youtube.com/watch?v=QluDzKVfp6A&t=505s) So you're getting a very high score on SWE-bench Verified compared to even state-of-the-art back in end of twenty twenty-four. Uh, of course, now all of these scores have pushed up, and we are roughly somewhere around eighty percent already.

[8:40](https://www.youtube.com/watch?v=QluDzKVfp6A&t=520s) But what we want to take away here is the fact that these lines of work, they showed that inference time compute predictably gives us gains, especially in domains where we can verify.

[8:54](https://www.youtube.com/watch?v=QluDzKVfp6A&t=534s) If we know how to verify the answers, then we actually know how to translate that into intelligence. And going back to my talk title, coding is one of those domains where we do have the capability to verify.

[9:07](https://www.youtube.com/watch?v=QluDzKVfp6A&t=547s) Um, and that gives us tremendous advantage in terms of building superintelligence on top of autonomous coding.

[9:16](https://www.youtube.com/watch?v=QluDzKVfp6A&t=556s) Of course, now you ask the question of what does automated verification mean here? So

[9:23](https://www.youtube.com/watch?v=QluDzKVfp6A&t=563s) for inference time scaling to work, you need basically some way to say this output is correct. Now in math, um, this is a very simple example. If you were to give the input to solve this mathematical equation, um, and if you were to do the same calculation on a calculator, you can actually verify that that problem is

[9:41](https://www.youtube.com/watch?v=QluDzKVfp6A&t=581s) correct or that solution is correct. Um, and similarly in math, you have formal proofs, so you can actually verify things are correct. Uh, in coding, you have unit tests.

[9:53](https://www.youtube.com/watch?v=QluDzKVfp6A&t=593s) In compilers, you can actually generate the code and then use PyTorch as a verifier-- uh, PyTorch, the compiler as a verifier. And in fact, uh, in domains where you don't have, uh, this kind of verification, then there's a large gap.

[10:07](https://www.youtube.com/watch?v=QluDzKVfp6A&t=607s) If you were to generate a lot of solutions and then do majority voting, you actually don't get as much gains. So what this roughly meant was that, okay, so inference time scaling would work in scenarios where I have automated verification, but that doesn't quite solve the problem for it to be-- have real world impact.

[10:26](https://www.youtube.com/watch?v=QluDzKVfp6A&t=626s) And the reason for that is shown in this graph as to typically, uh, if you do majority voting and these, uh, this is across multiple different models on GSM8K, which is middle school math problems and another math benchmark.

[10:38](https://www.youtube.com/watch?v=QluDzKVfp6A&t=638s) Uh, if you were to sort them by correct fraction, you have to sample a lot. The correct generations could be very rare. So who has time to sample ten thousand times and then get a correct solution?

[10:51](https://www.youtube.com/watch?v=QluDzKVfp6A&t=651s) You would be sitting there waiting just finding the correct solution unless you can actually figure out where the correct generation is.

[10:59](https://www.youtube.com/watch?v=QluDzKVfp6A&t=659s) So basically, scaling inference time compute with just majority voting or longer reasoning chains is great in the sense that there are some correct solutions somewhere there, but it doesn't work well across the board.

[11:11](https://www.youtube.com/watch?v=QluDzKVfp6A&t=671s) So what will get these models to learn to generate correctly, uh, during training? Well, in, in the chatbot application scenario, we saw that RL with human feedback did work.

[11:24](https://www.youtube.com/watch?v=QluDzKVfp6A&t=684s) So can we apply the same principle here and get the model to generate correctly in, uh, where, where we can automatically verify the outputs? So our belief at, uh, Reflection is that the next frontier for scaling is reinforcement learning, and we already have proof points from some of the frontier labs as well.

[11:43](https://www.youtube.com/watch?v=QluDzKVfp6A&t=703s) And as David Silver and, um, Sutton published recently, they agree with the-- or, or rather they, they are the pioneers in, uh, in reinforcement learning. They say that we are basically entering the era of experience, like starting from AlphaGo and AlphaZero, uh, where you had an era of simulation, and the next set of large language model era

[12:05](https://www.youtube.com/watch?v=QluDzKVfp6A&t=725s) was where you scaled up with RL using human data. But the next era from This year is really the era of experience, which was-- which will lead us to superintelligence.

[12:15](https://www.youtube.com/watch?v=QluDzKVfp6A&t=735s) So reinforcement learning will be a fundamental component in building, uh, superintelligent systems, uh, especially in areas where we have automated, uh, verification. And some, uh, proof point for why this makes sense is that in math, uh, over several papers, this is, uh, results from o1, but over several papers, we have already seen examples that if you

[12:39](https://www.youtube.com/watch?v=QluDzKVfp6A&t=759s) give the model on the right side, uh, test time compute, uh, on the y-axis, test time compute the same as inference time scaling, and you measure accuracy on the x-axis, it should go up.

[12:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=770s) Um, but as you can repeat this process, um, and with reinforcement learning, then the training time compute going up on x-axis also improves the accuracy on y-axis for a challenging benchmark in math called AIME.

[13:04](https://www.youtube.com/watch?v=QluDzKVfp6A&t=784s) Uh, most of these benchmarks saturate within a year, as you probably have learned by now. So, uh, this benchmark is already saturated. Um,

[13:14](https://www.youtube.com/watch?v=QluDzKVfp6A&t=794s) so now that I've hopefully convinced you that reinforcement learning and scaling reinforcement learning is, uh, the next frontier, you'd be like, "Okay, so why are-- why is not everyone doing it?

[13:25](https://www.youtube.com/watch?v=QluDzKVfp6A&t=805s) What's so challenging about it?" So as I have built large language models before, a big part of building, uh, these systems, uh, ends up being that the machine learning plus system stack for these, uh, systems themselves is very challenging.

[13:39](https://www.youtube.com/watch?v=QluDzKVfp6A&t=819s) So here is, um, an example of, uh, why scaling up reinforcement learning is challenging. So if you are trying to do reinforcement learning with, uh, PPO, which is, uh, one of the a-algorithms used for RL with human feedback, um, then it moved to, uh, DPO.

[13:56](https://www.youtube.com/watch?v=QluDzKVfp6A&t=836s) You have to keep four copies of, uh, different models. Uh, so if you imagine a really large model, and then you have to keep four copies, then you have to arrange them somewhere on GPUs in your large cluster.

[14:08](https://www.youtube.com/watch?v=QluDzKVfp6A&t=848s) You, you can have some fun figuring out the exact layout and, um, it's, it's, it's a fun and interesting problem, but it's a hard problem in the sense that, uh, to make maximum utilization of these systems and, and arranging them in the right way, just building that system is extremely hard.

[14:25](https://www.youtube.com/watch?v=QluDzKVfp6A&t=865s) And, uh, DeepSeek actually showed, uh, with DeepSeekMath, that GRPO, uh, gets rid of the value model and, and only has three copies of the model, but that doesn't-- that's still a very challenging problem.

[14:36](https://www.youtube.com/watch?v=QluDzKVfp6A&t=876s) So scaling up RL, uh, is more cha-- even more challenging, um, than scaling up, um, LLMs because you have multiple copies of the model and you have a training loop and an inference loop.

[14:48](https://www.youtube.com/watch?v=QluDzKVfp6A&t=888s) And then on the machine learning side, on the, on the reinforcement learning side, you also suffer a lot from reward hacking if your, uh, the model that is deciding that this is the correct answer is a neural reward model.

[15:00](https://www.youtube.com/watch?v=QluDzKVfp6A&t=900s) So you, uh, as we discussed before, in autonomous coding applications, you do have the ability to verify your output, uh, which roughly means that you can decide this is the correct answer or not.

[15:12](https://www.youtube.com/watch?v=QluDzKVfp6A&t=912s) Uh, that's how SWE-bench Verified scores, uh, work today. Um, you have execution feedback, you have unit tests. So all of these possibilities, of course, um, this is, uh, an ongoing list, all of these possibilities me-mean that you can design better reward functions.

[15:29](https://www.youtube.com/watch?v=QluDzKVfp6A&t=929s) Okay, so this means that autonomous coding is a great domain for scaling up RL. Then the question becomes: How does this have real-world impact? So in software engineering applications, generation of code is only one part of the system.

[15:44](https://www.youtube.com/watch?v=QluDzKVfp6A&t=944s) If you look at end-to-end workflows for software engineering, there is many more parts to that system. How do you scale up your system to generalize across all of those domains?

[15:53](https://www.youtube.com/watch?v=QluDzKVfp6A&t=953s) So that's the problem we are trying to solve at Reflection.

[15:57](https://www.youtube.com/watch?v=QluDzKVfp6A&t=957s) Our mission is that we would like to build some superintelligence, and we are starting with autonomous coding as the root node problem for this, um,

[16:06](https://www.youtube.com/watch?v=QluDzKVfp6A&t=966s) uh, mission. And, uh, we have a team of about thirty-five pioneers, um, who are-- who have pioneered various, uh, legendary works in LLMs and reinforcement learning. So if you're excited about this mission, uh, you can reach out to, um, one of us, um, or my, uh, my email is, uh, my last name at reflection[REDACTED:email_address], and we would

[16:27](https://www.youtube.com/watch?v=QluDzKVfp6A&t=987s) love to work with you. And with that, I can take questions. [clapping]

[16:37](https://www.youtube.com/watch?v=QluDzKVfp6A&t=997s) All right, um, same protocol as last time. If you have a question, please come up to one of these three microphones we have distributed throughout. We can probably take one or two questions.

[16:46](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1006s) So if you want to ask something, um, feel free. Um, I guess I can-- I, I'll do the first one while people are coming up. So I'm curious, um,

[16:55](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1015s) seems like the foundation models are trying to build one model and deploy it across everything. Do you have an opinion with the work you're doing right now if you think that's the right approach, or if you think there'll be more specialization on different languages or even, like, individual code bases?

[17:09](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1029s) Um, or do you feel like the best approach is just to have, like, one model that's trained across the, the greatest diversity of tasks possible?

[17:16](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1036s) Uh, I think I will answer your question, uh, in terms of building coding agents does require m-um, multiple capabilities, and how you get there, you will definitely need multiple LLM calls.

[17:26](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1046s) And then whether that's one model or multiple models, I think that's the secret sauce right now for most people.

[17:31](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1051s) Fair enough. All right. Please.

[17:34](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1054s) Hi. Um, I'm wondering in the slide with the chart of era of simulation, era of something, and era of experience-

[17:42](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1062s) Yeah

[17:42](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1062s) ... uh, they had put in AlphaGo and, um, the, uh, previous one where also you-- they played Star-StarCraft or something. They all used MCTS, uh, which, I mean, maybe it's my unfamiliarity with them, but it's also data sim-uh, simulation.

[18:00](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1080s) Mm-hmm.

[18:00](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1080s) Uh, so we're using synthetic data for era of experience as well. So how does-- why is that called simulation, and why is what we're doing right now not called simulation?

[18:11](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1091s) What's the sort of overlap between simulation, experience? How does that-- how do you think about that?

[18:17](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1097s) I can ask Dave that question, you know. But [laughs] going back to the point, I think, I think the better way to answer that question is, uh, roughly what Greg covered in the last talk, where his comment was that, um, so in gaming, you can envision what scenarios might happen next, and you're basically using that to build your

[18:35](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1115s) reinforcement learning. So you're doing rollouts, and you're, you're basically building, uh, based on that. Um, in, in real world, in most scenarios, you have an imperfect rollout, so you don't have full knowledge of how the system might work.

[18:50](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1130s) Um, simulation is possible in certain domains where you do build a world model, uh, which is closer to robotics and all the work that's happening in the physical AI space, right?

[19:03](https://www.youtube.com/watch?v=QluDzKVfp6A&t=1143s) But in the, in the real-world applications, which is what we're targeting, um, you will have imperfect things. So you have to actually experience the real world, and you have to collect some data, and that data is not gonna be in any way complete, nor will it completely search the exponential search space that could exist. [upbeat music]
