RL for Autonomous Coding — Aakanksha Chowdhery, Reflection AI

Why would a model get an arithmetic problem wrong when asked for an answer, yet get it right when prompted with worked reasoning? The opening tennis-ball example makes that question concrete: the standard prompt illustrates an answer without intermediate steps; the chain-of-thought prompt illustrates how to work through the calculation.

The path to that example begins with scale. Aakanksha Chowdhery introduces the problem from her experience leading research on PaLM and working as a lead researcher on Gemini, before turning to reinforcement learning for autonomous coding. The 2020 scaling-laws paper established empirical power-law relationships between language-model test loss and model size, training data, and compute. Increasing these resources made model performance more predictable. Chowdhery connects that progression to broader generalization, improving benchmarks, and capabilities that were absent in smaller models.

The arithmetic demonstration shows what one such capability looks like. After an example involving tennis balls, a subsequent problem receives an incorrect answer under standard prompting. With intermediate reasoning demonstrated in the prompt, the illustrated answer is correct. The prompt changes how the model uses its capacity. This is an example of improved performance, not a guarantee that a worked answer will be right. The public chain-of-thought paper and PaLM paper appeared in 2022; Chowdhery’s recollection of 2021 should not be read as their publication date.

The accompanying comparison plots model parameter count against solve rate on middle-school math word problems for LaMDA and PaLM. PaLM reached 540 billion parameters. The notable improvement appears primarily when the larger models receive chain-of-thought prompts, connecting model scale to the usefulness of the prompting technique.

That observation encouraged step-by-step instructions and many informal variations in how people asked models to reason. The broader attraction was that the gains were not confined to arithmetic: Chowdhery describes improvements across multilingual question answering, puzzles, and multitask natural-language understanding.

Reasoning capabilities were followed by increasingly useful instruction-following systems. ChatGPT, Gemini, and other chatbots could respond to requests rather than merely continue text. In Chowdhery’s account, reinforcement learning from human feedback, or RLHF, was a central ingredient in making those interactions useful.

The feedback starts with a comparison: give people a question and two candidate responses, then ask which response they prefer. Accumulating those judgments supplies a training signal that pushes the model toward preferred behavior. The same idea applies to code. The slide pairs the preference-learning loop with a Python task chart showing improvements from RLHF, extending the mechanism beyond conversational answers.

As questions emerged about diminishing returns from pretraining, another resource became attractive: compute spent after training, while solving a particular task. Chowdhery cites public estimates putting large-model training costs in the tens of millions of dollars, while describing individual inference calls as comparatively cheap. These are illustrative public estimates, not figures endorsed by her former employers.

That asymmetry suggests generating more than one answer. In the simplest example, the model independently produces three answers to the same mathematical problem. If two agree, their answer wins the majority vote. Across many samples, this becomes self-consistency: sample different reasoning paths and aggregate their final answers. Agreement provides a selection rule, although it does not independently establish correctness.

A second approach spends compute sequentially. The model produces an initial response, evaluates it, identifies a mistake, and revises it using the preceding attempts as context.

These strategies explore different opportunities: sampling searches across alternatives, while revision tries to improve a particular trajectory. Longer reasoning is useful only insofar as the extra work produces a better answer.

Repeated attempts are especially promising when an answer can be checked. Chowdhery presents a coding experiment using an open-source DeepSeek model, with the number of samples on the horizontal axis and pass@k, or coverage, on the vertical axis. More attempts increase the fraction of tasks for which at least one successful solution exists among the candidates.

Coverage across attempts is different from reliably selecting one correct answer. The talk describes the result as SWE-bench Verified and compares it with late-2024 frontier performance, but the benchmark identity of the displayed experiment is not established here. Its curve should therefore be read as evidence for the sampling mechanism, not as a precise Verified leaderboard comparison. Chowdhery separately puts then-current SWE-bench Verified scores at roughly 80%, without specifying a model, scaffold, or sample budget. The transferable point is that verification can make additional inference compute useful: coding supplies ways to distinguish successful attempts from failures.

What counts as verification depends on the task. An arithmetic result can be checked by performing the calculation independently. Formal proofs offer a different kind of check, and code can be evaluated with unit tests. Chowdhery also points to compiler- and PyTorch-based checking for generated code, without developing a particular implementation.

A small Python example makes the distinction between agreement and checking concrete. Suppose the task is to implement square. These candidates are checked against expected results rather than selected by how often their text or outputs resemble one another:

from collections.abc import Callable


def square_a(x: int) -> int:
    return x * 2


def square_b(x: int) -> int:
    return x * x


def verify(candidate: Callable[[int], int]) -> bool:
    cases = [(0, 0), (2, 4), (3, 9), (-3, 9)]
    return all(candidate(x) == expected for x, expected in cases)


candidates = {"a": square_a, "b": square_b}
accepted = [name for name, candidate in candidates.items() if verify(candidate)]
print(accepted)

The case 2 → 4 alone would accept both implementations; 3 → 9 distinguishes them. Passing this finite test set establishes only that the candidate satisfies these checks. It does not prove correctness for every possible input. This is the practical value—and boundary—of execution feedback.

Without an effective verifier, generating many answers and voting over them yields weaker gains. Even with one, search may be expensive because correct generations can be rare. The math analysis in the talk sorts problems by their fraction of correct generations, using GSM8K and another math benchmark. Some problems sit in a region where a successful sample is difficult to obtain.

Chowdhery’s example of waiting for 10,000 samples expresses that latency problem, not a universal sampling requirement. More attempts may establish that a model can sometimes solve a task, while still leaving the user waiting too long. Majority voting and longer reasoning chains therefore do not, by themselves, make correctness reliable across tasks.

The next step is to move some of the burden from inference into training. If human preferences can teach a chatbot which responses to produce, automatically checked outcomes can teach a coding model which attempts succeed. Reinforcement learning aims to make useful behavior more likely, rather than repeatedly searching for a rare success. Reflection identifies this as its next scaling frontier.

Chowdhery places that direction within David Silver and Richard Sutton’s era-of-experience framing: game systems such as AlphaGo and AlphaZero learned through simulation; language models subsequently drew heavily on human data; future systems would learn more from their own interactions. The forecast that this progression leads to superintelligence is an ambition, not a result demonstrated by the talk.

The supporting example is o1 on AIME mathematics problems. Its results show two distinct opportunities to improve accuracy: spend more compute at test time, and spend more compute on reinforcement learning during training. The paired plots keep those two axes of investment separate.

Chowdhery describes AIME as already saturated at the time of the talk. That is her assessment of benchmark progress, not an unconditional claim that every AIME evaluation setting has been solved. The evidence relevant to the mechanism is the upward relationship between compute and accuracy under the reported conditions.

If reinforcement learning offers this opportunity, why is scaling it difficult? A large part of the answer lies in the combined machine-learning and systems stack. A PPO-style RLHF setup involves four functional model roles:

Chowdhery describes these as four copies to place across a GPU cluster. The useful interpretation is four roles, not a universal requirement for four identical physical models. Their placement must sustain utilization while supporting the work each role performs. Direct Preference Optimization, mentioned briefly in the same discussion, is a separate preference-training approach; the four-role description belongs to the PPO-style setup.

DeepSeekMath introduces Group Relative Policy Optimization, or GRPO, which removes the separate learned value model and derives its baseline from rewards across a group of outputs. In the talk’s accounting, that reduces four model roles to three.

Removing a role helps, but it does not remove the fundamental scheduling problem. RL combines an inference loop that generates experience with a training loop that updates the policy. Both require substantial compute, and the overall system has to keep them working efficiently together. Scaling this arrangement introduces difficulties beyond scaling a conventional language-model training run.

The learning problem is also vulnerable to reward hacking. A policy can discover responses that score well under a neural reward model without accomplishing the intended task. Coding offers additional signals: execute the output, inspect the feedback, and run unit tests. Those observations can support more useful reward functions than a learned judgment alone.

SWE-bench Verified is the concrete benchmark example. Its tests check candidate fixes against expected behavior; “Verified” refers to human screening of the benchmark tasks, not a certification that arbitrary generated software is correct. Execution feedback improves reward design, but its usefulness still depends on what the checks actually cover.

That makes autonomous coding a promising domain for RL, but code generation covers only part of software engineering. Real-world impact requires a system to generalize across the other parts of an end-to-end workflow. Producing a patch is one capability within that larger job.

Reflection treats autonomous coding as the starting problem for its broader superintelligence mission. At the time of the talk, Chowdhery describes a team of about 35 people with backgrounds in language models and reinforcement learning, and invites people interested in that mission to join. The remaining technical challenge is broader than improving a code-generation score: the learning system must support the range of capabilities that engineering work demands.

The first audience question asks how those capabilities should be packaged. Should one foundation model serve the widest possible range of tasks, or should coding systems specialize by programming language or even by individual codebase?

Chowdhery answers at the level of the agent: coding requires multiple capabilities and multiple LLM calls. She does not disclose whether those calls use one model or several distinct models. That distinction matters. A workflow with several model invocations does not, by itself, imply an architecture with specialized models; the model-composition choice remains open in her answer.

The final question challenges the boundary between simulation and experience. The audience member invokes game-playing systems, Monte Carlo tree search, and synthetic data: if both eras involve generating trajectories, what makes the newer one different?

Chowdhery’s answer turns on knowledge of the environment. In games, a system can anticipate subsequent scenarios and build reinforcement learning around rollouts. Real-world rollouts are imperfect: the agent does not have full knowledge of how the surrounding system will behave. World models can enable simulation in domains such as robotics and physical AI, but they do not make all real-world applications fully predictable.

For the applications Reflection is targeting, an agent must interact with the real world and collect data from what happens. That experience will remain incomplete. It will neither describe the environment fully nor exhaust the exponential space of possible situations. Autonomous coding therefore needs more than a mechanism for generating and scoring answers: it needs to learn from partial experience in systems whose behavior it cannot completely simulate.