We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

Recursive self-improvement needs a test. In this recording, Elie Bakouch, a research engineer at Prime Intellect, starts with the prospect of models training models without human intervention. Large labs have described that prospect as approaching, but his concern is how researchers outside those labs can measure the capability for themselves.

The question reaches beyond AI training. If scientific research increasingly uses AI tools, researchers need to understand how those tools search, run experiments and respond to failure. A constrained training challenge offers a place to observe that behavior: an agent proposes a change, runs it and gets a measurable result.

The starting point is the GPT-2 training speedrun. The challenge treats reaching a target validation loss as its finish line: train a model until its loss matches the GPT-2 reference, then compare how long that took. Matching this loss supplies a useful benchmark proxy; it does not establish that every aspect of the resulting model’s behavior matches GPT-2.

The community’s modded-nanogpt effort pushed the reported training time from roughly 90 minutes through 45 minutes to less than two minutes over about two years. That history matters: the agents would enter a challenge that talented researchers had already spent substantial effort improving, rather than collect an easy win against an untouched baseline.

Two versions of the challenge expose different kinds of improvement:

• Training speedrun: Reach the target loss in the shortest time. The training and validation data stay fixed, while architecture and other implementation choices can change.
• Optimizer Speedrun: Change only optimizer-related parameters and methods. The comparison focuses on reducing the training steps needed to reach the target, making the search more about the learning method than making the program execute faster.

Speedruns also make a plausible training environment for research agents. A record improvement can receive a positive reward; a failed attempt can receive zero or a negative reward. Clear rules make proposals checkable, and short experiments make iteration practical. Bakouch describes optimizer runs taking about 15–20 minutes, long enough to test a training method but short enough to try many candidates.

The initial experiment put Claude Code and Codex on Prime Intellect’s cluster and let them iterate against the community. Runs labeled V1, V2 and V3 represented stopping and restarting the agents. Near the end, after newer human submissions had overtaken their results, the agents were explicitly asked to take the recent community records and improve them. A separate novelty track asked for improvements using novel ideas.

The harness was deliberately simple. A goal.md file supplied the objective, and an AGENTS.md file supplied rules. The agent proposed an experiment and submitted it with sbatch to a Slurm cluster. Jobs used available nodes under preemptible permissions: if another user needed a node, the experiment could be canceled. This let the search use spare compute while yielding capacity to other work.

After training, the agent read the logs and decided whether the candidate qualified as a record. Acceptance required passing a statistical threshold so that random variation would not masquerade as an optimizer improvement. How does an idea become evidence in this setup? The flow below separates proposing a change, obtaining compute and checking the result; writing a promising method is only the first step.

The first striking difference was whether the agents kept working. Claude Code stopped every nine or ten hours, concluding that it could not improve the record. Bakouch prompted it to continue exploring, after which it worked for another stretch and stopped again. Waiting for those interventions left it idle for roughly one-third of the time. Codex, by contrast, stayed active almost continuously and rarely asked questions.

The agents also had a scratchpad: files serving as active memory for the research process. Their behavior differed even after normalizing activity by active hours, so the differences were not merely a consequence of Codex running longer.

• Memory writing: Codex wrote substantially more scratchpad material. Its notes recorded actions, decisions and next steps; Claude’s notes celebrated new records with a more excited tone and emojis.
• Delegation: Codex spawned more subagents, distributing more work across agent processes.
• Token use and compaction: Codex consumed more tokens in this initial experiment and compacted its context more frequently. The large total included cached input tokens, so it should not be read as an equivalent quantity of newly generated output.

Bakouch reports that both agents surpassed the human record. The reported gains were roughly tens of training steps against a baseline of roughly 2,990 steps; the captions leave the precise gains and their assignment to each agent uncertain. These are reported experimental results rather than an independently verified benchmark, and the comparison allowed agents to fetch new human records throughout the run. It therefore measures their ability to extend available research as well as their own search.

Claude’s restart provides a concrete example of that distinction. The community advanced the record while the experiment was running. Once restarted, Claude fetched the newer human result, used it as the starting point and improved it. The observable change was a better record after incorporating a better baseline. Access to current research altered the search’s starting point; the agent did not need to recreate all the progress that produced that baseline.

A more controlled benchmark was still under development. It would use multiple seeds and comparable conditions for models and harnesses, then separate three sources of research capability:

• No external access: Work from knowledge stored in the model’s weights.
• Paper access: Consult research papers, without receiving the full stream of community solutions.
• Full access: Use external information, including the latest human records.

The plan covered both the original training speedrun and an optimizer challenge requiring novelty. Separating access conditions would make it easier to distinguish recalling a method, finding one in the literature and improving a known submission.

A longer comparison added Kimi and GLM alongside Claude and Codex. The run lasted roughly five to six days; GLM was still iterating when the presentation was given, so its trajectory was incomplete. Claude improved progressively, while Kimi showed a more abrupt change: around day four, it found a record that overtook Codex.

Changing the horizontal axis from elapsed time to output tokens changed the comparison. Kimi looked particularly efficient for the number of tokens it generated. That answers a different practical question: how much generated work was needed to reach an improvement? An agent’s position after several days and its progress per output token can favor different choices; token efficiency alone does not describe the full compute cost of its experiments.

Literature search supplied another route to progress. Claude found a paper that the other models did not find, and that paper led to the best record in the comparison. Finding the right existing idea was consequential research work: it changed which method the agent could try and ultimately changed the result.

But the novelty track exposed the central disappointment. Bakouch had expected unfamiliar optimizer ideas. Instead, the agents combined papers, used clever tricks and made incremental improvements to existing methods. Across these experiments, they produced no novel optimizer or mechanism. A breakthrough in a record curve can therefore mean a successful combination or application of known work, without constituting a new learning method.

The proposed next step changes the system around the agents. Inspired by AlphaEvolve, the design uses multiple generators—including closed models and cost-effective open models—to suggest ideas. Speedrun execution supplies a measurable reward, while a judge supplies qualitative feedback about the method. Humans can judge proposals and steer the search toward promising directions.

Which small-scale gains deserve a larger experiment? The proposed loop makes that selection explicit. Promising methods move to training with more parameters and more tokens, because a speedrun success may fail to carry over to large-scale training. Numerical reward identifies a local improvement; qualitative judgment helps select what to pursue; larger experiments test whether the improvement travels.

This was a discovery design being tried, with new discoveries still a hoped-for outcome. Another way to direct the search is to change the speedrun itself: different objectives and constraints create different search spaces. Rather than ask every generator the same broad question, a family of challenges can push exploration toward different kinds of methods.

Prime Intellect’s closing roadmap addresses the infrastructure needed to run this kind of search. Much of it was still unreleased: GPU sandboxes for experiments, agents that use a filesystem to write and retrieve information, programmatic tool use, and training open models to work well in that environment. The company also described already released tools for training and evaluating models across environments and agent harnesses.

The final motivation returns to participation. Researchers outside large labs need a way to understand how models do research and to take part in improving the process. Open experiments can expose the useful work—persistent iteration, literature search and tested combinations—alongside the point where the agents still fall short: inventing a new optimizer. That visibility is the foundation for deciding what to build next.