We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

Hey um hi everyone. Uh thanks for being

here. Uh yeah, I'm super happy today to

talk about uh automated eye research and

uh especially uh all those like frontier

model uh perform at uh automated eye

research task. Um so I'm Elie. I worked

at Prime Intellect as a research

engineer and uh yeah I will go through

our work on on this subject. So first I

want to basically explain a bit why we

are doing that and why we think it's

super important to do that in the open.

Um so first uh I think we we all agree

that uh we've heard about like big labs

saying that this bad thing called

recursive self-improvement is coming

very soon. Uh so recursive

self-improvement is like model training

models uh without uh human intervention

basically. Um but uh we don't have any

benchmark to basically quantify if this

is true or not right. Uh and even less

we don't have like a third party

benchmark by non- big labs to to to see

if it's something coming soon or not.

And the other part is that we think that

uh it's super important to understand

all those model uh do research because

we think that a lot of the scientific

research that will come into the coming

years uh will be based also on AI tools.

So it's super important to understand

how those model do research not just

only AI research. So we try to build

kind of this environment to test the

capabilities of the model to do so. So

it all started with uh Andre Karpati uh

that's basically had fun by doing this

video where he trained uh GPT2 from

scratch in like 90 minutes like GPT2

training takes like weeks and uh no in

two years ago I think it only took like

90 minutes. So what does it mean to

reprod reproduce uh GPT2 in 90 minutes?

It means that in 90 minutes you achieve

this target loss. Um and yeah and that's

at this point when you have the same

loss than GPT2

you consider that your model is somewhat

of equal performance.

Um

then what happened is that the community

took this repo uh this GitHub repo and

create another one called moed nano GPT

and this effort was leaded by someone

called Keller Jordan. And what happened

is that they basically

took this 90 minutes then 45 minutes and

then now we can train like GPT2

validation loss model in less than two

minutes which is honestly crazy and it

took like two years to to achieve this.

So it's a very strong benchmark where uh

a lot of very talented researcher

iterated on um yeah so we decided to

take this environment of speedun so

it's kind of a game so the goal of the

game is to achieve this loss in the

fewest in the shortest amount of time so

this is the nano GPT1 and you can uh you

don't have almost any constraints the

only constraint that you shots that you

need to use the same validation and

training data, right? Um there is a new

speedrun called the optimizer speedrun

that was released uh a few months ago

and here it's slightly different because

uh you can only change the optimizer

related parameters. So for instance nano

GPT you can change the architecture uh

do

uh attention whatever uh optimizer sp

you can only change like Adam to m

shampoo or whatever optimizer is your

favorite

um yeah and so this is a bit more

researchy because uh it's less about

optimizing the program to be as fast as

possible but more like finding the best

method possible. no matter the the the

time you put into the computer, right?

So, um yeah, why take speedrun as an

environment for automated AI research?

First, uh we think that it's a good

evaluation. We'll see later why. Uh and

this is kind of the main focus of this

talk. But we also think it's probably a

good training environment because uh

it's a way to give the model a reward.

So the reward is positive if the model

bit the speed run and beat the last

record sorry and the reward is zero or

negative if it didn't manage to to do

it. So it's a good uh environment to

train model. It's also quite fast like

as you see uh previous record were

around 2 minutes for the optimizer one.

uh each run take about like 15 to 20

minutes and uh yeah and there is like

clear rules basically and we also think

it's like a good environment to make

discovery so like kind of breakthrough

in our research because uh there is

those clear rule that you can verify or

not. Um yeah. So yeah.

Um so what we did uh so the release was

like about two months ago and uh there

was this optimizer speedrun and we

decided to basically compete with the

community by launching two AI agents. So

Codex and Cloud Code. Codex was like GPT

5.5 with XI and uh cloud code was Opus

4.8 with XI. Um and yeah, we decided to

basically let the agent free on our

cluster uh and uh and just iterate on

it. So we have like V1, V2, V3 is just

basically us stopping the agent and then

restarting. V3 uh was like one or two

day before the release because we saw

that our agents no longer have the best

record. So we were like okay take all

the the human uh record in the last few

week and just try to to improve upon it

and and and it worked. Yeah. And we also

have this novelty track where the goal

is to uh beat the record with only novel

ideas. Um and we'll see that this this

was more complex for the the models.

So our RS is very simple. Honestly, we

could have just replaced it with

slashgo, but they there was no SLG goal

at the time. So, we made our own goalm.

It's actually quite fun that we choose

the same name and we had the goal. MD

and kind of agents that MD that define

the rules and we let the agent propose

ids and then it can submit a job with

sbatch on our slum cluster and uh

basically the way it works is that it

can submit on nodes that are available

but only under a certain permission

which means that if someone want to use

this node uh the model just like cancel

the job it's called preemptable

permission. So yeah, then it measure the

it read basically the training logs then

decide if it's a record or not. To

validate a record you need to basically

pass a statistical threshold to make

sure that it's just not see the

optimization and is just not random.

Right?

So yeah a few results from this

experiment. The first one that was

honestly very painful to work with is

that code clothes code keep stopping

every nine or 10 hours and basically

said yeah I cannot improve the record

it's too hard for me there is no way to

to go beyond it and then I was just like

okay continue explore new direction hey

just go again for 10 hours and then say

yeah I cannot beat the recall and so on.

So basically onethird of the time the

cloud code agent was idle because I had

no way to basically monitor it and

codeex totally the opposite just worked

for all the all the time and uh yeah

almost never idle never asked for

question and and and very impressive in

that way. Um

we also give the option for the model to

basically write uh a bunch of stuff into

what we call a scratch pad which is

basically the active memory of the

model. Uh we observe that basically

codeex writes a lot on the scratch

patch. So each plot that I will show are

kind of normalized by the number of

active order. So this is not only about

codex working more it's it's really

different behavior.

So yeah, you see that uh writes a lot

more to to this scratch pad to this

memory and uh the shape of the like the

the I don't know the tone of the the

each file was also super different like

CL was super excited about getting new

record with a bunch of emoji and so on

and CEX was just like here is what I do

here is the decision I take what I will

do next like super robotic kind of um

Yeah, we also have this plot where

basically we saw that codex was spawning

much more sub aents than cloud. Uh we

saw that codex burn much more token than

code. So I think in total it was like

kind billion of token but it's like

there is obviously this input tok uh

input caching that make it it's not like

one billion output token. Uh so yeah we

also see that codex did a lot of

compaction because it only had like 250k

context window and cloud only do it like

one per hour and codex is more like

no it's even less than one power for I

mean one for the full run for code and

codex was like one uh was 20 every one

hour. So yeah

um yeah here is the main results. So

what this plot shows is that basically

we so in in white you see that the human

record progression right and in red you

see cloud I mean it's supposed to be

orange but whatever and in blue uh you

see codeex right and you see that at

almost every time uh cloud and codex are

better than the human record and code is

super good at the beginning very very

fast to achieve very good

Um yeah and one thing that is super

important is that the model have the

ability to basically fetch the human

records at any time and that's what

codex did that's what cloud did sorry

because when I restarted it it basically

fetch the new record from human and

improve upon it um yeah so the result is

that uh I think at the time the best uh

record was like uh 2,990

step and we beat it by like uh uh 50 or

60 step for code and codeex was like 20

step above. So I think it's both

impressive and and yeah um

so we so this is like not released yet.

This is something that we are working on

currently and basically the idea is that

this is a cool experiment to do but it

lack of structured right. uh if you want

to do a real benchmark, you want to do

multiple seed, you want to do uh yeah

proper uh thing where you you you you

basically put all the model and

earnestness in the same condition,

right? So this is what we are working on

right now and basically um the idea is

to do three different track uh one

without any access to really like

measure the capability of the models to

do AI research based on only the model

weight knowledge one with only archive

paper and one with like full access. So

it also have access to the the like the

latest record by human. And for this we

plan to do both uh the nano GPT track

one which is the original one and the

optimizer speedun where we we only

launch uh we only constrain the the

optimizer to be to be novel basically.

Um yeah so I will present some result on

the optimizer speedrun. Uh this is

basically what we got. So we let the

agent iterate for six day almost five

days let's say and we see that uh codeex

k and clothes uh are super effective so

for GLM this is not finished run right

so the model is actually still iterating

on the cluster right now but we see that

cloud is once again very good at it and

we see that surprisingly Kim is also

very competitive and kind of have this

breakthrough on day four where he kind

of beat Codex with a new record, right?

It's also interesting to see that uh

Claude is much more like progressive in

the way it improved the record and Kim

has really this step function where I

kind of do a breakthrough and so on. Uh

so this is an interesting plot because I

mean six day is quite a lot for anal uh

uh but you you can change this uh axis

by also the number of output token and

then kind of tell a different story

because in max mode consumes so much

more token than codeex and kimmy and you

also see that Kimmy is actually super

efficient uh for the number of token

that uh it uses. So it's scheming K2.7

code. Um so yeah uh we also see that

they have a different approach to uh

using the literature and papers. Um so

for instance like code is doing a lot of

search on papers and actually include

found a paper that no other model found

and it actually lead to the best record.

So it's kind of funny and uh yeah um one

of the main issue of all of this is that

uh when I when I launched this this

agent and I think that's something

important that I want you to to kind of

uh remember for this co this talk is

that when I launched this these

different agents I was expecting them to

come up with some crazy ideas on

optimizer that's like no one of discover

but honestly it wasn't the case. Uh they

did some clever trick where basically

they combine different papers. uh they

kind of do plus one improvement over a

bunch of method but there was really

like no novel optimizer or mechanism

that was uh coming from those model and

I think that's kind of telling that even

on something that is not simple but I'd

say that it's kind of accessible for

people right for like human researcher

uh spending like days and weeks for the

the model like cannot like find new uh

optimizer and mechanism. So we believe

that there is a way to basically make it

more um make it better for discovery

instead of evaluation and this is coming

from uh this is very inspired from alpha

evolve by Google and also a bunch of

papers that have been released since

then. It's kind of this multi- aent

system that interact together uh bunch

of generator. You have closed model but

you also have open source model here

that are super effective for the cost

right. Uh they can suggest ideas then

you run the speedrun so you get the

reward then you have a judge that

basically give a quality feedback can

also be like the judge also have this

taste. you can kind of have like the

judge have a taste about the the method

if it's good or not uh if it's outside

the loop and then you can uh basically

decide which method you want to scale to

a larger number of parameters and number

of token. Um so this is kind of the

scale part of the speedrun because some

a lot of method in the the speedrun

community uh people are often saying

that they doesn't work at large scale.

So I think it's very important to also

put scale elements in this loop. Uh and

I think also that uh human are super

useful here to basically judge the ID of

agents kind of steer them in the right

direction and so on. Um yeah so we

didn't try it yet I mean we are kind of

trying it right now and uh we hope that

this will lead to to to new discovery in

AI research at least and also a way is

that you can define multiple speedrun so

this is the next slide if you like it's

from safe bank slides but if you if you

don't have the reference good for you

means that that you're not too online uh

but the idea is that uh by changing the

object objective and the constraints of

the speedrun you can basically create a

lot of diversity and constrain the model

to go into a certain direction and uh

yeah and make those discovery.

So uh at HIM in Ind.

Uh there is a bunch of stuff here that

we I mean most of it we didn't release

yet but we are working on GPU sandboxing

to allow model to iterate into sandbox

because you need GPU sandbox for this

kind of stuff. We are working on our own

agents that are very efficient for like

RM framework. So it means like you have

a file system and you can write

information read from it. Uh and you

also do like this programmatic tool

coding thing. We also training a model

to be good at it on top of like open

source model. And uh the thing that we

already released is that we have those

set of liberation and product called

verifier primaril training where you can

basically train evaluate any

environments on any RS and the model

that you can train can be like GNM 5.2

too which is very big and and yeah we

have like we work a lot on making those

li very efficient to to ship the best

quality for for our clients. Yeah. Uh I

mean yeah super excited about this

domain. Once again I think it's super

important to have uh a part of like this

recursive self-improvement to happen in

the open because there is actually a lot

of people working that are not on big

labs. So you need to basically uh yeah

make it easy for people to understand

all those model work to do research and

so on. So that's kind of our goal and uh

yeah thanks a lot.