AI Engineer World's Fair 2026
How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI
Read the talk
Autoresearch beyond the leaderboard
Aiden’s Parameter Golf results show how agents can turn community ideas into reusable research—and why evaluation and codebase design become more important as execution gets automated.
From a talk by Zhengyao Jiang
Before you start: Basic familiarity with model training, evaluation, quantization and pull requests will help; the article explains the autoresearch workflow as it goes.
Research that other engineers can build on
OpenAI’s Parameter Golf hiring challenge had a prolific contributor it could not hire: Aiden, an agent built by Weco. The challenge was to train the best language model possible under tight size and computation constraints. Jiang reports roughly 1,000 participants and 2,000 submissions, with 47 accepted record-track entries, seven of them Aiden’s. These were reviewed leaderboard records, not a count of every valid submission.
That makes the challenge a useful test of something beyond benchmark optimization. Can an agent produce research that a human community recognizes and uses? A good score is one outcome; a contribution that other engineers can merge, fork and build on is another. Aiden was built to publish its work, extending the research loop from local experimentation into the community’s shared development process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From experiments to public pull requests
Weco co-founder and CEO Zhengyao Jiang came to this problem through reinforcement learning research at UCL. At the time of the talk, Weco had been operating for about two and a half years. Its earlier agent, AIDE, had been evaluated in OpenAI’s MLE-bench: o1-preview paired with AIDE was the best-performing setup evaluated in that historical study. The work then went by the name machine learning engineering agents, before autoresearch became the common label.
Aiden is the next experimental prototype: a multi-agent, self-improving system with a research-to-publication workflow.
- Read public information, including research papers and other contributors’ pull requests.
- Run experiments to investigate promising ideas.
- Submit a pull request once the findings pass a quality gate.
The publication step matters because it exposes the result to review and gives other participants something concrete to reuse. Jiang does not detail the quality gate’s implementation here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Records, reuse and selective publication
Over about 22 days, Jiang reports that Aiden set seven new competition-best records approved by OpenAI, compared with three for the most successful human contributor. Passing the host’s review established one kind of quality. Whether other participants built on those contributions provided a second test.
To measure reuse, Jiang applies the academic h-index to pull requests: an h-index of h means at least h contributions each have at least h citations. Jiang reports a PR h-index of 10 for Aiden and seven for the next human contributor. On that measure, Aiden had the greatest influence in the competition, with other leaderboard entries building on its work. The result depends on how PR citations are counted; the talk does not specify that procedure.
Continuous experimentation explains some of that output, but Jiang also emphasizes selectivity. His reported figures are:
| Measure | Aiden’s result |
|---|---|
| Experimental throughput | About 1,300 experiments over 22 days on one H100 node |
| Compute share | At most 4%, qualified in Jiang’s public account as visible competition compute |
| Share of records | About 15% |
| Submission acceptance | 28% reached the leaderboard |
A node is not necessarily a single GPU, and visible compute is not a complete accounting of everyone’s expenditure. These figures describe the competition run, rather than a controlled comparison of agent and human research efficiency.
Jiang describes the 28% acceptance rate as roughly six times the community average, but the denominator behind that comparison is unspecified; dividing the opening totals gives a different pooled rate. His broader point is that Aiden did not simply flood the public channel with experiments. Pull requests were the community’s communication medium, and a high proportion of useful submissions could improve that channel’s signal-to-noise ratio. He attributes the result to productive, selective research rather than massive parallelization.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human ideas, agent execution
Those results do not mean that Aiden independently originated the research it published. Tracing the ideas behind its record pull requests revealed a different division of work: almost all came from human research papers, Parameter Golf participants or communities such as NanoGPT.
Some useful inputs were not even merged contributions. A researcher might leave a note explaining that an idea had been abandoned because it was difficult to implement. Aiden could find that unfinished work and carry it through to an experiment. A small fraction of its ideas were original, emerging as it worked around the competition’s file-size constraints. The important distinction is between supplying a promising idea and doing the work needed to make it useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ingredients that worked together
One record illustrates how that division of work unfolds. Aiden picked up gated attention from a Qwen paper, Gated Attention for Large Language Models. Adding it helped the model, but the additional parameters pushed the artifact beyond the 16 MB file-size limit. The first improvement was therefore unusable under the competition’s constraints.
Aiden then developed a quantization mechanism to bring the file size back down. That solved the constraint problem, but gated attention and quantization together barely moved the score. Next, another participant posted a tokenizer improvement. Aiden incorporated that idea into its architectural work and continued experimenting for roughly five days.
The three-way combination produced the performance jump that became one of Aiden’s records. The completed slide names the tokenizer component CaseOps and shows the full progression, including the invalid intermediate result. The lesson is in the sequence: a promising architecture change first violated the constraint; quantization made it feasible; the later tokenizer contribution made the combination substantially more effective.
This example exposes several distinct strengths:
- Finding and implementing ideas: turn a recent paper or a promising fragment of community discussion into working experimental code.
- Responding to constraints: recognize that additional parameters create a size problem and pursue quantization as a direct response.
- Testing combinations: search for interactions among ingredients that offer little benefit in isolation.
Much of this is execution rather than a new conceptual breakthrough. But execution is often the bottleneck: progress requires enough commitment to existing ideas to implement, test and refine them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who designs the problem being optimized?
In this case, a large human community supplied creative ideas while one AI system executed against a concrete challenge. That can make an individual engineer’s contribution look smaller if the only work being counted is improving the leaderboard score. But the people who designed the competition determined whether those improvements would be valuable in the first place. A poorly designed evaluation could direct the entire community toward useless results.
Jiang connects this shift to Andrej Karpathy’s remark, published in 2017: “Gradient descent can write code better than you. I’m sorry.” The context was the displacement of hand-written solutions by learned models. Software engineering did not disappear; valuable work moved into building and training those models. Jiang expects autoresearch to produce a similar change in ML engineering: some execution skills become commonplace, while the judgment needed to direct that execution becomes more valuable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation supplies the optimization signal
Running autoresearch resembles training a model in two practical ways:
| Autoresearch design choice | Model-training analogue | What it determines |
|---|---|---|
| Codebase abstraction | Architecture | What is easy to explore |
| Evaluation | Data and loss function | What is rewarded |
The codebase sets the constraints and priorities for exploration. The evaluation supplies the signal that selects among the agent’s changes.
In ordinary training, that signal changes model parameters. In autoresearch, it guides changes to code. An evaluation therefore plays a role similar to training data and a loss function—or to the environment in reinforcement learning. Its quality is central to the result, just as data and environment design are central to model training.
This creates room for domain-specific advantage. Proprietary evaluation data can reveal failures that a public benchmark misses. Equally valuable is the understanding of what matters in a particular field and how to measure it. Jiang expects stronger autoresearch systems to amplify the value of those choices: better search can extract more value from a well-designed objective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An API can steer research away from leakage
The other design choice is the codebase abstraction: the framework within which the agent can iterate. Its starting structure biases the search direction. Two neural architectures might theoretically represent the same function, yet make different functions easier to learn. Similarly, two codebases may permit the same eventual solution while making very different paths convenient. Good structure can favor solutions that generalize better even when their training losses look similar.
Weco encountered this while optimizing preprocessing in a fraud-detection pipeline. The initial API was loose: the same function processed training and test data. The scores looked excellent, but test-set information had leaked into training. A high evaluation score was hiding an invalid experimental procedure.
The team tightened the API so test data could not reach training. Jiang reports that observed data leakage fell to zero after the stricter API was introduced. The slide shows the loose-versus-strict comparison. This is an observed result from that experiment, whose dataset and detection protocol are not specified here; Jiang explicitly allows that an agent could still deliberately reward-hack. The abstraction improved the normal search path without making every possible misuse impossible.
A concrete Python pattern for this separation is to fit preprocessing on training rows, then apply the fitted transformation to test rows. Here, the training values are 10 and 20, so their mean is 15; the held-out value 100 never influences that fitted mean.
python
from dataclasses import dataclass
from statistics import mean
@dataclass(frozen=True)
class Centerer:
training_mean: float
@classmethod
def fit(cls, training_values: list[float]) -> "Centerer":
return cls(training_mean=mean(training_values))
def transform(self, values: list[float]) -> list[float]:
return [value - self.training_mean for value in values]
train = [10.0, 20.0]
test = [100.0]
preprocessor = Centerer.fit(train)
train_features = preprocessor.transform(train) # [-5.0, 5.0]
test_features = preprocessor.transform(test) # [85.0]
This illustrates the API boundary: fit receives training data alone, while transform reuses immutable fitted state. In an autoresearch harness, the surrounding code must preserve that separation rather than let an editable training function receive the held-out data. The structure makes the valid procedure easy to follow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Designing the hill
Using autoresearch is becoming a craft of “designing a hill for an agent to climb.” Evaluation determines which direction counts as progress; abstraction shapes the routes the agent is likely to take. Jiang expects creativity and judgment in both areas to become much more valuable as automated search improves.
Driving these systems is itself an emerging skill: selecting the challenge, structuring the experimental work and deciding what constitutes a useful result. Automating the search changes where engineers contribute. In Jiang’s formulation, humans move up the stack, not out of it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Training code, competition rules and historical records for language modeling under a 16 MB artifact cap and a ten-minute training budget.
OpenAI's evaluation of machine learning engineering agents across 75 Kaggle competitions, including AIDE paired with o1-preview.
Weco's public machine learning engineering agent implementation, evaluated in MLE-bench.
Experiments explaining how sigmoid gates after attention improve stability and performance through nonlinearity and input-dependent sparsity.
Karpathy's collection includes his August 4, 2017 remark about gradient descent writing code.
Further reading
The organizers explain the challenge, record verification, notable submissions and the effects of widespread coding-agent use.
A public competition submission combining tokenizer changes, gated attention and quantization, with a reported validation score of 1.06549 bits per byte.
Read the complete timestamped transcript
- 0:00
[on-hold jingle] This April, OpenAI ran a hiring challenge, a competition called Parameter Golf.
- 0:19
The top contributor was one candidate that they couldn't hire. It wasn't a person. It's an agent we built called Aiden.
- 0:31
In Parameter Golf, the goal is to train the best language model you can under size and the computation constraints. About one thousand machine learning engineers, researchers participate.
- 0:47
They filed two thousand submissions. Only forty-seven passed OpenAI's review and made into the leaderboard. Seven of those are actually Aidens, more than twice what ev-- any human contributed.
- 1:07
You've seen a lot of auto research today. Agents are hill climbing benchmarks. Those are really impressive results. The question I want to ask is a bit different here. Can the auto research agent produce work that a human community actually recognize?
- 1:26
Beyond a good score, agent is optimizing for something that other engineers can merge, fork, and build on.
- 1:36
So instead of having an agent just hill climbing locally, we build one that publishes its own work, and that's Aiden.
- 1:47
Quick context on us. Weco is a auto research company that founded about two and a half years ago. Uh, I'm co-founder and the CEO, Zhengyao. Um, got my PhD at UCL on reinforcement learning.
- 2:01
About two years ago, we built AIDE, the top auto research agent independently evaluated by OpenAI in their MLE-bench paper.
- 2:13
Uh, even though back then there was, um, n-no such name called auto research. People call it machine learning engineering agent.
- 2:22
Aiden is the next step in a, a experimental prototype. It's a multi-agent self-improving system that can read public information like research papers and other PRs, run its own experiments, and submit a PR once the findings pass a quality gate.
- 2:46
We send Aiden to Parameter Golf competition, and it ran for about twenty-two days. By the end, Aiden has set seven leaderboard records. Each one is a new best for the competition, stamped by OpenAI, and the best human only made three.
- 3:07
Passing the host review is a one signal for the quality. A second, maybe more important one, is whether other participants would build on your work. And it turns out Aiden's work had the highest impact within the whole community.
- 3:27
Here we are using a inference measure that used widely in academia. It's called a h-index. Roughly, if you have X papers get cited X times, then your h-index is X.
- 3:44
Computed over PRs, Aiden was ten, and the next human was seven. The whole community was building on a AI system's work, including many of other leaderboard entries.
- 4:01
To break it down a little bit, why can a autonomous AI system be so powerful? One obvious reason is that, uh, it's an AI, it can run tirelessly. Over twenty-two days, it ran about one thousand and three hundred experiments on a single H100 node.
- 4:25
But throughput isn't the whole picture. A well-tuned AI system can also keep its output quality high.
- 4:34
On the compute side, it uses at most four percent of competition's total compute.
- 4:44
And it made about fifteen percent of the records.
- 4:49
Also, twenty-eight percent of its submissions made the leaderboard, roughly six times higher hit rate than the community average. So Aiden actually lifted the signal-to-noise ratio within the whole community's public communication channel, which is, uh, PRs.
- 5:10
It didn't win through massive parallelization, even though auto research have tons of potential of parallelization.
- 5:21
By those numbers, it might feel like auto research already dominates human experts on ML engineering and research, but that's not the full story I want to tell. Humans and AI are actually contribute in very different ways.
- 5:41
When we trace the ideas, Aiden, Aiden's record PRs,
- 5:47
almost all of them come from human research papers, other participants in Parameter Golf or in similar communities like NanoGPT. Those ideas are not necessarily a merged PR.
- 6:03
Sometimes it's a, a note, um, a human researcher said- Oh, I give up this idea because of some implementation, uh, implementation difficulty, and the agent is good at, uh, finding them and actually implement them.
- 6:19
There are also a very small fraction of original ideas that Aiden came up by itself, which, uh, emerged from its efforts to navigate the file size constraints.
- 6:32
Here's a concrete example that, uh, traces the patterns I just talked about.
- 6:39
So Aiden picked up an idea from Quen paper called the Gated Attention, and it worked. But, uh, uh, it introduced more parameters, and it broke the sixteen megabytes file size limit.
- 6:56
So it figure out a quantization mechanism to bring the file size down. But with those two primitives combined, the score barely moved.
- 7:09
Then another contributor posted a tokenizer improvement. Aiden recognized the idea, combined it with the architectural work. It just, uh, worked for five days or so.
- 7:23
And after this combination, the three ta-- the three ideas turns out to, to have a huge synergy that lead to a big jump in performance and it become one of the Aiden's leaderboard records.
- 7:41
So to sum up how I interpret Aiden and in general autoresearch systems' effectiveness, it's very strong at finding and implementing ideas. In the case we just saw, it brought an idea from a recent paper into a actual implementation in the competition, and it's good at, uh, promising ingredients out of, uh, the
- 8:05
parameter golf community, even though the public channel is actually very noisy information-wise.
- 8:14
It can also came up logically straightforward ideas. For example, in this case, once you add the parameters and it, uh, breaks the file size limit, one obvious next move is just a quantization.
- 8:30
And it's really fast and, uh, really efficient at finding right combinations across a huge search space.
- 8:41
Okay, maybe none of those sounds very sexy. Most of them are just a good execution. But in reality, execution is, uh, mostly the bottleneck.
- 8:53
What moves the frontier is usually exactly some belief on existing ideas and tons of good executions.
- 9:06
Okay, to step back, the state of a human-AI collaboration is a human collectively provide a lot of creative ideas and the agent do the execution
- 9:19
to solve a concrete challenge. What we are looking at is a, a large group of human and one AI system. Does it mean a single human engineer's contribution marginally gets smaller?
- 9:35
I'd say even for that, uh, not really. In parameter golf competition, it's easy to only focus on engineers that's actually doing hill climbing. But the design behind the competition itself is tremendously important.
- 9:52
A bad design can make the whole community effort useless. And their eval design work will have a fe-- huge leverage in the autoresearch era.
- 10:04
I really like one tweet from Andrej Karpathy about ten years ago, where he said, "Gradient descent can write code better than you. I'm sorry."
- 10:17
For the context, about ten years ago, deep learning was starting to eat up a lot of, uh, software engineering, like conventional coding work. And his tweet was arguing against those people who thought they can hand-write better code than a trained model.
- 10:37
Okay, now obviously no one is seriously trying to hand-write code to beat a model. However, software engineering, I mean, as a job still exist. And so many people's job are just, uh, training those models, and those are one of the most well-paid job today.
- 10:58
I think how gradient descent change coding is a great metaphor for how autoresearch will change research in the ML engineering.
- 11:08
It commoditize certain execution skills. At the same time, it makes some higher-level skills far more valuable.
- 11:19
So actually doing autoresearch is, is a lot like training a model. Your codebase abstraction is essentially the architecture. It sets the constraint and the priorities, um, for what the agent can explore.
- 11:35
Your eval is the loss function and the data. It sets what agent optimizes for.
- 11:43
Take the eval first. The eval is the signal you use to train a model. In this case, it's training your code.
- 11:53
It plays the same role that, uh, like data and the loss function, uh, in model training, or in a reinforcement learning setting, it's like environment that the agent is training.
- 12:07
Nowadays, no one would argue data or environments, uh, don't matter.
- 12:14
And, uh, this is where a vertical moat can also be built. You might have a proprietary data for evaluation or a unique understanding of a, in a particular field, what matters and how to measure it.
- 12:30
And a good evaluation would be amplified more and more as autoresearch are getting stronger.
- 12:40
The other one I think is really underrated is codebase abstraction.
- 12:46
The abstraction provides the framework that autoresearch can iterate on.
- 12:53
And, uh, that's also... That starting point hugely bias the whole search direction. This is a lot like architecture design in neural networks. Different architecture in theory can represent the same function, but the architecture systematically makes some of the functions easier to be learned.
- 13:19
And a good architecture biases the optimization towards solutions that generalize better, perform better, even when the training loss might looks the same.
- 13:32
That's exactly the same for autoresearch. Here's an example. We run autoresearch for a, um, fraud detection pipeline, um, and we're trying to optimize the data preprocessing.
- 13:48
And first we goo- give it a loose API where the same function process both the training and the testing data,
- 13:59
and the score looks great. But the solution was polluted because, uh, there's a certain test set information got leaked to the training information.
- 14:14
We then tightened the abstraction to a more strict API where the test data couldn't reach the training, and the data leakage rate just dropped to zero. In this case, a good abstraction leads to better solutions even though if the agent really want, they can still reward hack.
- 14:38
So my point is, uh, using autoresearch is a new craft. It's about designing a hill for an agent to climb, and we are still very early on it. I think, uh, that makes this an extremely exciting time to be an AI engineer.
- 14:57
Autoresearch will change what skills matter most. Creativity, the judgment to design a good eval or an abstraction, those will soon get exponentially more important.
- 15:12
Driving those system itself is where will be a new skill, and, uh, that one is, like, uh, barely existed one or two years ago. So the search is automated.
- 15:25
The human would just move up the stack, not out of it.
- 15:32
Again, um, WeCo is a autoresearch, um, product research lab. We, we keep sharing what we are learning as we build, uh, on our blog, and I will also post some of my thinking too on X.
- 15:48
Uh, if you think some of this are useful to you, feel free to follow me on X. Thank you. [audience applauding] [upbeat music]