How Autoresearch Is Changing ML Research — Zhengyao Jiang, Weco AI

OpenAI’s Parameter Golf hiring challenge had a prolific contributor it could not hire: Aiden, an agent built by Weco. The challenge was to train the best language model possible under tight size and computation constraints. Jiang reports roughly 1,000 participants and 2,000 submissions, with 47 accepted record-track entries, seven of them Aiden’s. These were reviewed leaderboard records, not a count of every valid submission.

That makes the challenge a useful test of something beyond benchmark optimization. Can an agent produce research that a human community recognizes and uses? A good score is one outcome; a contribution that other engineers can merge, fork and build on is another. Aiden was built to publish its work, extending the research loop from local experimentation into the community’s shared development process.

Weco co-founder and CEO Zhengyao Jiang came to this problem through reinforcement learning research at UCL. At the time of the talk, Weco had been operating for about two and a half years. Its earlier agent, AIDE, had been evaluated in OpenAI’s MLE-bench: o1-preview paired with AIDE was the best-performing setup evaluated in that historical study. The work then went by the name machine learning engineering agents, before autoresearch became the common label.

Aiden is the next experimental prototype: a multi-agent, self-improving system with a research-to-publication workflow.

1. Read public information, including research papers and other contributors’ pull requests.
2. Run experiments to investigate promising ideas.
3. Submit a pull request once the findings pass a quality gate.

The publication step matters because it exposes the result to review and gives other participants something concrete to reuse. Jiang does not detail the quality gate’s implementation here.

Over about 22 days, Jiang reports that Aiden set seven new competition-best records approved by OpenAI, compared with three for the most successful human contributor. Passing the host’s review established one kind of quality. Whether other participants built on those contributions provided a second test.

To measure reuse, Jiang applies the academic h-index to pull requests: an h-index of h means at least h contributions each have at least h citations. Jiang reports a PR h-index of 10 for Aiden and seven for the next human contributor. On that measure, Aiden had the greatest influence in the competition, with other leaderboard entries building on its work. The result depends on how PR citations are counted; the talk does not specify that procedure.

Continuous experimentation explains some of that output, but Jiang also emphasizes selectivity. His reported figures are:

A node is not necessarily a single GPU, and visible compute is not a complete accounting of everyone’s expenditure. These figures describe the competition run, rather than a controlled comparison of agent and human research efficiency.

Jiang describes the 28% acceptance rate as roughly six times the community average, but the denominator behind that comparison is unspecified; dividing the opening totals gives a different pooled rate. His broader point is that Aiden did not simply flood the public channel with experiments. Pull requests were the community’s communication medium, and a high proportion of useful submissions could improve that channel’s signal-to-noise ratio. He attributes the result to productive, selective research rather than massive parallelization.

Those results do not mean that Aiden independently originated the research it published. Tracing the ideas behind its record pull requests revealed a different division of work: almost all came from human research papers, Parameter Golf participants or communities such as NanoGPT.

Some useful inputs were not even merged contributions. A researcher might leave a note explaining that an idea had been abandoned because it was difficult to implement. Aiden could find that unfinished work and carry it through to an experiment. A small fraction of its ideas were original, emerging as it worked around the competition’s file-size constraints. The important distinction is between supplying a promising idea and doing the work needed to make it useful.

One record illustrates how that division of work unfolds. Aiden picked up gated attention from a Qwen paper, Gated Attention for Large Language Models. Adding it helped the model, but the additional parameters pushed the artifact beyond the 16 MB file-size limit. The first improvement was therefore unusable under the competition’s constraints.

Aiden then developed a quantization mechanism to bring the file size back down. That solved the constraint problem, but gated attention and quantization together barely moved the score. Next, another participant posted a tokenizer improvement. Aiden incorporated that idea into its architectural work and continued experimenting for roughly five days.

The three-way combination produced the performance jump that became one of Aiden’s records. The completed slide names the tokenizer component CaseOps and shows the full progression, including the invalid intermediate result. The lesson is in the sequence: a promising architecture change first violated the constraint; quantization made it feasible; the later tokenizer contribution made the combination substantially more effective.

This example exposes several distinct strengths:

• Finding and implementing ideas: turn a recent paper or a promising fragment of community discussion into working experimental code.
• Responding to constraints: recognize that additional parameters create a size problem and pursue quantization as a direct response.
• Testing combinations: search for interactions among ingredients that offer little benefit in isolation.

Much of this is execution rather than a new conceptual breakthrough. But execution is often the bottleneck: progress requires enough commitment to existing ideas to implement, test and refine them.

In this case, a large human community supplied creative ideas while one AI system executed against a concrete challenge. That can make an individual engineer’s contribution look smaller if the only work being counted is improving the leaderboard score. But the people who designed the competition determined whether those improvements would be valuable in the first place. A poorly designed evaluation could direct the entire community toward useless results.

Jiang connects this shift to Andrej Karpathy’s remark, published in 2017: “Gradient descent can write code better than you. I’m sorry.” The context was the displacement of hand-written solutions by learned models. Software engineering did not disappear; valuable work moved into building and training those models. Jiang expects autoresearch to produce a similar change in ML engineering: some execution skills become commonplace, while the judgment needed to direct that execution becomes more valuable.

Running autoresearch resembles training a model in two practical ways:

The codebase sets the constraints and priorities for exploration. The evaluation supplies the signal that selects among the agent’s changes.

In ordinary training, that signal changes model parameters. In autoresearch, it guides changes to code. An evaluation therefore plays a role similar to training data and a loss function—or to the environment in reinforcement learning. Its quality is central to the result, just as data and environment design are central to model training.

This creates room for domain-specific advantage. Proprietary evaluation data can reveal failures that a public benchmark misses. Equally valuable is the understanding of what matters in a particular field and how to measure it. Jiang expects stronger autoresearch systems to amplify the value of those choices: better search can extract more value from a well-designed objective.

The other design choice is the codebase abstraction: the framework within which the agent can iterate. Its starting structure biases the search direction. Two neural architectures might theoretically represent the same function, yet make different functions easier to learn. Similarly, two codebases may permit the same eventual solution while making very different paths convenient. Good structure can favor solutions that generalize better even when their training losses look similar.

Weco encountered this while optimizing preprocessing in a fraud-detection pipeline. The initial API was loose: the same function processed training and test data. The scores looked excellent, but test-set information had leaked into training. A high evaluation score was hiding an invalid experimental procedure.

The team tightened the API so test data could not reach training. Jiang reports that observed data leakage fell to zero after the stricter API was introduced. The slide shows the loose-versus-strict comparison. This is an observed result from that experiment, whose dataset and detection protocol are not specified here; Jiang explicitly allows that an agent could still deliberately reward-hack. The abstraction improved the normal search path without making every possible misuse impossible.

A concrete Python pattern for this separation is to fit preprocessing on training rows, then apply the fitted transformation to test rows. Here, the training values are 10 and 20, so their mean is 15; the held-out value 100 never influences that fitted mean.

from dataclasses import dataclass
from statistics import mean

@dataclass(frozen=True)
class Centerer:
    training_mean: float

    @classmethod
    def fit(cls, training_values: list[float]) -> "Centerer":
        return cls(training_mean=mean(training_values))

    def transform(self, values: list[float]) -> list[float]:
        return [value - self.training_mean for value in values]

train = [10.0, 20.0]
test = [100.0]

preprocessor = Centerer.fit(train)
train_features = preprocessor.transform(train)  # [-5.0, 5.0]
test_features = preprocessor.transform(test)    # [85.0]

This illustrates the API boundary: fit receives training data alone, while transform reuses immutable fitted state. In an autoresearch harness, the surrounding code must preserve that separation rather than let an editable training function receive the held-out data. The structure makes the valid procedure easy to follow.

Using autoresearch is becoming a craft of “designing a hill for an agent to climb.” Evaluation determines which direction counts as progress; abstraction shapes the routes the agent is likely to take. Jiang expects creativity and judgment in both areas to become much more valuable as automated search improves.

Driving these systems is itself an emerging skill: selecting the challenge, structuring the experimental work and deciding what constitutes a useful result. Automating the search changes where engineers contribute. In Jiang’s formulation, humans move up the stack, not out of it.