Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI
Read the talk
Parameter Golf with AutoResearch
Weco AI turns language-model research into an inspectable search over code, then applies the same machinery to a Flappy Bird controller. The useful work lies in defining the evaluator, bounding experiments and knowing when to steer.
From a talk by Vayum Arora, Zhengyao Jiang, Dixing Xu and Dhruv Srikanth
At a glance
Ideas worth remembering
A useful AutoResearch setup specifies editable code, a replayable evaluation, the metric's direction, a search budget, a timeout and initial guidance.
Fast and cheap evaluations increase the number of affordable experiments; low noise makes score comparisons more trustworthy. None of those properties prevents reward hacking on its own.
An experiment tree keeps candidate lineage, scores and code changes connected, allowing people to inspect results and steer new branches.
The reported tokenizer breakthrough combined an outside contribution with accumulated agent work after a five-day plateau; continued autonomous search was only part of the process.
Flappy Weco combines multiple environments, hidden validation cases and restricted controller edits to make the score more closely reflect the intended playing behavior.
Give the agent code and a replayable test
A coding agent usually waits for the next instruction. An AutoResearch system keeps proposing and testing changes on its own. In this workshop, Vayum Arora, Zhengyao Jiang, Dixing Xu and Dhruv Srikanth of Weco AI begin with a small specification: give a language model some code and a replayable evaluation script, let it try ideas, keep changes that improve the score and discard the rest. The score might measure accuracy, speed or cost.
Autonomy makes the initial setup matter more. A person steering each coding step can correct a misunderstanding quickly; an unattended search can spend its whole budget pursuing the wrong signal. Zhengyao's phrase is that the right evaluation lets the agent “boil the ocean for you.” The same persistence becomes waste when the evaluator rewards something other than the intended behavior.
The team's earlier system, AIDE, released in April 2024, applied this approach to machine-learning engineering: write code, run experiments, read results and improve. Vayum reports that AIDE achieved about four times the medal rate of the next best agent in OpenAI's MLE-bench evaluation. The workshop moves from that background to a more constrained example, Parameter Golf, before giving participants a smaller version they can watch run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parameter Golf makes quality compete with size and training time
Parameter Golf asks participants to train the best language model they can while satisfying two hard constraints: the submitted artifacts, including model weights and training code, must fit within 16 megabytes, and training must finish within ten minutes. Within those limits, the objective is to minimize bits per byte, or BPB, on the FineWeb validation set. Lower BPB means better prediction of the validation text. A larger model or a longer training run cannot simply buy a better result if it breaks the rules.
OpenAI supplied a baseline, data and evaluation scripts, and participants submitted improvements through GitHub. Dex describes a 44-day competition with 1,016 researchers and 2,048 pull requests; 47 PRs from 31 contributors were merged as records. That open submission process also made other participants' implementations available as research inputs.
Aiden, Weco's multi-agent system built around the Claude and GPT model families, searched against that single metric. Its inputs included arXiv papers, GitHub implementations and its own experiment logs. It proposed and tested ideas, checked whether they complied with the competition rules, retained useful work and submitted a PR after results passed its quality gates. The legality checks matter: an improvement only counts if the resulting program still satisfies the challenge.
Dex reports seven merged leaderboard records and 435 downstream PRs that cited and built on Aiden's work. The team also reports using 4% of the community's GPU compute to produce 15% of the merged records, which it describes as 3.7 times the field's average cost effectiveness. Here, cost effectiveness means merged records per share of GPU compute; it does not establish an equivalent advantage in total research cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Shrink the experiment until several attempts become visible
Mini Parameter Golf, shortened to MiniPG, preserves the task's essential structure while reducing its scale. Each candidate still trains a real language model and receives a validation score, val_bpb, from a fixed evaluator. A tiny dataset replaces FineWeb, and training plus evaluation takes a few minutes. The point is to make several actual experiments visible during a workshop rather than spend the session waiting for one large run.
The setup divides the work between Weco, which runs the research search, and Modal, which supplies compute for training and evaluation. Participants are asked to clone the workshop repository and tell their coding agent to follow its README and start a run. Internet problems interrupt this plan, so the workshop turns first to the configuration that makes the search possible.
The first configuration choice is what the agent may edit. In MiniPG, that is solution.py, which defines the model configuration and training dynamics. This file defines the search space: the agent can change how the candidate model is built and trained, while a separate evaluation procedure measures the result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The evaluator determines how much useful search you can afford
Dhruv puts evaluation at the center of the system. Every generated solution needs a simple evaluation procedure that returns a scalar score. Three properties serve different purposes:
- Fast: shorter experiments allow more attempts within a time limit.
- Cheap: lower experiment cost allows more attempts within a spending limit.
- Low noise: a score change is easier to trust when repeated measurements do not vary widely on their own.
The first two increase search capacity. The third helps the agent and the person decide whether a candidate deserves further work.
A reliable number can still reward the wrong thing. Agents can find holes in a benchmark or evaluation protocol and exploit them. That makes reward hacking a design problem before the search begins: the evaluator must reward the intended behavior, and the allowed edits must not let the agent redefine success. MiniPG's stated goal is explicit—minimize validation BPB to improve prediction on unseen text.
The workshop chooses 20 search steps to give participants room to see different ideas succeed or fail. That is a practical starting budget, not a claimed optimum. There is no known step count that guarantees a breakthrough; cheaper and faster evaluations simply create room to test more possibilities under the same constraints.
Each evaluation also needs a timeout. Generated code can hang, loops can stall and hardware can fail. MiniPG sets a 210-second limit, budgeted as 60 seconds for training, 60 seconds for testing on unseen text and 90 seconds for GPU startup and other overhead. The overhead allowance keeps normal infrastructure delays from consuming the whole useful experiment window; the outer limit prevents a broken candidate from blocking the search indefinitely.
Initial guidance supplies knowledge the search would otherwise have to rediscover. Weco can read a Markdown file containing constraints, promising ideas and approaches that have already failed. For MiniPG, suggested directions include compression, quantization and weight sharing across model blocks. These are starting hypotheses, not demonstrated improvements in this walkthrough: the evaluator still decides whether a particular implementation helps.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the code change attached to the score
Weco organizes the search as an experiment tree. The root contains the baseline program and its measured score. Each later node contains a modified version of the code and the score obtained by evaluating it. Promising branches receive further refinement; unsuccessful directions are pruned. This preserves the lineage of an idea, so a useful change can become the starting point for another experiment.
What changed when the score moved? That is the question the tree and diff view are meant to answer together. Hovering over a node gives a summary; opening it exposes the code change. The tree locates the candidate within the search, while the diff shows the intervention associated with its result. Keeping both matters when a run lasts hours and accumulates hundreds of experiments.
Steering adds a way to act on that inspection. A person can submit an idea in plain English, open a new branch and assign budget to nodes so compute goes toward the desired direction. This lets domain knowledge enter during the run, after the experiment history has exposed a dead end or a promising opportunity. The human contribution can change what the system tries next without discarding the work already accumulated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A plateau breaks when an outside idea meets accumulated work
The Parameter Golf example gives steering a concrete purpose. From April 15 through April 19, Aiden kept trying changes involving attention, compression and a richer stack, but its score essentially plateaued. A community participant then published a new tokenizer. Within eight hours of that change appearing, Weco reports that Aiden incorporated it, combined it with the pieces it had already developed and produced its biggest improvement in weeks.
What does the chart make easier to compare? Vayum identifies the agent's work as the pink line and the community tokenizer as the blue line. The useful relationship is the stalled search followed by improvement after the outside contribution was incorporated. The recording does not explain the tokenizer's implementation or isolate its contribution from the accompanying changes, so the example supports the value of combining research streams rather than a specific tokenizer recipe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A printed score drives the search; constraints define acceptable tradeoffs
The guided setup resumes over a mobile hotspot. A coding agent receives the repository instructions and starts the default Weco run. The important distinction in the configuration is between the editable training program and the evaluation command. The command runs the evaluation and prints a score to the terminal; the agent parses that output. That parsed number becomes the signal for choosing which changes to pursue.
While the run starts, an audience question asks whether AutoResearch can apply to reinforcement learning. Zhengyao's answer is yes: the editable object can be an RL training script. The same outer search proposes changes to that script and evaluates their consequences. This places AutoResearch around the training procedure; it does not require every code proposal to be a reinforcement-learning update.
Multiple objectives require a decision about tradeoffs before a single-score search can compare candidates. The answer offers two approaches:
- Weighted combination: combine the objectives into one score with different weights.
- Constraints plus an objective: require some metrics to satisfy limits, then optimize another.
Parameter Golf uses the second approach: artifact size and training time restrict eligible solutions, while validation BPB determines which eligible model is better. This preserves the hard limits instead of allowing a sufficiently good prediction score to compensate for breaking them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect an improvement, then ask for a different kind of experiment
The live dashboard reaches a concrete result: node zero is the baseline, with validation BPB of 3.93, and the agent has generated a candidate node. The presenters describe an apparent improvement and point to the option to open its code. What score belongs to the baseline, and what changed in the candidate? Those are the useful details to inspect together; the walkthrough does not state the candidate's new score or explain its diff.
The proposed steering instruction is more specific than “make it better”: explore quantization, rather than only changing the model configuration. That asks the search to try a different class of intervention instead of continuing with hyperparameter tuning. It is a useful example of how human guidance can change the kinds of hypotheses being tested.
The live steering attempt runs into a connection problem. The presenters say they do not think the session is connected and suggest that Claude Code did not follow the setup instruction. They then explain Weco's bridged session, which puts Claude Code alongside the experiment tree and metric so a person can direct the search through the coding agent. The intended quantization branch is therefore an instruction example here, not a demonstrated successful run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Flappy Weco tests whether the controller survives different worlds
The final exercise removes the Modal dependency after participants encounter connectivity and payment problems. Flappy Weco turns Flappy Bird into a code-optimization task: a controller named should_flap decides every frame whether the bird should flap. The objective switches direction from MiniPG—maximize average game score, which depends on how far the bird flies.
The evaluator tests four worlds:
- Normal: the classic game.
- Moving gap: the opening between pipes drifts up and down.
- Wind: wind pushes the bird around.
- Variable: conditions are mixed.
The final score averages performance across all four. A policy that excels in the normal world but fails in wind loses ground in that average. The aggregation rewards performance across conditions, although an average does not itself require a minimum score in every world.
The leaderboard uses a hidden validation set, so the local score is not expected to match the submitted score. This withholds the final test cases from the search and reduces the opportunity to tailor a controller to the public game alone. The editable program is also tightly restricted: participants may change only ai_controller.py, must preserve the should_flap signature and may not modify the game or evaluator. Imports, external libraries and file I/O are prohibited.
The competition also prohibits parallel compute and assigns everyone the same model through Weco. These rules address a different source of advantage from the hidden test: they constrain the research resources available to participants. Evaluation design has several jobs here—measure play across conditions, keep the final cases hidden, protect the game and evaluator from edits, and limit differences in search resources.
The suggested starting point is the working baseline controller. Improve it with small, focused edits: adjust safety margins, change the bottom buffer or adapt behavior to a particular world. Then inspect successful nodes and build on them, while trying other directions when useful. This gives each experiment a narrower question and keeps a functioning controller available as the search proceeds.
The original 30-minute exercise budget, estimated at about 50 steps, is extended; the closing deadline becomes midnight. A Nintendo Switch is offered as the prize. As scores begin arriving, an entry with only one node prompts the organizers to say they will inspect runs for cheating. A thin experiment history is not proof of misconduct, but it gives them a reason to look beyond the final number. The workshop ends with the same requirement it began with: a useful score needs an evaluation process—and an inspectable path to the result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explains how execution traces guide candidate edits and how preserving alternative candidates can help search escape a stalled line of improvement.
Read the complete timestamped transcript
- 0:12
Uh, welcome. Please grab a seat if you haven't already. My name is Vayum. We have Zhengyao, Dhruv, and Dex. For anything else, uh, a huge thank you to Sean and the AI Engineers World Fair for having us. This is generally, um, a great room of AI, and we're really, really happy to be here. Um, if you haven't been able to get set up yet, no worries. Uh, we're gonna have ample time to get it to, to, to it later. Um, Zhengyao will give us a brief background on the basics behind
- 0:42
AutoResearch.
- 0:52
Okay. Um, in case, uh, any of you still don't know of AutoResearch is, this is just a quick brief. So AutoResearch is basically a loop of a trial and errors. You give a language model your code and, uh, a replayable evaluation script. It will keep trying ideas and then test them against the eval, and it keeps the ideas that improve the score, and it discard
- 1:22
everything else. So some example metrics includes accuracy, speed, uh, and cost. Similar loop has been exist, uh, in academic for a while, like two years, but, uh, it really got viral in this year when Karpathy, uh, posted about it in his, uh, model training scripts, and it turns out, works really well for
- 1:52
him.
- 1:55
In just a few weeks, it just-- the use case of AutoResearch spread from training to everything, like, uh, backend engineering, prompt and agent engineering, speed up model inference, and it, it also quickly show up in production. For example, it reduce the latency of, uh, Shopify's CI/CD pipeline, and it improved the browser use-- the company's, uh, uh, browser use agent.
- 2:27
In April, our experimental research agent, Aiden, even won OpenAI's hiring challenge, uh, called the Parameter Golf, uh, which we will tell you a bit more soon, and, uh, I also have a talk on Wednesday specifically about it. Um, so okay. AutoResearch can be really powerful, but, uh, it's also hard to master. With a normal coding agent,
- 2:57
you basically steer, um, agent every single step. But for AutoResearch, it's like the agent is running in a loop on its own. So it's much bigger multiplier on your prompt and your setup. Point it to the right eval, you basically let it boil the ocean for you. But then otherwise, you're just, uh, burning through all of your RM tokens. Um, so hopefully this workshop
- 3:27
will give you a good starting point to learn about AutoResearch and, um, apply that in your own work. Okay, we come back to Vayum.
- 3:40
Thanks, Zhengyao. Um, so here's how we'll spend the next two hours. Um, Dex will take you inside Parameter Golf, uh, the challenge that we entered our agent into, um, what that challenge was actually all about, and how our agent ended up as the most influential contributor. Um, then it's your turn. Uh, we'll get you set up and kick off an AutoResearch run on your laptop. While your AutoResearch is running in the background, we'll break down the basics of an AutoResearch loop and the pieces that actually shape what an agent does.
- 4:11
Um, then we'll go deeper and do a guided demo where you're able to run, steer, and inspect what your agent is doing live. And we'll finish with a fun part, which is a live competition. Um, you'll take everything that you learned today, point AutoResearch at a game that we've all played before, and go head-to-head to see whose agent can get the highest score, uh, with the winner walking away with, uh, a really cool prize.
- 4:42
A little bit more on who we are. Um, so we've been working on AutoResearch for quite some time now. Before Weco, uh, we released AIDE in April 2024, which was an autonomous ML engineering agent. The basic idea was give it a machine learning problem, let it write code, run experiments, read the results, and keep improving. When OpenAI built MLE-bench, AIDE was one of the first agents they evaluated independently, and on that
- 5:11
benchmark, it scored about four times the medal rate of the next best agent. More recently, as we discussed, we pointed our agent to Parameter Golf, and across more than a thousand research, it, it, researchers, it became the number one contributor. I'm gonna hand it off to Dex, who's gonna tell us more about the competition.
- 5:39
Uh, hello, everyone. My name is Dex. Uh, so I'm going to talk about, uh, Parameter Golf. So Par-Parameter Golf is OpenAI's, uh, model challenge. The goal is to train the best language model you can under two hard constraints. Uh, one is the four artifacts, including model weights, plus training code, has to fit in sixteen megabytes, and the training has to finish within ten minutes.
- 6:06
Submissions are scored by bits per byte, BPB, on the FineWeb validation set, and the lowest better. OpenAI provided a baseline data set and evaluation scripts, so participants can fork the repo, improve the model, and submit their result through GitHub. So everything is open sourced. They also put up a million dollar in compute credits and reviewed every submission. During the forty-four-day competition, there
- 6:36
are a thousand and sixteen researchers entered and filed two thousand and forty-eight PRs, roughly about, uh, two fifty K of GPU compute burned. Finally, the board is, uh, short only forty-seven PRs merged as records from thirty-one contributors. And for this competition, we built Aiden, a self-improving multi-agent system based on the Claude and GPT model families that run indefinitely to hill-climb on a
- 7:06
single metric. In Parameter Golf, Aiden automated the full research loop. It pulls public page-- papers from arXiv, strong implementations from GitHub and its own past logs, then propose new ideas, tests them, audits them for legality, keeps the ones that work, writes the implementation and run the experiments. Once the results clear the quality gates, it will create and submit a PR for
- 7:36
review.
- 7:38
And the results are remarkable. Aiden landed seven merged le-leaderboard records. That's twice the next best human. It also had the highest all-PR h-index in the whole competition, which means Aiden didn't just produce accepted records. It published reusable work that other participants kept building on. Four hundred and thirty-five downstream PRs cited and built on Aiden's work. And it got there on a fraction of the budget. From just four percent of the
- 8:08
community's GPU compute, it got fifteen percent of total merged records. That's three point seven times more cost-effective than the field average. And this wraps up Ai-- how Aiden becomes the number one contributor to the OpenAI's Hiring Challenge. Thank you.
- 8:30
Cool. So we're actually gonna run AutoResearch on a version of the same Parameter Golf challenge that Dex just walked you through. Uh, for future reference, we will just call this MiniPG, which stands for Mini Parameter Golf. Um, as Dex discussed, the real Parameter Golf competition was heavy. Every single experiment was training a language model on a massive data set, which is fine when the competition is over four weeks. But in a workshop like today, you'd
- 9:00
never actually be able to see your AutoResearch run. So think of MiniPG as a scale model of the challenge. Uh, you're still gonna train a real language model, score it on the same fixed evaluator, and drive down the same single number, val_bpb, which is essentially measuring how well the model predicts text it hasn't seen. But what we did is we shrank this in scale. We're using a tiny data set instead of FineWeb and training an eval that only takes a few minutes. So you'll actually
- 9:30
be able to get a loop going and see several steels-- several real steps live.
- 9:38
So before I get to the instructions on the next slide, just a little bit of orientation. Um, you're gonna run AutoResearch yourself on Mini Parameter Golf. You're gonna sign up for two platforms if you haven't already. One is Weco, our AutoResearch agent that runs the search, and the other is Modal, uh, to compute your experiments train and evaluate on. Free credits. Thank you, Modal. Um, then you'll clone our Parameter Golf repo, uh, and hand it to your coding agent, tell it to kick off a run. It handles
- 10:08
the setup, and it'll start the loop. Once it's going and it's running in the background, uh, you can kick it off, and we will check back in in around fifteen or twenty minutes to take you inside the details.
- 10:22
Instructions are here. Um, sign up for Weco on dashboard.weco.ai. Um, Modal for modal-modal.com/signup. Uh, there's a code to claim your Modal credits. Um, if you already have a Modal account, you can go directly to modal.com/credits and enter your code there. Um, and once you're done with those two steps, uh, clone the repo shown here, um, and tell your agent to set up MiniPG from the README and kick off a run. If you run into any issues
- 10:52
or have any questions, feel free to scan the QR code and enter it there, and we'll be monitoring that, or, uh, raise your hand, and we'll come to you. Check back in in around ten, fifteen minutes. Thanks, guys.
- 11:04
Uh, all right. So sorry about the internet problem. I think it's being worked on. But until then, I think it would be good to maybe go through, uh, the anatomy of an AutoResearch run. And we're gonna be doing this in the context of, uh, Weco as the AutoResearch system and MiniPG as the task. But really, this, this kind of contract that we have here is fairly general. Most AutoResearch systems are gonna have you ask very similar questions and, uh, think about the same things.
- 11:36
Okay, so the first thing is the source file. Uh, this is what the agent actually has to modify. This defines the search space and where it can make changes. So this can be a single file. It can be a m-- it can be multiple files. In the case of MiniPG, it's, uh, this one file called solution.py. And solution.py essentially defines the model config and the, uh, uh, the training dynamics that's gonna go on while the model is training.
- 12:06
And then the second thing, and in my opinion, this is probably the most crucial part of an auto research system, is the evaluation. And so you wanna have a very simple way of allowing your auto research system to evaluate every solution that it generates. Um, and you wanna have it output a single scalar metric, and you want the eval to be fast, you want it to be cheap, you want it to have low noise, uh, because this essentially leads to
- 12:36
your ability to run more steps if you have a, a fixed budget or a fixed time constraint. Um, and if there's low noise, then both you as well as the agent are more confident as you're kind of hill climbing on, uh, whatever task you're, uh, auto researching on. And so, uh, one thing to be aware of though is agents love to reward hack. So they love to find holes in your evaluation protocol, holes in your benchmark, and so you really wanna figure out how to make your eval as robust as
- 13:06
possible.
- 13:10
The next two things are the metric and the goal. So you really wanna ask yourself two questions: Um, what do I want to improve? And is higher or lower better? Um, and in our case, for MiniPG, you wanna minimize the validation BPB. So you essentially want to find a model that is better at predicting on unseen text.
- 13:35
The next thing is the number of steps you actually run the auto research system for. This is a bit tricky because it's difficult to know a priori how many steps you should give your, your system. We've chosen twenty in this case just because we wanna give your agents enough time to kind of explore different ideas, and you can, you can see what the agent is actually exploring, what works, what doesn't. Um, but really more ideas that gets tried, uh, generally leads to
- 14:05
more breakthroughs. And so, uh, this goes back to what I was saying earlier, where if you have a fast eval, um, or if you have a cheap eval, then when you're under some kind of time constraint or cost constraint, you can run more steps. Um, and so, uh, again, the eval is, is always super important to, uh, consider.
- 14:30
All right. So you also wanna bound the, uh, time for which your eval runs for, uh, because a lot of things can go wrong. So loops can stall, hardware can fail, uh, sometimes the agent will produce really bad code that causes the entire system to hang, and so you wanna bound the, the actual evaluation running. Um, in our case, we set it to be two hundred and ten seconds, and the reason for that is we give sixty seconds for, uh, training the model,
- 15:00
sixty seconds for testing the model on unseen text, and then ninety seconds, which is a buffer for, you know, Modal to spin up the GPUs for training, spin it up for, um, uh, for the testing, and any other kinds of, uh, overheads that exist.
- 15:21
And the last thing is the guidance that you give the agent at the start of your optimization. Uh, so this is how you steer the agent from the start. And you really wanna think about it as, you know, you've probably run experiments before, you've compounded a bunch of domain expertise and human knowledge, and you might have some ideas about things that work, things that might be interesting to try, and maybe things that don't work. And so this is where you can provide that kind of guidance. Um, in
- 15:51
the case of Weco, you can simply point it to a markdown file, have all of your information and context in there, um, and the agent is able to use this and, uh, it gives the agent the best chance at, uh, at finding success. Um, in the case of MiniPG, this can be, uh, some hard constraints or different ideas that seem to work quite well, like, uh, focusing on compression, uh, different quantization methods, uh, maybe even weight sharing across different blocks in the model.
- 16:31
Um, we are just going to demo how to get started on AutoResearch for you considering the internet issues. Um, before we get to that, a little bit more about what we think, um, makes auto production-grade AutoResearch actually production-grade. Um, the first is structure. So, uh, we have a tree-based search, uh, where the agent shows you every attempt it's run, so that ideas, as Dhruv had mentioned, can sort of compound on
- 17:01
each other instead of starting from scratch. The second is observability. Um, you can sort of open any attempt, read the exact code change behind its score, so you can see why the agent is taking the approach it is. And the third thing is steerability. So you have a sense of control, you can inject your own ideas and knowledge while the loop is running and have it try new directions in case it's stuck.
- 17:29
A little bit more about the tree-based search. Um, we will shortly demo this on our AutoResearch dashboard. But, um, tree-based AutoResearch essentially treats the problem as a structured optimization process over code. Uh, it starts with the root of the tree. As you can see, the score was zero point five, um, which is just the baseline program and a measured score. Every node after that root is a modified version of the code run against your eval with a score attached. You can think of this
- 17:59
tree as an experiment graph. As you can see, the search builds on what works, refining good candidates and pruning bad ones. So when a branch improves the score, it builds on those ideas and goes deeper next Which means that you can inspect the exact changes your AutoResearch system is making and build off of those.
- 18:22
So the next idea we talked about was observability. Um, because you can see every one of the nodes in this tree-based search, you can open any one of them and see exactly what changed. You can see your metric move in real time, correlating to a movement-- correlating the movement to a reason. You can hover of a n- over a node, get a summary of what changed, and you can open it and read the exact diff. So when the metric jumps, you can read the change that caused it. And typically,
- 18:53
these runs go for hours across hundreds of experiments. So once you can see what it's doing, the next natural step is to be able to change what it does.
- 19:03
This is essentially when you stop watching and start to take over. Um, if I have an idea I wanna try in plain English, um, I take the idea, it opens a new branch, and you can see it try this new idea and have new nodes develop and a new curve. I can also put a budget to each node, so it spends compute where I want. And this is the part that made it usable on real problems. Because you might have context and intuition that an agent doesn't, at least for now, we believe it's important to
- 19:33
find ways to fold human judgment into the system whenever it's necessary.
- 19:40
I wanna ground that point, uh, steerability, in something real. Um, because parameter-- because in Parameter Golf, steerability was a big part of why we did well. Um, this is an actual breakthrough from the competition. It started with our agent, uh, which is the line in pink. Um, for five days, from April 15th to April 19th, uh, it kept trying new things, um, new attention, compression, a richer stack, but the score essentially plateaued. There was a human in the community, uh,
- 20:10
that shipped a clever new tokenizer, which is the blue line. Um, and within eight hours of emerging that change, our system picked it up, fused it with the pieces it had already been working on, and shipped the biggest jump in weeks. Running AutoResearch on something this hard is less of a set it and forget it. Uh, it's a collaboration where we steer the agent, we pull in ideas from the people around us, and steerability is essentially what makes that loop possible. It's how real
- 20:40
breakthroughs actually happen.
- 20:44
Um, I'm gonna hand it over to Dex and Zhengyao to talk a little bit more about how to get AutoResearch set up. Um, we'll essentially mimic the process that you guys were trying to do on your laptops.
- 20:56
Okay. Um, so what we are going to do here is we just, uh, walk you guys through the high-level idea, like, uh, how, how does it happen. We just use our mobile hotspot, um, so we will be freestyling a little bit here. Uh, hope you don't mind. Um, so yeah. Can you just, uh... Yeah. Okay, maybe we can also walk through the whole doc
- 21:26
being fed to, uh, Claude. So-
- 21:30
Yeah. So, so basically, uh, so you, you, you can go into the Rebel and, uh, some, like, simple instruction, like start the Weco raw based on the, the readme. And it should be, like, if the internet is good, I think it should start the, the, uh, default Weco run. And, uh, we can also go over via the, like, the how, how is the Weco command working.
- 21:58
Yeah. This case, you have a source file. The solution is basically the training script of, uh, uh, MiniPG, I assume. Like, eval command is, I think Dhruv also mentioned, and this is additional guidance or prompt that will guide the whole, uh, search process. And the metric is a validation BPB. Uh, this is like a holdout, um, uh, loss, like a loss-only holdout
- 22:28
set. Uh, goal is, um, minimize, and, uh, others are just, uh, configs we just, uh, mentioned. Oh, sorry. Eval command is actually the, the, the batch command that, uh, you're going to run the, uh, uh, run the eval. And eventually, it will output a, uh, score. Like, it, it print a score to the terminal, and then it's going to be parsed by the agent. And that's the key,
- 22:58
uh, signal that will drive the whole trial and error loops.
- 23:05
Yeah. Um, in meanwhile, I can also show you the dashboard. So if you go into the Weco dashboard, um, and I think there I, I did some basic run, and you, you will basically see a lot of things. And, uh, after the, the command, like, auto-booted by the agent, um, I think then you, you will see the run being popped up, and, uh, then you can try out the features we
- 23:35
provide.
- 23:39
Yeah. Um, do we have a run just to get started?
- 23:45
Yeah. I think it's just starting.
- 23:47
Okay. Still running, saying, um, yeah, Claude is working.
- 24:02
Okay, so we each do the Claude code. Uh, question from the audience while you guys are doing that. Um, one, is it possible to turn up the brightness on the dashboard or switch to light mode?
- 24:17
Sorry, again
- 24:18
I'm, I'm right here. Can you turn on light mode on the dashboard? It's a little hard to see.
- 24:23
I see it.
- 24:27
Uh, light mode. God, where is it? Is it f- here? Yeah
- 24:41
Uh, second question is, can AutoResearch apply to RL? What is the scalar metric when there are multiple things to optimize?
- 24:51
What is the-
- 24:53
Can AutoResearch apply to RL? What is the scalar metric when there are multiple things to optimize?
- 25:00
I see. Yeah. Um, it can definitely a- be applied to RL, like a applied to RL training scripts. Um, I think there's a, um, derived project called the Sokoban, a speed runner that's actually exactly doing it, like a, for post-training a QLearn model, uh, for playing a game. And, uh,
- 25:30
yeah, in that project, you can apply AutoResearch to the, uh, RL, RL training scripts. And, um, the second question, you have multiple objectives to optimize for. Um, the idea is you eventually just combine them to a single metric, like either apply different weights on those different metrics, or you treat some of the metrics as constraint
- 26:00
and optimize for one of them. For example, if you're doing things like fraud detection, you might hold the, uh, accuracy and trying to optimize the precision. Um, and, uh, for Parameter Golf exactly, you hold the model size as a constraint, and you optimize for the accuracy of the system.
- 26:26
Yeah, we can see the, um, agent already generated, uh, one node. So the zero, the zero node is, um, basically the root node, the baseline solution. And, uh, three point nine three means the validation BPB attached to that solution. And, uh, you can see the WeCo's AutoResearch already generated, uh, one node here.
- 26:55
Um, um, here we also have a summary of the, uh, ideas it's tried. And, uh, seems like already get improvement. Oh yeah, you can also-
- 27:10
We can also open the code and see what's actually been changed.
- 27:18
Yeah.
- 27:22
Do we have the steerability?
- 27:24
Okay. Uh, for steerability, you can basically tell the agent, for example, say like, uh, explore the quantization method.
- 27:33
Is, is, is it connected?
- 27:36
Yeah. Uh, quantization method, uh, do not change the-- do not just change the model config because we don't, like, just want to do the hyperparameter tuning. We also want to try, like, more aggressive method.
- 27:58
I don't think it's connected.
- 28:00
Oh.
- 28:07
I think, uh, Claude Code didn't follow the instruction.
- 28:17
Yeah, basically, we also support you bring your Claude Code agent to our dashboard. We call it a bridged, uh, session. Um, so basically, yeah, you can, you can see the dashboard, uh, the tree structure, the metric, and the, uh, Claude Code at the same time. And then you can tell Claude Code, "Okay, this is the, uh, the direction I want to steer the whole, uh, search
- 28:46
process to."
- 28:48
Um, checked in with some of you. I think internet might be a little better. Uh, some of you guys are using hotspot. I know there's some modal payment issues. We're gonna move on to the final part, which is the competition aspect. Uh, you don't need modal for this. And originally, this was designed to be a thirty-minute competition, but we will give you guys until the end of the day. Um, a little bit more on what this is.
- 29:19
You guys have all played Flappy Bird. Um, we're gonna do a Flappy Weco Cup. So essentially, everyone hopefully has played Flappy Bird. You tap to flap, you dodge the pipes, and you fly as far as you can. And your final score is dependent on how far you fly.
- 29:40
We have turned this into an AutoResearch problem. So instead of playing it, there's a controller called should_flap that decides every frame whether the bird should flap. Your job is to optimize that controller. The metric is the average score, and you want that as high as possible with the highest score winning.
- 30:04
With all games, there's a twist. Uh, we're not just running this on Flappy Bird. Uh, we're running this across four different worlds. So the first world is normal, which is the classic game. The second is moving gap, where the pipe gap drifts up and down. The third world has wind pushing the bird around, and the fourth world is variable, which mixes all the conditions. Your score is the average
- 30:34
across all four of these worlds, so you can't just focus on one. Um, a controller that does well in a normal game but dies in the wind won't win, and the leaderboard scores you on a hidden validation set that you never see, so you can't obviously overfit on the public game.
- 30:55
A few rules, uh, mostly so it's fair and it doesn't break. Uh, you only... And we will share the GitHub repo for this shortly. Uh, you only edit the controller, which is ai_controller.py. You keep sh- you should keep should_flap signature exactly as it is, so you can't touch the game or the evaluator. No imports, external libraries, file I/O, so we're keeping this self-contained. And in the spirit of keeping it fair, uh, no parallel compute
- 31:25
all- allowed, so you would be using WeCo for AutoResearch. Um, everyone is defaulted to the same model, Opus 4.8, so it's a level playing field. And remember, the local score that you see on your laptop isn't going to match the leaderboard because of our hidden test set.
- 31:46
Some tips. Um, you'll get about 50 steps on WeCo. Uh, actually, this is with the 30-minute times- timeframe, so, uh, I guess you can run unlimited steps until the end of the day. Um, can start from the baseline. Uh, the baseline already works, so improve the code rather than rewriting the controller. Uh, think about making small, focused edits, um, things like tuning the safety margin,
- 32:16
the velocity keyboard, the bottom buffer, adjusting the behavior per world, and, um, maybe implement some of the features we talked about today, whether it's inspecting nodes that scored well, building on them, bringing in ideas from outside, um, and trying several directions.
- 32:37
This is the instructions for the game. Um, by the way, you guys are playing for a Nintendo Switch. Uh, the way that we're gonna determine who wins is, for your team name, put in your email, and by the end of the day, the highest score, uh, we will reach out to the person that that is.
- 32:59
In the meantime, we're gonna continue answering questions from our Slido, or, uh, please raise your hand if you have any questions. Okay, we're just about to wrap up here. Um, scores look like they're coming in. I don't know who [REDACTED] is, but a little suspicious because there's only one node, and so we will be inspecting your runs to ensure that there's no cheating going on. Um,
- 33:30
nice try. Uh, we will be opening this competition, uh, till midnight tonight. Um, we will find a way to get to you. You can put in your email, or we're also, we also have your emails, uh, from the runs that you guys are doing, or you can just contact us on our website. But, um, feel free, feel free to keep going at it, and, um, thank you all for coming.