An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

A research workspace can accumulate experiments faster than a person can explain them. Tim Sweeney, Principal Engineer at Weights & Biases by CoreWeave, opens a project with more than 200 training jobs. A scatter plot shows loss declining over time. The next problem is deciding what those runs teach—and what experiment to run next.

The project uses Karpathy's autoresearch, a small codebase that trains an LLM and provides a manageable setting for repeated changes. ARIA, the AI research and iteration agent, opens as a chat inside the workspace. Project context and images can accompany a request, so the conversation can begin with the experiment data already close at hand.

The useful starting point is a long-running conversation rather than the agent's onstage introduction. In that conversation, ARIA has helped download the code, configure Launch jobs, and set up GPUs. It can then iterate on both the code and its hyperparameters. Those are different kinds of changes: one alters the implementation being trained; the other changes how that implementation trains.

Sweeney asks ARIA to conduct another batch of experiments and try to find the best model live. He adds encouragement—because, naturally, the models need encouragement. The consequential decision comes next: ARIA avoids a large architecture change, treating it as too risky for this iteration. Sweeney expects it to try hyperparameter modifications instead, and a shell call starts the experimentation loop.

W&B Launch supplies the connection to compute. A Launch queue lets humans or agents submit long-running experiment jobs to a cluster, including jobs that need GPUs. Sweeney opens terminal output from his Kubernetes cluster to show experiments executing, then returns to ARIA, which is polling while it waits for the work to finish. The chat initiates and monitors the work; the cluster performs the training.

Where does the time-consuming work happen after a research request? The flow below separates the agent's decision and shell execution from queued GPU training. The polling path makes the separation visible: launching a job does not mean its result is immediately available.

While training runs, a second request asks for a summary of the project's highest-performing runs. This is useful when a teammate joins a project or returns from time away: the experiment history needs to become an explanation of what worked. A previously prepared pattern-finding example goes further, identifying an emerging model family, batch size as an influential parameter, and a promising architectural recipe. These are leads for further research, rather than a demonstrated guarantee that any one change causes an improvement.

The analysis can become a native W&B artifact rather than remaining a chat answer:

• Reports: A report combines Markdown with embedded plots, charts, and graphics—Sweeney calls it a “markdown file on steroids.” The example includes a project thesis, data panels, and a parameter-importance chart showing relationships among parameters.
• Workspaces: ARIA is tuned and prompted to construct workspaces and plots using W&B's built-in chart types. That gives the analysis a form colleagues can inspect in the interface they already use.

A check-in shows the summary request querying W&B, applying patches, and writing code, while the training conversation continues polling for results. Analysis and training therefore have their own work to complete. ARIA's value here comes from connecting research data, executable tools, and visualization primitives within the same product.

The iOS announcement extends that workflow to a phone: researchers can steer hyperparameter-tuning jobs while away from their desks. The larger ambition is an end-to-end automated research platform that handles job orchestration, GPU workloads, research lookup, and hypothesis collaboration. Sweeney frames the division of labor around letting ARIA handle mechanics while researchers concentrate on new ideas, architectures, and parameters.

ARIA's backend begins with a familiar sequence: web and iOS clients communicate with an API server, the server writes to a turn database, and a worker harness processes the work. The harness then connects to the capabilities needed to carry out a research request.

Those connections serve distinct purposes:

• Sandbox execution: Shell calls and Python data-science work give the agent a way to manipulate code and analyze data.
• Model inference: An LLM provider supplies the model calls used by the worker; W&B Inference is presented as one option.
• External training: Launch handles workloads outside the agent's main loop, including training that can take days, with CoreWeave GPUs supplying compute.
• Observability: Sessions, turns, tool calls, and errors are logged so the team can inspect what the agent actually did. Sweeney says ARIA logs 100% of its traces to W&B Weave.

The trace store also feeds development. Offline work turns observed behavior into tasks, evaluates candidate agents, and records results in a shared dashboard. One loop improves the tests; another forms hypotheses, implements candidates, and studies their evaluation results. The loops are complementary but also adversarial: better tests can expose weaknesses in a candidate that previously looked promising. The selected agent returns to production through a registry.

The Weave dashboard gives a bird's-eye view through span volume, conversation volume, and token tracking. Sweeney then opens a conversation feed filtered to internal employees. Its span view makes trace topology visible: different colors and shapes distinguish tool calls, LLM calls, and thinking blocks. Opening a conversation reveals the system prompt, user messages, shell calls, and reasoning blocks behind that shape.

That detailed view is where the research lead, engineer, and product manager add notes and feedback before turning behavioral discoveries into tasks. A contextual Summarize button can also start an ARIA conversation about the item being inspected. In this example, ARIA analyzes one of its own conversations to recommend improvements to ARIA. The agent assists the review, while the team still has the underlying trace available.

Signals add automated triage to live traffic. LLM judges flag user frustration, low-quality responses, and other behaviors, helping the team find clusters to address in a later iteration. One displayed judge identifies frustration because the user expresses dissatisfaction with a loss curve. Its reasoning is inspectable, which gives reviewers a concrete explanation to assess rather than just a flag. The flag identifies dissatisfaction; it does not by itself establish what caused the poor experience.

A task is a YAML file that acts as a unit test for the agent. The demonstrated prompt asks ARIA to compare two runs that both give good results: what differs, and what can be learned? The task pairs that request with definitions of success. This makes a broad product expectation—give useful research advice—specific enough to evaluate repeatedly.

The example checks three separate properties:

• Correctness: An LLM judge evaluates the answer against correctness defined for this particular question.
• Interesting insights: A second LLM judge asks whether the comparison produces insights worth knowing. A correct answer can still be unhelpful.
• Expediency: A rule-based judge checks whether the agent produces a result within six tool calls, giving efficiency an explicit test alongside answer quality.

About 200 tasks form a suite that runs nightly, with results tracked in Weave. The displayed candidate scores 73%, compared with the production model's 72%; Sweeney says the team plans to move it forward that Friday. These scores describe performance on the team's suite. The recording does not establish whether the one-percentage-point difference is statistically significant or predicts improvement outside that suite.

How does a production conversation influence the next release? The cycle below shows the intermediate steps: review produces tasks, tasks test candidates, and the results inform a team decision. Production traces provide the starting material; they become useful release evidence only after the team defines the behavior it wants to test.

The production advice follows directly from this development loop:

• Instrument behavior: Log sessions, turns, tools, and feedback to catch behavioral bugs. An agent can complete its calls without producing the behavior a user needs.
• Use evaluations for decisions: Researchers and engineers should develop tasks together and treat performance metrics as go/no-go evidence.
• Keep human judgment: Use the product and review its best and worst traces as a team. LLM judges miss behavioral nuances, so automated scores cannot cover the whole review.
• Improve context and tools: Before adding elaborate harness or memory machinery, give the agent the business domain, available primitives, and relevant business data. Sweeney's team found substantial opportunities in those simpler additions.

The final check returns to the batch started onstage. A new point has appeared in the workspace, and the batch has run 12 experiments. The reported result is 5.833, close to the earlier best but without beating it. ARIA has carried a request through experiment selection, shell execution, queued GPU training, waiting, and a visible result. The automation completed a research iteration; an improvement remained something the experiments had to earn.