An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

[music]

Okay. Hello everyone and thank you for

attending this session. My name is Tim

Sweeney, a principal engineer at Weights

and Biases and Coreweave. And for the

next 20 minutes, we're going to talk

about Arya, our new AI research and

iteration agent. Let's go ahead and get

started.

So, uh, first off, just by way of making

some noise, some clapping, uh, who here,

um, identifies as an ML researcher?

You're someone that trains models,

trains the brain?

I heard one. Wow. Okay. Great work.

Great work. Uh, what about who here is

the applied engineer, the namesake of

this conference? Who here actually

builds the bots?

>> Okay, good. Expected much more. And who

here is in AI management? You are

helping fund this compute.

Okay. Okay. Nice. From the back. Lovely.

Um, well, now that I know a little bit

about you, just a little bit about me.

Uh, again, my name is Tim. I have a

masters in machine learning, uh, and

reinforcement learning from Georgia

Tech. So, I've been that, uh, researcher

currently building Weights and Biases

agent, Arya. So, identify as that

applied engineer. And in a previous life

was the PM of Twitter's ML stack. So, I

hope you hopefully can connect with you

middle management as well. [laughter]

>> [gasps]

>> Um today's agenda is kind of broken into

three sections and hopefully each of you

personas walk away with something

valuable. So first we're going to learn

about Arya itself and how it can

supercharge your AI and ML workflows.

We're going to dive into auto research

and see that live in a live demo in just

a moment.

Then we're going to pull back the

curtain and learn how we use weights and

biases and uh coreweave to actually

build Arya because a lot of you in the

audience are building agents yourself

and we believe a lot of these components

can help you in your endeavors. And then

towards the end we'll just take a step

back and identify a few key tips and

tricks for making sure that you're able

to productionize your systems

effectively.

For those of you who might not be

familiar, Weights and Biases is the

world's leading AI development platform.

We've been in business now for nine

years and have happily joined the core

family about a year ago. Uh we have a

number of products in our suite but are

really known for our models training

inference and weave stack which really

helps collect data uh about the AI

development and machine learning

workflows and makes that information

actionable and uh enables users to make

the best decisions about what to do

next.

So without further ado, let's go ahead

and dive into Arya, our agent. Uh we'll

show a demo and then we'll get back to

some slides.

Okay, beautiful.

Let's make this a bit bigger. Holler at

me if you need it to be bigger. So, uh,

what you're looking at here is a weights

and biases workspace. For you, for

anybody that isn't familiar, on the

lefth hand side, I actually see a list

of a bunch of different experiments. In

this particular project, I have over 200

training jobs. And on the right hand

side, I see a scatter plot of, in this

case, declining metrics, which is good.

means our loss is going down over time.

And this view would be very familiar for

anyone that uses our tool. Now, to

ground this, we're actually uh uh using

the Carpathy Auto Research project,

which I'm sure many of you are familiar

with, but if you're not, it's just a

very simple project that trains an LLM,

and it's a great foundation for auto

research type demonstrations because

it's a very simple codebase and allows

us to improve iteratively over time. So,

let's jump back to the project and open

up Arya by clicking this blue button in

the upper right. When I click this

button, I'm uh presented with the

familiar chat interface with, you know,

how can I help you today, a few call to

actions, and you know, I can add

different context in my project or maybe

add images, etc. Um, everyone here is

agent builders, so I don't need to bore

you with the details of what an agent

interface looks like. But let's go ahead

and just, you know, enter in a basic

intro here. Let's say, "Hello, Arya.

You're on stage at AI World's Fair 2026.

Please introduce yourself." So, it's

going to go ahead and chug along and

hopefully emit some sort of nice emoji.

Yay. He I'm Arya. I'm talking to the

audience. Great. But now, let's dive

into the meat of why you came here. So,

I'm going to open up this chat here. And

this is a longunning chat where I've

been running again over 200 experiments

using the auto research loop. Um, it

helped me download the code, set up my

launch jobs, set up my GPUs, and is able

to autonomously iterate on the code

itself and the hyperparameters.

We'll take a look at what it's doing in

a moment, but while we're doing this,

I'm going to kick off a live iteration

right here. So, what I'm going to say is

please conduct another batch of

experiments. You are on stage at the AI

Engineer Worlds Fair 2026 and we're

hoping to find the best model live. I

believe in you. Uh, because we know we

have to encourage our models. Um, so

it's been doing this for a while. What

it what it's doing here is it's saying,

"Okay, great. Um, I don't want to make a

big architecture swing. That feels a

little bit too risky." So, it's probably

going to go for uh some modifications to

the hyperparameters. And then it's

kicking off a shell call here that is

actually um executing that uh executing

that experimentation loop. And we're

going to check in on this periodically

throughout this presentation.

But I want to help explain what's going

on behind the scenes. So behind the

scenes I have set up a weights and

biases launch queue. Launch is our our

product that allows you to connect your

compute clusters and allows humans and

agents to launch longunning

experimentation jobs particularly by

leveraging GPUs.

Here I'm looking at a uh a terminal

output of my Kubernetes cluster where

we're actually seeing live execution of

experiments happening. So this is

happening live right here. This is not a

fake demo. Um great. And if we jump

back, we see that at this point it

started the cues and now it is simply

polling and waiting for our work to be

complete. So we'll jump back to that in

a in a moment. But before but let's dive

into a few other examples. So uh

something else that is interesting you

can do is maybe you might want to ask it

something like please summarize the

highest performing runs in this project.

This use case would be something like

maybe a new user come or a new uh team

member is joining your project and want

to understand the research. Um or maybe

you've uh someone's been doing some work

while you were on PTO and you want to

get caught up. We'll see what this comes

up with in a moment. Some other

pre-anned uh examples are finding

patterns in your project. So here we can

see that I asked it, hey, can you find

some patterns in this research? And we

see that um it identified that a new

family of models emerged as the as the

auto uh auto research was happening. Uh

it identified that batch size seems to

be a really high high uh lever uh

parameter. It identified an

architectural recipe that seemed to be

quite promising and a number of other

insights that would have taken me hours

or days to discover on my own. And Arya

is able to do it right for me directly

in the interface that I already live.

Not only is it able to emit text based

uh textbased outputs, but it also deeply

integrates with a number of weights and

biases visualization utilities. So here

I've actually asked it to emit a weights

and biases report which for those who

aren't familiar is essentially a

markdown file on steroids. It's got uh

embedded embedded plots, charts and and

and graphics. And so here uh you know

it's talked about the thesis of the

project. It's it's emitted a number of

of data panels. And uh I actually think

it's quite interesting. It used um one

of our more esoteric panels, the uh

parameter importance chart to uh tell me

the correlation of various different

parameters within this uh within this

training job.

Uh in addition to uh reports, it's also

great at working with workspaces. So if

you're a weights and biases user, uh you

spend a lot of your time uh designing

and working with workspaces. Well, Arya

is actually customtuned and prompted to

really understand how to build

workspaces, build plots, and complement

that that data analytics with real live

graphics using the built-in proprietary

charts that weights and biases users

know and love. Um, so with that, let's

go ahead and check back on some of our

our prompts. We can see that the please

summarize this project prompt is cooking

away. It's querying weights and biases.

It's applying patches. It's writing its

own code. So, we'll come back and check

on that in a moment. and our longunning

training job is uh still pulling for the

results. We can see that we're cooking

away on our GPUs. So, we're we're frying

some GPUs [music] and doing some data

science all live. And while that's

cooking, let's go ahead and jump back to

the presentation. We'll come back in a

moment.

Uh

oh, no, we're not looking at a

dictionary. We're looking at a PO.

Great. Uh okay, so quick recap here.

What did Arya show? What did we show in

these last five minutes? First, we show

that uh Arya can serve as your data

science companion right inside of

Weights and Biases, helping you discover

insights that you wouldn't you wouldn't

be able to discover as your experiments

and as your team size grows.

Next, we address the problem of

complicated reporting and complicated

plotting. Weights and biases users are

are really want to turn their insights

into visual communication tools. They

want to communicate with their peers and

their colleagues. So Arya's built from

the ground up to understand those

primitives and help co-pilot and drive

right along right alongside in the UI

and announcing now today for the first

time we are releasing Arya on our iOS

device or on our iOS app. So uh uh Arya

released on Monday and our iOS app now

has Arya built in. So if you're

conducting hyperparameter tuning jobs,

if you're training models, or if you're

just researching within the weights and

biases ecosystem, you can go touched

grass at Yerba Buena uh gardens and

steer your uh hyperparameter tuning jobs

all from your mobile device. And what is

this all building up to? This is

building up to an a fully automated

endto-end research platform where we're

not seeking to replace uh RL

researchers, but complement your

workflows. Arya's great at orchestrating

jobs, understanding GPU workloads,

responding to events within the within

the Wandi ecosystem, and listening to

researchers, uh, uh, looking up archive

papers, and collaborating on hypothesis.

So, we can let Arya drive the mechanics

that you don't want to deal with while

you focus on the new ideas, new

architectures, and new parameters that

you wanted to try.

Um, great. So, that's Arya in a

nutshell. We're really hoping that you

give it a shot. And uh we'll jump back

to the auto research at the end and see

if we got a new best record. But before

we do that, let's talk about how we use

weights and biases and coreweave to

actually build Arya. So now speaking to

a lot of the the AI agent builders in

the room, here's a quick architecture on

the lefth hand side. You see that we

have a web client, iOS client that

communicates with our API server that

then dumps data into our turn database

and is worked on by our harness, our our

worker harness. This is sort of

archetypical of probably what most of

you are all building in the room and is

exactly what we have on our back end.

But that harness worker is a magic is a

is a magic box and it connects to a

number of important utilities. First is

a sandbox where it can execute arbitrary

shell calls uh do do Python data science

etc. And we invite you to try coreweave

weights and biases sandbox to fit into

your architecture.

Next up you need an LLM provider of

course and so if you're maybe using GLM

5.2 two or one of your fine-tuned

models. We invite you to use uh weights

and biases inference and connect that to

your worker as well.

If you're like us, you need to run

longunning workloads outside of the main

loop of the agent where you're actually

training for day for sometimes days at a

time. Weights and biases launch can

actually help facilitate that and

coreweave GPUs can help make that

compute even better.

And then lastly, and really most

importantly, we need an observability

layer. It's critical that your agents

are able to log out their what's going

on with their sessions, their turns,

their tool calls, any errors they're hap

that that's happening, etc. Uh we have a

product called Weights and Biases Weave

that we log 100% of our traces to where

us and our team can learn from. And

that's where we move from production to

offline where our team is able to use

Weights and Biases Weave to drive

insights and identify behaviors,

implement tasks with tasks which are

essentially unit tests for your models

and evaluate those models in a loop.

We have a model repository which you

might choose to use weights and biases

artifacts to store your agents or models

and you we emit our evaluation results

to weave where we have a common

dashboard that we can make go no-go

decisions on various prompt changes or

architectural changes that then feeds

into a research loop which we call our

improvement loop where we form

hypotheses implement candidate agents

and analyze the evals. So we have two

sort of complimentary yet adversarial

research loops going on going on offline

feeding data from weights and biases

weave ultimately to identify the best

model so that we can promote that to

production through our registry and

close the data flywheel. So in the next

just uh three seven minutes or so we'll

just talk about uh weights and biases

weave and show how we as a team actually

use weave to facilitate this workflow

and we believe this is something that

you would benefit from as well all of

you agent builders in the room.

Yes, another demo. Great.

Okay. Okay, we have new responses. So,

it's going to be exciting when we open

this up later. See if uh we've got some

better metrics. Um, okay. Let me zoom

out just a little bit here. So, here I'm

looking at the agent dashboard. This is

the live weights and biases agent or

Arya agent dashboard uh built in weave.

Man, that is a lot of uh branded

buzzwords there. This is the dashboard

that you would get if you use our tool.

and uh you have a you know uh span

volume, conversation volume, token

tracking, etc. Think of this as like a

uh a bird's eye view of your agent. For

me, however, I really like this

conversations view, which I do have

pre-loaded in this tab. This

conversations view is a live feed of all

of the conversations that are going

through Arya, but it's filtered down to

just the internal employees. So, it's a

little bit of a of a reduced set here.

Um what I what I love is this middle

spans view which gives me a visual

indicator of the topology of a trace.

Different colors and and shapes indicate

different things that are happening

within the agent. So things like tool

calls, LLM calls, thinking blocks, etc.

which really help me understand again

the shape and topology of that

particular conversation. I can of course

open up one of these conversations and

view our our conversation view where I

can see the system prompt, the user

message, shell calls, reasoning blocks,

etc. This is where my research lead,

myself and my PM go to add notes, add

feedback, add emojis, and talk about and

discover those insights and those

behavioral nuances we spoke about

earlier so that we can turn them into

tasks.

Arya's built in to the weights and

biases system as well. Here you'll see a

summarize button and these are sprinkled

throughout the weights and biases

application. I simply click summarize

and we start a new chat contextualized

to the thing that I'm looking at. So it

it sees this and says give me a brief

summary of this particular conversation.

So if you if you're paying a attention

closely, you'll realize that what we're

doing is using Arya to analyze Arya's

own conversations to then make

recommendations about how to improve

Arya all within the UI.

Um okay, great. While that's cooking

away, I want to show you the last item

uh within the Weave ecosystem here, and

that's signals. We've heard a lot today

about the value of evals and the value

of LLM judges. Weave actually offers an

integrated LLM judge experience. So

here, if I zoom out a little bit, you'll

see that I have a user frustration

signal, a lowquality response signal,

ask user signal, etc. These are LLM

judges that run live against against our

live traffic. And we can see various

different signals like user frustration

moments or lowquality responses. These

help our team identify these clusters of

behavior for us to go fix in next week's

iteration. Let's go ahead and do a live

look and see what it says. Um this says

the user explicitly states that I'm not

satisfied with the loss curve. It looks

bad and it apparently that indicates

frustration. So here we can see an LLM

judges live reasoning for why that

particular flag was uh indicated.

Uh let's see, four minutes left.

Perfect. Um so, uh with that, I've been

using the term task a lot. And so what

we're do, what I've showed so far is is

this live production loop where we are

are are are tracing our our prod logs.

We're looking at them as humans, maybe

even using LLMs to complement that

analysis. And what we end up doing is

transforming those into tasks. Now, this

gets a bit technical here, but our tasks

are all described as YAML files. You can

think of a task as essentially a unit

test for your model. So here we say we

have a an example user prompt that says

check this run and that run. Both of

these are giving good results. What can

we learn from this? What's the

difference? So this is an example of

something we want Arya to be good at for

all of you. And after the uh requisite

metadata we see that we've defined an

LLM judge. So here we've defined what

correctness means in the context of that

question.

And we've then we've defined a second

LLM judge that determines if the

insights are actually interesting.

[laughter] And then we've uh defined a

third rule-based judge that says were

you able to actually generate a result

within just six tool calls meaning it

got there with some degree of

expediency. These are all then clustered

together into we have about like 200 of

these. They're all clustered together

into an eval suite that runs nightly.

And again we use weave to track all

those evals. So here, I know it's a bit

small on this screen, but what you're

looking at is a listing of every night's

eval. This is literally two nights ago,

the evaluation for our candidate model

got 73% on our production or on our eval

suite against the 72% that our prod

model got, which means we're definitely

going to push that forward uh this

Friday. Uh and we can see a kind of a a

performance plot on the right. So these

utilities are what you would get out of

the box if you're uh if you decide to

pick up weave and use this tool. Um,

jumping back to the last conversation we

had where it asked me where we asked,

uh, can you please give a quick summary

of this trace, we see that it actually

analyzed the conversation, understood

what the user was doing, and then

ultimately decided that this was a

pretty strong trace. Um, let's see,

we've got two and a half minutes left,

so let's just quickly recap here. Uh,

first off, uh, what we use weave to do

is a, collect production traffic. Super

critical to collect all of your

production traffic so you can learn and

iterate. Secondly, we use it to generate

insights both as humans as well. We we

do it as humans. We use Arya and we use

LLM judges to identify those behavioral

nuances. We then enrich our tasks. We

implement models and we evaluate using

weights and biases weave as a shared

dashboard where we can make decisions

together as a team that then ultimately

allows us to promote the best model

forward with confidence.

So speaking of confident

productionization, let me speak uh

briefly to the managers in the room. So

a few tips for being successful here.

First is um invest in agent-oriented

observability. Uh I'm a bit biased. I

believe that weights and biases weave is

the uh observability platform of the

future. Uh but pick your favorite

flavor. Whatever it is, log your

sessions, log your turns, log your tools

and feedback. This introduces an ability

to catch a new class of bugs in our

world called behavioral bugs. Not

exceptions, not performance, but

behavioral bugs.

Next up, tasks and evals are the new

world of CI. You've heard a lot about

this. If you are a software engineer,

you've written unit tests your whole

life. You must develop a practice where

your researchers are sitting on the same

scrum team as you developing tasks and

you're viewing the performance metrics

as true go no-go decisions. But in order

to complement that, you must use humans

as a necessary judge. There are

behavioral nuances that LLM will not

catch. You must be using your product

and you must be manually reviewing these

traces as a team at the end of the week

on a board looking at the best and worst

traces to understand how your model is

performing.

And then lastly, um just maybe one one

more tip is to add value through context

and tools. It can be really tempting to

uh try to overengineer the harness and

do a bunch of creative stuff around

memory and things like this. We found

that a a lot of lowhanging fruit can be

ascertained through simply giving your

agent context about your business

domain, the underlying uh primitives

that you have available and your

particular uh business data. Um so with

that, let's go ahead and check in on our

uh our our research agent here and let's

go ahead and toggle our workspace. And

what we should be seeing is yes indeed a

little dot that uh oh okay our previous

dot which was done at lunch was 5.83.

831. This got 5.833. So we were right on

the edge of having a live improvement,

but pretty darn close. Uh so that's what

the uh that's what the model was able to

produce. It actually uh ran uh quite a

few tests here. I see I'm over time, so

I will click close pretty soon. But we

ran 12 different experiments within that

experiment batch and uh we'll be running

more all night. So please try out Arya,

scan the QR codes, check out the docs.

Uh we really love to see what you do

with it and um looking forward to

serving you. Thank you very much.

>> [music]