Your Coding Agent Should Do AI System Engineering

[upbeat music] Hi everyone.

As you heard, I'm Ben from Hugging Face, and the talk that I'm gonna present to you today is called Your Coding Agent Should Do AI Systems Engineering.

So there are two main takeaways that I want you to get from this talk. One, and, and probably the fun part, is that we can use coding agents to tackle the hardest engineering problems in AI, so systems engineering and machine learning engineering.

And maybe the boring part is that in order to do this, we're gonna need standard repos, um, we're gonna need those on the hub, and in many cases, we already have them.

So I think in this case, I'm preaching to the choir here, but in case you haven't noticed, coding agents have been accepted. Many of s- of, of us have been using them for a few years.

But in the last few months, they seem to have cro- crossed a sort of acceptance gradient where a broader group of people are using them. So with this in mind, how do we keep our careers, our engineering kinda contemporary, and how do we keep challenging ourselves in new areas?

And my proposal is that we need to go kinda closer to the silicon and tackle harder problems, and, and that's where AI systems engineering comes in. I've broken this talk down into three progressively c- more complex steps, and more autonomous steps as well, and I've defined those like three bosses from games.

The first one is a hybrid approach where you interactively use an agent to solve a, a... to, to write a CUDA kernel. The second is a zero-shot task where an agent takes a prompt and trains an LLM on Hugging Face.

The third is a multi-agent auto research setup, like a kind of automated AI lab.

So let's get started on, on the first boss, right? This is writing CUDA kernels. So for a while, writing custom kernels was seen as this unattainable goal for the humble agent.

They required complex DSLs. They required integration with rele- relevant hardware to be benchmarked and to be tested, and it was s- seen as something that couldn't be, uh, achieved by agents.

However, that in most cases was wrong. If you look at kernel hackathons like those on GPUMode, the recent AMD hackathon, if you look at papers like KernelBench, you'll see that agents are able to write valid and optimized CUDA kernels.

And, and that's really cool and something that totally inspires me. I, I'm a part of GPUMode, I contribute to that, and, and something that I think everyone should be doing.

However, what do we do with them? How do we distribute them, and how do we get them into our inference engines so that we actually are using these optimized kernels that we're generating?

And that's part of the question of this part of the talk.

Let's take a step back now and just, uh, say what a kernel is, right? So when you run an AI model on a GPU, the actual work is executed through a kernel.

This will be defined in a relevant language for that hardware, and it will use relevant features to that hardware that may not be r- available on other hardware.

We can write custom kernels that will take advantage of that hardware for a specific math operation, kinda squeeze everything we can out of it, so that the model will infer faster.

In general, this requires a lot of expertise about writing CUDA kernels, about the hardware, uh, and it's also a bit of an installation hell as you deal with a pretty large install matrix from hardware to software to generations and versions of, say, CUDA and these kind of issues.

So in short, it's hard. Efficiency in deep learning, so efficiency in kernels, is split into three main sections. One, compute, two, memory, and three, overhead. Compute is the flops.

This is... These are the matrix multiplications and, and the, the real math of the process. Memory is the time spent moving data or tensors around memory, typically from slow to fast memory.

And overhead is basically everything else, the Python environment, PyTorch dispatch of those kernels, these kinds of things. In general, most people might assume that the compute is the bottleneck here because it's doing most of the math, right?

Th- that's not correct. So in most cases, memory is usually the bottleneck, and that's because a modern GPU, let's take a H100, for example, can do a petaflop a second of computation, but its memory bandwidth is three terabytes.

So in short, the GPU is often waiting idle for this tensors to come back for it's... for them to be computed.

There are custom kernels, custom optimized kernels that exist, uh, p- FlashAttention being the poster child of these, and in general, what they do is increase arithmetic intensity. They basically make the GPU do more sums at once per read and write.

So we move the tensors across, we do as much math as possible in the GPU in one go, and then we write it back. In short, people like to say, "We keep the GPUs warm."

And that's the objective of writing a custom CUDA kernel.

Hugging Face has a library called Kernels, which is maintained by kernel writers, and we're beginning to scale up to a kind of agentic workloads. So at its core, this is a way of distributing kernels.

It has a TOML file, like any kind of project, which says which hardware it works on, which versions of CUDA and other kind of softwares it requires to work, and it's a...

it's now also a repo on the hub, just like models. So if you are a kernel writer or you're an aspiring kernel writer with an agent that you wanna set up, you can now be a kernel publisher just like a model publisher.

And my point is that this is like a kind of super fervent ground for, uh, AI engineers looking to kind of scale their career.

If you check out these repos on the hub, you'll see that there's compatibility for different hardware. You can configure that so you'll know, like, "Okay, this works on my GPU or on my laptop."

Uh, and, and this is what it looks like here

Right. Let's take a look at what this looks like for an agent and how we're helping an agent to do this. So

first, we're gonna go to how we do this, so skills. So I, I, I'm sure everyone here is familiar with skills, and, and I'm sure there have been a number of talks that really go deep into skills.

I don't like to... I like to keep them pretty simple, and, and really they're just kind of file-based context with all the wonders of files. We can open them and close them, we can version them, we can source control them, and these kinds of things.

And agents can also do the same. They can open them when they need them. They can use them when they don't. And so in the context of kernels, that means that we can give examples of how to write and how to use kernels in skills, and they can open those and use them when they need.

I like to say that it takes a task from being zero-shot to being few-shot, which in ML is quite a familiar concept, right? We're just giving the agent examples of how to do things, uh, and, and we can be quite verbose and descriptive about that.

At Hugging Face, we're focusing on integrating skills into their projects. So what you'll find is that inside each project, there's managed skills by that project, which we think is the best way to do this because it means that those projects, uh, the, the maintainers of those projects are maintaining their skills, right?

That means that they're not necessarily the most, like, YOLO skills because they're kind of, like, well-maintained and, and robust. And we have another repo for those kind of more experimental skills, which is called Hugging Face Skills.

Go and check that out, uh, if you want to try some of these examples you'll see today.

In kernels, this is what the skill looks like. It focuses on benchmarking, so it has scripts that allow you to, uh, benchmark and test the skill... Uh, sorry, to test the kernel and see, uh, how performant it is, and references with examples of, of how to do this.

We benchmarked this skill, and we used, uh, we generated a, a kernel for Qwen3-8B for H100, and we found that we had a 94% speed-up. This isn't a state-of-the-art speed-up on this model by any means.

It's really just about compatibility and a compatibility matrix. So in many cases, these models and their kernels won't be optimized for the respective hardware or generation of hardware that you want to use them on.

So you have some low-hanging fruit here where you can just come and pick up some, some optimizations for that specific hardware. Maybe because your hardware's cheap on your cloud provider, uh, but it's not necessarily the most ideal for the, for that model that you're using.

So my recommendation would be to come here and, like, pick up some easy speed-ups.

How do we know that these skills are any good and, and, and that we should be sharing them and telling people to use them? We use an open source library called UPskill that we're also maintaining.

This is a, is a gateway to using cheaper and open models with skills. Y- it basically just generates skills, generates an eval for the skill, and then allows you to compare different models on the same skill.

So you can see things like this. So okay, GPT-OSS is slightly less accurate using the same tokens. Kimi is more accurate using less tokens. Haiku is a bit more accurate using less tokens, and these kinds of things.

So if you've got a skill and you're using it regularly and you're thinking to yourself, "Okay, how can I save a few pennies here and, and get a different model on the go?"

Then try out UPskill, and it'll allow you to iterate on your skill and, and improve it.

Right, let's move on to Boss2. I'm gonna go through this one pretty quickly. This is about fine-tuning models. If you're really into this, there was a talk yesterday by my colleague, Merve, that went into this deeply, and there's also a blog post here where we got Claude to do this.

This was from back in November, December time now. Go and check this out. Basically, you can just say fine-tune Qwen3-6B on this dataset. This is a chain of thoughts dataset, and you'll improve the model's chain of thoughts.

This is fully integrated to the hub now, so you can even run the GPUs on, on the hub, and it uses, uh, HF CLI skills, so it's all very available.

I would try this one out. You can also try this one out. This is, uses unsloth, so it's even cheaper. This runs with, like, optimized models, and it's maintained by unsloth and by us, and it's another blog post, and there's also often free credits that you can get, uh, around these blog posts.

So I'd go and check these out. Okay, let's move on to the, the big one. [sighs]

Uh, this is AutoLab multi-agent research, which is a project that kind of, um, basically keeps me up at night.

Andrej Karpathy, a few weeks ago, maybe a month ago now, released a project called AutoResearch, which was based on his other projects, NanoGPT and, and NanoChat, and it took the NanoGPT architecture and got Claude Code to create improve-- to write improvements to that training script so that it would improve the training process.

So we can see here the experiments going over, and for each experiment, there's a change in the training script, which increases the efficiency measured in bits per bytes of that run.

And we can see that the efficiency ends at its best at the end of the process. I, like everyone, thought this was super cool, and I had to start implementing it straight away.

But one of the things that stood out to me was I found it kind of weird that we had one agent working in a single way, iterating, going and finding improvements, and then implementing them, and it would make sense to kind of distribute this.

So that's what I did. I distributed the task amongst a research team with four types. We have a researcher that basically looks up papers. For this, we use HF Papers, but we can also use Archive Papers.

HF Papers is cool because it has a CLI, so you can just pull and search papers from the hub, and it acts as a litris- literature scout. So it just looks up for papers with ideas, and it formulates those as hypotheses.

We then have a planner, which takes those hypotheses and maintains like a queue of jobs.

We then have a set of workers, and they pick up those hypotheses, and their job is to implement them as training scripts. So in many cases, just like change the architecture or change a parameter or something.

And then we have a repo- a reporter agent that goes and monitors all these jobs and maintains a dashboard that we can use.

So this is what it looks like. If you see here that we have... We- we're working in a, in a GitHub project, right? So, uh, in a Git project, sorry, and we have a main branch that we maintain with our train scripts that we're updating in each branch, and then, like, a train original that we, that we

keep. And then we have a data structure on the main branch that we use to just keep the scores. Then we implemented this in OpenCode, uh, for this example, but in the repo, which you can also go and check out, the, uh, it's also implemented in Codex and Claude, if you wanna try those.

I also implemented it in Gastown, but that's kind of Wild West stuff, so I did it in, like, a separate project. Um, but basically it works, um, really anywhere because it's more just a conceptual implementation, right?

And first, you have your planner creating hypotheses. You have your researchers looking at paper, and then your reporter picking all of this up, handing to workers, as I said.

Those workers integrate with HF Jobs, so they start these jobs off on the hub that run with the hardware that they need, and then they submit these patches that go back.

The reporter operates in Trackio, which is a da- an open source dashboard that we use for all metrics. Trackio is useful with agents because it uses a completely open data layer, basically Parquet.

So if you don't want the dashboard or your agent doesn't want the dashboard for any reason, it can just get into the Parquet and just do whatever you want.

So if you need a Gantt chart or some other visualization, it can just go and, and do that. So I would say it's, like, the best agent dashboard tool because it is basically just a data store.

You know? It's basically just a data structure. Okay, so let's just walk through this now.

So this is, uh, implemented in OpenCode. If you don't know OpenCode, you have, like, agent configuration. So in this one I just set, uh, AutoLab, which was the name of, of the agent configuration that I have.

It has skills. This is the prompt, so it says, like, "Run one autonomous local research, or auto-research pass in the repo using defined roles." I tell it to use planner, uh, to propose up two fresh single change experiments, use reviewer to reject duplicates or stale ideas.

I also tell it to use, like, a HF bucket because I want all of the storage to be in the same bucket so that I don't have to upload or download the training scripts every time.

And then we go and we select one of the sub-agents. There's a nice, uh, interface in OpenCode, but it's similar in other tools. So I select the planner, and then you'll see that the planner receives this prompt, and it uses a specific template which I defined in my configuration of, like, it's gonna have current state, it's gonna

have a, a, a list of the jobs so far, things that have worked which were defined by the reviewer, current hyperparameters that it can change, and it's basically just defining these jobs, which we'll go into the, the job list as I mentioned.

We then switch over to a reviewer agent, which will receive all of these jobs. It has a similar kind of struc- structure based on a template, uh, a reference to where it should be working from and the latest score that it should be using.

It gets an overview of all the failed and successful experiments, which it will then, like, use to base its decisions of what goes into the next queue on, and it creates this little table which we don't really need to look at.

It- it's really just for the agents to, um, interact with each other and to get this information back. To be honest, that's a little bit of a verbose example, and, uh, we maybe don't need this many tables, and you could probably trim that bit down.

But in general, I'd recommend that if you think this is cool, go and try that out in the repo. After that, so this agent runs in parallel, sometimes for hours, and this is the Trackio dashboard that we use, and these are all the runs that are pushed to Trackio.

As I said, the main advantage here is that this is fully open source and it's just a data layer, but we get all of these kinds of visualizations. Trackio can also have, like, events and warnings, so we can have all of these events being reported by different agents, and we can filter those down.

We can also even tie those up to, like, notifications, so you can get emails from Trackio if you want, if, like, your agents are kinda going rogue or something and, uh, you need help.

But best of all, Trackio just has this, uh, like, just freeform structure, so you can just throw tables in that don't necessarily fit with any other structure.

And then on the hub side, all of these jobs are just run inside Hugging Face, so you can explore those jobs. And in most cases, you can tell the agents to use, uh, like, labels, and you can sort those labels and review through what they're doing.

Or you can just look at it like this, as I mentioned. You can a- access that underlying data layer and just create a Gantt chart because this was a kinda convenient way to look at what the agents were doing over time.

So you can see, like, this Amber agent went off, and this was the score that it got. But you could visualize this however you want because you have access to this data layer.

The kinda TLDR of the whole thing is that, yeah, you can go and just have your kind of own AI lab, and you can try it out. And if you have a verifiable experiment, like training a model or doing, uh, k- or writing CUDA kernels, then it is pretty easy to, to implement and set up and, and

to learn some stuff. So let's now look at the, the takeaways, I'd say. So the... In, in simple terms, I'd say that agents work really well with primitives and, and open primitives, and we want tools that are fully open, things like Trackio, things like kernels, that we can expose to agents and they can kind of control in

their own way. Even though abstracted APIs are, are really useful, if we have a layer that we can't necessarily get behind, that, that is a ceiling. So we don't always need to extract.

It, it's more about exposing well. And the other takeaway is that the hub is, is ready, the Hugging Face hub is ready for these kind of workloads. We have the, the fundamentals in place like storage, tracking, and compute, which I think will allow us to scale our engineering to, yeah, new levels.

If you found any of this interesting, I've shared it all o- on X, I've shared it all on Hugging Face, and y- there's a blog post about basically each one of the examples that I just shared with you, and they all have repos attached to them, so you can go and try that out for yourself.

If you find anything that's broken, like, please tell me off. If you think that this was completely wrong, come and find me afterwards and, and sort of bully me.

Uh, that's fine. Uh, uh, but most of all, thank you. [audience applauding] [upbeat music]