Fast Models Need Slow Developers

[upbeat music] [clapping] Hi, everyone.

So we'll just get right into it. So over the past few years, we as developers have developed a series of bad habits when it comes to developing as a result of slow AI code generation.

And so we-we're all familiar with it. We do things like write massive prompts and try to one-shot. We'll make huge commits, or we'll have our ten agents all on the screen at the same time combobulating, cogitating, thinking.

And so about a month ago, we at Cerebras and OpenAI released a new model, state-of-the-art model, called Codex Spark. Codex Spark can generate code at twelve hundred tokens per second.

And to put that into perspective, if you look at the Sonnet family or the Opus family, those can generate code at about forty to sixty tokens per second. So in this new era, as we're starting to see much faster coding models, this is twenty times faster, not only does it unlock new capabilities and use cases, but it

also requires us to rethink how we as developers interact with the coding model. And of a lot of these bad habits that we had before, that were generating maybe fifty tokens per second of bad code, unless we fix them, they're gonna start generating twelve hundred tokens per second of bad code.

And so that is the topic of today's, uh, talk.

So to get started, my name is Sarah Chieng. I'm the head of developer experience at Cerebras, where we are building the world's largest and fastest AI processor. A large part of my job is that I get to introduce fast inference and fast coding models to developers for the very first time.

And for most people, it's a very exciting moment. There's no thinking and waiting and starting up that you might be really annoyed about. But at the same time, as I said, unless we change our habits, we are not gonna have good code in the future.

And so this talk really is a practical playbook for how we as developers can think about how we interact with the models in this new regime, es-especially in a future where the models are generating code faster than we, the human, can keep up.

So I want to look back at history a little bit. We've had a very exciting past two years. The models have gotten bigger. They're getting smarter. We have bigger context windows.

But the thing that has remained relatively constant over the past two years is coding speeds, is model speed. So if we look at a lot of the popular families, we have Gemini, Claude, GPT, Sonnet.

Over the past two years, they've always been within, you know, fifty to a hundred fifty tokens per second.

And this is Codex Spark. Again, Codex Spark is just the first of many models that we as developers can expect to be much faster than what we're previously used to.

And we even had to change the Y-axis because it's so much faster. And so before we get into the actual playbook and tips, I want to talk about why this is happening.

Why are we suddenly seeing such faster models? And it's actually a very exciting development. It's, it's what many of you probably work on on a day to day, but it's-- there's so many companies that are working on this problem all at the same time.

And as a result, the entire AI inference stack is getting optimized all at once. And so breaking it down, let's go through it really quickly. We have hardware. This is a physical device that inference, training, all of our compute is happening on.

One of the biggest things that we have to think about with hardware is the memory wall. And this is exactly why hardware and memory movement takes up fifty to eighty percent of that latency time for inference.

This is where a lot of the frustration comes from. And so when we are running inference, we have to constantly move our weights and KV cache values between memory and our actual chip.

On the NVIDIA GPU, this is the most traditional type of hardware, all of this memory is stored off-chip on off-chip HBM. And we now have a memory bandwidth bottleneck.

What a lot of newer companies are doing are thinking about companies like Cerebras or Groq. They're thinking about how do we move this memory to be as close to the chip as possible.

And so here's an example of the Cerebras wafer, where all of the chip is, um-- all the memory is distributed across the chip in SRAM, so every core has direct access to the values it needs.

Even more exciting, we have disaggregated inference, and this is an in-- Um, disaggregated inference really has become commercialized in the last few months. This is why NVIDIA bought Groq for twenty billion dollars a few months ago, and this is also why Cerebras and AWS are now partnering to serve the wafer and AWS Trainium together.

So in traditional inference, there's two steps. There's prefill, and there's decode. Traditionally, both of these steps have always been run on the same piece of hardware. Prefill is where we're taking every token that the user inputs and processing it, embedding it, and adding it to our KV cache.

This is a seque-- this is a step that can happen in parallel, and so it's compute bound. Decode, on the other hand, is where we're actually generating the output token by token, and this is sequential and is, as we mentioned, memory bound.

Again, it goes to the same problems that we mentioned before. And so what we're doing and seeing now commercially is that we're splitting up these two steps so that prefill is done on one type of hardware that is compute optimized, and decode is done on another piece of hardware that is memory optimized.

Going up the stack... There's the diagram. Going up the stack, we look at model architecture. There are so many ways that we are training our models and shaping our models to cater to our hardware.

We have specific layer dimensions and memory and model size that we're always thinking about. A great example is a very standard model architecture mixture of experts. Here, instead of activating the entire model all at once for every single token, we're only activating a subset of experts for every time.

And what this does is it allows us to have the intelligence of a much small-- larger model for the compute cost of a much smaller model. And again, we're always thinking about memory and the size of our models.

And a lot of people have been building on top of this in recent years. An example is REAP, Router-Weighted Expert Activation Pruning. I had to read that one [laughs].

Um, and here we're looking at the specific use case. We're seeing which experts aren't being activated all-- at all, and we're pruning them altogether. We're getting rid of them.

Again, we're always thinking about model size. And then at the very top layer of this stack, we have inference optimizations. And this is where many of you might be working in, and a lot of companies that you're far-- probably familiar are also working in.

These are companies like Together, Baseten, Modal, who's also here, Fireworks. And one of the biggest things that we're thinking about at this level is KV cache reuse. And so by storing and reusing previously computed token representations, we don't have to recalculate attention over the sequence at every step.

And now I wanna get to the very top and most exciting part, the developer. This is the current state of what the internet looks like or what Twitter, LinkedIn looks like.

We have someone running six Claude Code terminals at once, f-a five hundred plus agent coding swarm, um, someone running eight agents across five screens. And I get how tempting doing something like this can be.

I feel like if you're on Twitter at all these days, unless you are doing something like that, the internet is basically convincing you that you are living in the Stone Age and that you need to catch up.

But the reality-- what is the reality of what is happening in all these setups? Is that we're generating massive amounts of code that nobody is verifying. And in the new future with much faster inference, this becomes increasingly dangerous.

And so especially with fast inference, we're now gonna be generating technical debt at a level that we've never seen before, and we're not gonna know what to do with it.

And so I'm gonna pivot now to spend the rest of the talk on the practical playbook and tips and workflows and how we can reimagine how we as a developer should operate in this new regime of faster inference.

And as I mentioned, Codex Spark operates at twelve hundred tokens per second. But it really is just the first model in what we should as developers expect and prepare for to be a new regime of faster models across the board.

And so starting with the first one, the first category is just choosing the right models and how do we orchestrate our agents so that we're leveraging different model strengths.

I think historically, we always think about intelligence. There's n-there's no secret that we as developers are not particularly loyal and that we will switch to whatever model, whatever family is the most intelligent at a given time.

And maybe we also think about cost unless our company pays for whatever we want. And so here, now the inference speed is a twenty X difference. Now we also have another vertical to dif-- think about, speed.

And so a good mental model is to use a larger model like GPT-5.4, 5.3 for your planning or your long horizon workflows, and then using a s-faster model like Codex Spark as your actual executor.

And so here's an example. You might ask your five point-- GPT-5.4 to generate your plan. You would generate a s-- um, you would spawn all of your sub-agents with Codex Spark and have it actually oper-- uh, have it actually execute on all of the steps o-- um, one by one.

Another really helpful trick is to actually make skills out of successful sessions and capture trajectories that are working really well. A thing that you can do here is use a model like GPT-5.4 to actually have it do the initial harder, larger task, capture that as a skill, and therefore making it a verifiable, repeatable workflow, and then having

a small, um, s-faster agent like Codex Spark just do it again and again in the background.

The next category I think is even more ex-exciting because this is a ca-category of things that just were not possible and were not practical. These are things we wouldn't do because we're tired of the cogitating, gesticulating,

germinating that you might have seen. And so here I really want us to think about this and internalize this. But at twelve hundred tokens per second, a model like Codex Spark makes validation basically free.

There is no excuse and no reason why you should not be doing things like this, test suites, linting, pre-commit hooks, diff reviews, browser-based QA automations. There's all these things that you can add to every step of your workflow because it is instant.

It's not slowing you down, and it's not you do this all of this at the very end or right before you're about to push your code.

Another tip that I really like is exploring cherry picking. So let's say that I want to code a nav bar and I want it to be midnight blue, I want four different icons, I give it to the model and, and the result's fine.

Instead, what I can do with Codex Spark or s-- much faster model is I can have it tell it to generate fifteen versions in the same time that it would have taken me a previous model to generate one version, and I can cherry pick the version that I like the best.

Even better, I can generate five sub-agents that are each generating fifteen versions, and now I have seventy-five versions, and I pick the one that's best. And this is great for things where we really value quantity or variety, so things like research direction, different types of architecture diref-- um, directions, or even just graphic design.

And the reason why I really like this one is because it almost allows us to artificially induce taste into our model output. So traditionally, it's no secret, it's very easy to sniff out any UI or text that a model writes.

The models themselves do not have taste. And the ways that we've kind of brute force worked around this is that we either create an example ourselves or we find examples for the model, which is time-consuming, or we give the prompt so much detail that we might as well have completed the task ourselves.

This is a great way of saving our time and also getting much better results.

The next tip is kind of more, more so a, a mental model where now that the models are so fast, it should not be you spawn a session, you go get a hamburger, you scroll Twitter,

and then you come back. Now, you can actually sit down, and it's a real-time collaboration that you're able to have with this model. You should view it much more as a pure programmer, and this is the only way that you are going to avoid having bad code.

So you can sit down and ask questions like ha- having it collect all the context across your repo and actually asking it how does it work, being the one in the front seat making decisions and implementations.

The AI should always be helping you make decisions, not the other way around.

The next one, I hate this slide because it's everyone's trigger word and overused word, but [laughs] how do we avoid slop? So as I was mentioning before, it really shouldn't be, you know, you spawn ten agents, you never verify the code, you don't know what's happening under the scene.

Someone asks you to explain, you have to read the code for the first time. Now you can actually have two to three sessions and actually sit down next to your code, and I know this is something we're not really used to, but sit down with it and actually steer it.

Understand what's happening because, again, we now are now experiencing real-time collaboration as we code with this agent. You can be super specific. You can thing-- do things like ban the model from deleting files, give it a max diff size, the model-- have the model only read and write, and even give it steering directions, things like, "Only change

this. Don't touch types yet. Wait, that implementation wasn't quite right. Let's redo that." The graph on the left is a, is a helpful mental model as an example of how the developer, the AI agent, and the code base can all work together and what that should look like.

This next step, refactoring, is very similar to what I was talking about with valida- with verification. Just like with verification, something like constantly refactoring and cleaning up your code automatically is basically free at twelve hundred tokens per second.

So you can do things, instead of doing it at the very end right before you're about to commit your code, you can just re-- you can just bake this into your automatic workflow so that after every single task on that checklist is complete, you're just asking the model to automatically, you know, delete unused imports, clean up unnecessary

lines of code, make it so that all of my functions are structured the same way.

The last category that I want to talk about, and I'm sure that so many of you guys have already heard these two words a countless [laughs] amount of time over the past few days and across so many talks, is context management.

But the reason I'm going to talk to you about it again is because let's say that historically it took you ten minutes to fill up your context before you saw, you know, the God-feared word, compaction.

Now, if you take ten minutes, divide it by twenty, you are now getting compaction in thirty seconds. And so context management, especially with fast inference, is more important to think about than ever, and you can't get away with sloppy practices anymore.

And so all of these, these really are just good practices no matter what coding model you are using or what speeds. But a general, very high-level framework is just always, always break up large tasks into smaller bounded goals.

And this graph on the right is a good mental model for how, how full your context is will then affect your behavior, the model behavior. So you always want to avoid the eighty to a hundred percent because you're gonna get compaction.

Right now, we all know some things might get lost.

And so a good way that you can think about, "How do I externalize this memory so that I can have these small bounded goals?" Like, what does that look like?

So an example of how you can do this and set up an external memory system that is persistent every time you set up a new session is with this four file system.

We have agents.md, which is where we're actually defining all our agents, our sub-agents. We have plan.md, which is what we're creating at the very beginning, and this is where we're just generating the entire plan and step-by-step chep- step-by-step checklist that we're going to go through.

We have progress.md, which is where we're keeping track of what do we need to do and what has been done before. So every time you spawn a new agent or session, there's no context.

It comes in, it looks at progress.md, it sees what's been done before, and it's like, "Okay, here's where I pick up. Here's where the next task needs to be done."

And then the last one's verify.md, and this is what we're using at every single step to just make sure everything looks good, it's clean code, and we can move on to the next step.

And so an example of this is, again, leveraging different models, using a GPT 5.3 or 5.4 Codex, having it create your plan, and then having your GPT 5.3-Codex-Spark actually execute the checklist one by one much faster than before.

And as a final slide, I want to do these, um, a few helpful commands for how you can get the best out of Codex, things like permissions, experimental skills, review, and rename.

But the biggest thing that I really want to emphasize here is that honestly, it's not really about just having faster coding models. What it really means is that the ex- developer experience is actually gonna become so much better.

And when it's becoming so much better, there's so much more we can do, and there's so many ways that we can now avoid creating bad clo- bad code in a way that isn't miserable or us staring at a screen for thirty minutes.

So thank you guys so much for welcoming today-- me today. My name is Sarah Chieng. Um, I'm visiting from SF. It's an honor to be here in London. Um, if you have any questions or need any credits, my handle is [REDACTED:username] across every platform.

Thank you, guys. [audience applauding] [upbeat music]