What Is an Inference Engine, Anyway? — Charles Frye, Modal

All right. Um, so I'm gonna go ahead and get started while people get seated. Um, we got a pretty tight turnaround, a lot of really wonderful workshops at this event, um, so I wanna make sure we have plenty of time. Uh, so I prepared for ten or fifteen minutes of questions, and I think it'd be best, especially given this is a workshop, if people just interrupted me as I was going. Raise your hand and I'll call on you and answer your question. Um,

and so just, like, let's keep this interactive. Um, I think it's, it's better that way. Um, and it seems like-- talked to a couple of people, it seems like we got a lot of folks here who know their, who know their inference, uh, know their engines, and so we can get a, um, sophisticated conversation going. Um, so the title of this talk is What Is an Inference Engine Anyway? And the goal is to sort of peel back some of the layers that are behind

APIs and, uh, what is go-actually going on inside of the code that runs your inference, uh, when you run it yourself. So roughly, you know, their tokens go in, tokens come out, um, and a lot of the time you get to avoid thinking about what happens in between. Um, there's this really high quality open source software out there for running your, uh, running your language or multimodal model inference. You've got SGLang, vLLM, TensorRT-LLM. Many

people write similar things to these in-house. Uh, they design these, these, uh, engines that, like, at a high level, just taking in some tokens, putting out some tokens is not that hard. Um, you can write something like this in using PyTorch and Hugging Face Transformers in, like, an hour or two, especially if you're good with agents. Um, but making it, uh, extremely high performance so that you can produce tokens with good tokenomics is the challenging part.

Um, and that dr- like, sort of drives the design and architecture of these systems. Um, so just as, like, a very quick sketch, um, before we jump back to the high level and talk about why, why we're here. Um, the, uh, these are generally architected as a set of communicating processes, one that does IO, does HTTP or RPC serving, and responds to the outside world. Um, and then,

uh, there's a little bit of pre and post-processing with tokenizers and detokenizers, uh, and this wraps, uh, the sort of core of the system where the real work gets done, um, in sort of two pieces. One, uh, a sort of scheduler that s- like, defines and schedules work to run on the accelerator, usually a GPU, um, and then the actual code that runs on the GPU. Um, and so this, these two components in the bottom right

here are where the sort of majority of the performance sensitive work gets done and where the complicated problems are. Um, so that's where we'll spend the majority of our time, uh, discussing.

Um, all right. So with that stage set, um, I think, uh, I've been to AI engineer since it was the summit and talked about, uh, running, uh, running inference, uh, every time, and the answer to this question gets better and better each time. Uh, the question of like, why inference and why you should care about it, why you should think about being, uh, an inference engineer who operates inference engines. Um, our, uh, fearless

leader and, uh, uh, one-man conference army, Sean Wang, uh, tweeted recently, uh, that everybody in AI infrastructure, the boring stuff, uh, like inference engines, not the sexy research stuff, um, is finally getting filthy rich. Um, and so what, uh, I think he's pointing to here is there's tremendous demand, uh, for these, uh, to run inference. Uh, there's demand at the labs, there's demand within companies who want to own their own stacks,

there's demand to set up and rethink the infrastructure s-stack, and all of this wraps this core inference workload. Um, but Sean is also pointing out that a lot of what people have been thinking about up to this point, um, is sort of the research side, the production of models. So, um, why not, you know, why not think about training? Why not go to the training engines, uh, talk with your precious time? I think the fundamental reason why this is such a critical place, uh, for

engineering work to get done and why it's such a, like, cool place to work as an engineer is that in the end, training is a cost center and inference is a revenue center. Um, so when you train models, um, it turns out, um, people have not generally been able to make businesses work where they sell model weights to people directly. Um, and so you can't really turn a training setup into a, like, a proper self-supporting system. Uh, but, uh, inference, once you have a model, um, is

something that people will pay for. People will pay for a system to, you know, take in their tokens and their context to produce new tokens, produce software for them, whatever. Um, there's a lot of opportunities to do inference. We have a setup where there are a small number of essentially model foundries who produce foundation models that are then deployed and customized by inference engineers. Um, and then even for training, inference has, uh, snuck its way in. A lot of the post-training

adaptation of these foundation models pr-requires that you generate samples from the model, uh, in inference, and a lot of the same technologies come to play, a lot of the same concerns are present. Um, and then finally, sort of more selfishly as an engineer, there's-- But right now, inference is still very new, even though we're years into this new technology. Um, and so you end up kind of straddling the whole stack from application architecture, uh, through linear algebra down to, like, electrons and hardware. So it's a,

an engineer's playground. Um, so we'll see a lot of that as we go. Um, I'll maybe in the interest of time, since we kinda had to get started very quickly, I'll skip over this. But I've been working on inference for a couple of years, mostly at, uh, at Modal, um, since coming from the sort of training world before that, um, and helping a lot of teams sort of deploy and optimize their inference on hundreds or thousands of GPUs on the Modal platform. Um, I've also been making a lot of, uh, inference applications myself. This is a little art piece that I made. Um,

we, uh, uh, had it in the, uh, Legion of Honor just this past week. Um, this art piece takes what it sees, passes it through a vision language model, uh, one of the Qwen MoEs, running on, uh, uh, served via the inf-- SGLang inference engine and running on Modal. Um, it takes that input, and it describes it as though it were an art piece. So it produces one of these little museum placard, uh, type things. And as everybody knows, you're not art until somebody writes one of these little placards to describe what you've done. Um,

so I pointed it, I pointed it at the line on the way in, um, and it came up with a cute little description. Anyway, so this is the kind of, you know, uh, sort of in the middle of the set of inference applications between the lowest latency and the, uh, highest throughput applications that get deployed. Um, so, uh, I'll start by talking in more detail about those workloads, like what workloads land on these inference engines, and then jump quickly, jump-- move from that quickly into architecture, how these engines are designed, some of the key

techniques, uh, that fit within that architecture to make their, uh, improve their performance. Um, and then lastly, a key piece for making these things work in production, observability and, uh, like debugability of these engines for correctness and for performance. All right, so let's start with workloads really quick. Um, uh, I like to break things down into kind of three pieces. There's chatbot plus, which is like the original killer app of inference, something like ChatGPT or Claude Code.

There's a person waiting on the other side interacting with this inference system and providing additional context. Um, the plus, uh, indicates that not only does it talk to a person, it also probably interacts with external systems via tool calls. Um, then you have background agents. They often do similar tasks to something like Claude Code, um, but they do it in the background, not directly in response to a person. So something like Devin, Ramp Inspect, uh, OpenClaw or Herbies for a non-coding case of a background agent. Um,

so, uh, this is, uh, characterized by a similar sort of, uh, affordances or workloads, but a different performance profile. And then lastly, data processors. Um, these generally run relatively small models. They take unstructured data, like a PDF, and they turn it into structured data. Um, they extract some information from it, um, usually the sort of thing you might insert in a database. So this is something like the, uh, Reducto document processing platform or, uh, something like, uh, Fathom that takes a, uh, a

video and extracts the transcript. Um, okay. So under the h-hood, what do these, uh, workloads look like? Um, you send in some tokens from a client, uh, that goes to some server somewhere. Um, for now we're gonna focus on cloud deployment of inference 'cause that's where the majority of it happens right now. Obviously, in the future, a lot of these inference engines are gonna end up running on people's, uh, machines like in a couple of years. Um, yeah, go ahead.

Is there anyone that can adjust this side of the screen? Or if it's like that, always-

The size of the screen?

No. The brightness.

Brightness.

Brightness of the screen. Um, I'm-- Yeah, so the question was about adjusting the brightness. I'm not sure if we can, uh, fix that. That's what I get for making dark mode slides. But yeah. Um, yeah. All right. So a client submits, um, an input, uh, and then it's processed by our engine running on a server somewhere. Um, first, we process the input tokens in the prefill phase, and then we generate the output tokens in the decode phase. Um, and so we have, uh, one, uh,

one prefill, uh, run, uh, does a forward pass through the model, and then multiple decode runs produce one or a few tokens at a time. And then once we're done, return them to the user or generate a tool call and, and interact with an external system.

Um, so with these work-- these-- both the external interface and this sort of internal prefill decode split defines these, uh, the sort of metrics that you care about for a workload that sort of tells you how to deploy your inference engine and, and how to measure its success and performance. So you have something like queries per second in and queries per second produced. Um, this one pretty tricky to think about, uh, under control of your users. Um, uh, you probably have a better time

thinking of it, uh, at the engine level, thinking of how many queries per second you can handle per replica of your inference engine, um, and then you have aggregate demand coming from users. Uh, then with those queries, you wanna look at how many tokens there are per query, um, input and output, AKA prefill and decode. Um, so this one is tricky because it, because it's not only under control of users, but it's also under the control of the model. Uh, so the determination of when to finish processing a query is not under your

deterministic control. It's under the control of the model inside the inference engine. Um, so this is something not-- Both of those sources of nondeterminism means you need to kind of like measure this, benchmark it to get it. Um, and then, uh, when we do our processing, we, uh, we produce some intermediate artifacts that we wanna cache in the KV cache. Uh, and then the degree to which that cache is reusable is a key component of the definition of our workload. Um, this is also mostly under control of users, but you

can get, like, depending on the workload, there's actually, like, pretty, uh, relatively fixed patterns of, of prefix reuse. Um, uh, and then finally coming from the application layer, um, you have some, uh, latency budget, some SLO of, like, how quickly you want to return tokens. That could be how quickly you finish something like the prefill phase and have your first tokens ready to send to users, or it can be how quickly you can generate each set of tokens during your decodes and get them back to users.

So that's your time to first token and your time per output token or intertoken latency. Uh, so this is gonna be the, like, primary constraint that you operate under. Okay. Um, and we can map those back onto our application arch-archetypes, uh, like a chatbot plus background agent, really high prefix reuse. There's some interactivity going on, whether that's a person or an agent. Um, the chatbot plus generally has relatively short decodes, relatively short outputs of tens to hundreds of tokens, um,

and a very tight latency tolerance on that. Like, a person, uh, wants to see something in a few hundred milliseconds at the very most, or they get bored, uh, they get angry, they turn away. Um, with a background agent, you maybe have a few minutes to produce a PR, right? Um, if it's as high quality as an engineer, maybe, uh, as, like, what would be produced by a good software engineer, maybe you have a few hours. Um, and so there's not as tight of a, of a budget on your milliseconds of network overhead or whatever. Um, and then with a data processor, you generally have very low prefix reuse. Um,

you are seeing different documents each time. There's maybe a system prompt, but it's generally small relative to the other inputs. Um, and then you have a very short decode. You're producing some structured object that's generally short relative to the input. Um, the structured version is shorter than the unstructured version. Um, and then the demand here is less on latency generally and more on aggregate throughput because you're generally, like, ripping through a database backfill type job in a lot of these. All right.

Um, and so may- maybe we just wanna call out, uh, like now that we've looked through this, you could see that, like, e-even though at the application layer this feels like kind of one s-- it's one single workload from the pref- perspective of the user, once you start drilling down into what the engine is doing, it really feels like there's two sub-workloads going on. There's the, like, workload to do your prefills, to do the processing of input prompts, um, and then there's work to do the decodes to produce the outputs. Um, and

these, uh, they sort of operate semi-independently inside the engine, um, and so need like a sort of like different perspective, and they follow often, like, different code paths. Um, and they're sort of defined by the fact that the per user in the decode phase, you do-- you're doing far less arithmetic, far less math per, uh, per decode per user than you would do in the prefill phase, which is a lot, like, much higher arithmetic intensity, a lot more,

uh, uh, a lot more math per request. Um, and so that's your fundamental constraint, but also the sort of space in which you operate in configuring an engine and then, like, adding features or, um, adjusting the engine behavior to serve these two things.

Okay. So I wanted to make sure we're all on the same page about some of the core stuff about what these engines are actually doing before we peel open, uh, the insides and start looking at them. So what is the architecture of an inference engine? Um, so, um, this is a little, little diagram showing at a high level what an inference engine does. So the goal of an inference server is to take an HTTP request or a gRPC request and turn it into an HTTP response to the client. Um, and so it does that

by passing through all these things down here. Um, so the idea in this diagram, um, is that if you follow any set of arrows, you get the same thing. So, uh, you can sort of peel back a layer of the arrows and look inside and say, "All right. So what, what is the-- the inference server does this high level thing. What's the next level down?" The inference engine is the sort of like interesting part of it, is the, is the part that is not doing the HTTP serving because we aren't in one of those situations where HTTP serving is the hard part, is your

bottleneck. It's not like a proxy. Um, it's, uh, it's a-- this is a case where you're handling tens, hundreds, maybe thousands at most of requests and responses per replica, and so general purpose hardware just, you know, kinda eats this for lunch. Um, so the real interesting stuff is all at the layer below this. Um, so at the top is your, like, your server IO for HTTP requests. At the bottom is the actual engine, and its, like, collection of processes. Um, and you can kinda break this out into three

pieces. The request comes in expressed using, like, sort of human-friendly, um, or external representations. So you have Unicode strings, you have PNG images, you have MP4 videos or WebRTC streams, um, and these need to be preprocessed before they can get passed to the actual inference. So they get turned into collections of tokens, um, which then get mapped into tensors on the GPU. And then we can actually do the inference part of the inference engine, which

takes, uh, you know, in the simplest case, it takes a tensor with K entries in it and gives you a tensor with K plus one entries in it. And these are machine learning models that predict the next token. They guess what would the next entry in this big tensor look like, um, based off of the data I've seen in training, whether that's data from the internet or data from what a helpful agent would do or, uh, RL data, um, whatever that you've learned to predict this next tensor entry. So that's the core right there,

um, and that is what is wrapped by the rest of the inference engine. Um, so then we have post-processing on the side here. So post-processing turns that new tensor into a response, uh, and that's generally done by this detokenizer that goes back to the sort of- Token space first and then, uh, and then back to our, um, back to the human and external system friendly representation in the response, so it can be returned to the user.

Question?

Could you, could you explain the difference between the inference server and the inference engine, and where does vLLM lie?

Great question. Yeah. So the question was what is the difference between an inference server and an inference engine, and where does something like vLLM lie? Um, so inference server, uh, means that you've wrapped some like input/output around the core operations of like taking requests and turning them into responses. Uh, so vLLM has both of these. Like vLLM operates as a, an HTTP or gRPC server. Um, and then you-- the

engine layer is sort of like part of the vLLM or SGLang SDK as like vLLM dot engine. Um, so most deployments are-- look like are deployed as a server with a like a single request, single response kind of interface, um, as depicted here. Um, yeah. Does that answer your question? Yeah. Um,

cool. Um, I said this earlier, but some people have come in since. Like please interrupt or raise your hand with questions if I'm like ... if there's anything confusing or if you wanna dive deeper. Yeah, [REDACTED]?

I'm assuming this is a stateless type of workload.

Yeah. So the question from [REDACTED][REDACTED] is, is this like a stateful or stateless, uh, workload? So the answer is, it is stateless if you don't care about performance. Um, it's stateful once you do care about performance, which is pretty much every case. So in the primary state that you accumulate over time for requests is the KV cache of like previously computed things for, for a previous request attached to some notion of session. Um,

and that handling things like sessions actually lives outside of the inference server/inference engine in general. So the-- an in-individual engine replica can track this KV cache, but is up to the wrapping layer, which might be something like NVIDIA Dynamo or LMD to handle like defining what a session means. So it's like the clients in that architecture sort of define the, the sessions. Um, and that is where a lot of the trickiness comes in. Um, so we won't talk too much about that 'cause it's kinda outside the-- There's enough to talk about with

engines, um, already. But yeah, uh, come find me afterwards, um, if you wanna talk about that layer.

Um, right. So this diagram, we also talked about it previously. We just wanna return to it now that we've seen a little bit more. So this outer layer is our server IO process. This, um, is it-- this is specifically SGLang's architecture. There's some slight differences with vLLM, but this server IO process communicates with the outside world and then with our pre-processing and post-processing. Those things communicate with a scheduler process, um, which is the one that defines and schedules work onto the GPU, handles things like

queuing. Um, and then the... Actually, not ready for that yet. Um, and then the one or more processes that are sort of closely tied to the GPU are the ones that do the real work of doing the inference. Um, so the, yeah. So the key things to pull out here is that if you are an old school ML person who's been in this world for a long time, the model forward passes layer is the one that you're familiar with. This is where the vast majority of the sort of like PyTorch, uh, and

similar code shows up. Um, and this is the pla-- like the GPU i- or the accelerator in general is the component that costs the most, um, and is capable of the most work, like petaflop per second scale operations. So this is where the like hardcore shave a, shave a microsecond performance engineering is focused. Uh, but critically, this scheduler process out here, or its equivalent in another architecture, is your-- is

the bottleneck to that component. Um, and so this is, it's generally operates, um, it is literally single-threaded in the implementations we'll talk about, but it's also like semantically it's single-threaded. It's this one thing that is defining-- somebody has to define the work that happens on the, on the GPU. Somebody has to manage a bunch of resources there, and there's some, some fiddly, like tricky things to do concurrently. And so this becomes this, uh, like s-s-semantically

single-threaded, uh, like point of control that can potentially, despite the fact that it's not doing much, it's not doing as many operations, uh, as the work on the GPU, it actually becomes your bottleneck. It's the sort of gatekeeper to the GPU. And so this is the other place where the majority of like performance engineering and care needs to go in the design and operation of an inference engine. Um, so we'll see more on that as we go, but that, that's like the high level takeaway about these things.

So then is the solution to that bottleneck having better, you know, detokenization processes to run schedulers in parallel by breaking up the problem?

Uh, yeah. So the question was, is the solution to that bottleneck to, you know, operate the schedulers in parallel or, um, uh, or to improve some of the external components that consume from it? I think the current state of affairs is that this is not reached the-- like that you can operate these server processes in Python, and you just have to be faster on the host side than the GPU side. And

there's not-- the like complexity increase that you get from trying to manage like locks or coordination on the GPU resources is much-- that complexity increase is much greater than the gain. Um, because even if you were able to get it down to a microsecond, the work wouldn't start happening until the previous work on the GPU had finished. And that's- You know, depends on the particular workload, but you're talking generally tens of milliseconds to hundreds of

milliseconds. And so the, like, um, yeah, there are general-- most of the time you can get around it. We'll, we'll talk a little bit more about this, so maybe ask again once we get to that if, if that wasn't satisfying. Um, but yeah, great question. More good questions, please. Okay. Um, so then just looking at that s-- that was, like, the architecture diagram that you think of as you're, like, engineering. Um, let's look at the, like, life of a request in this architecture. So the, um, the server,

like, receives an input from the outside world. It says, "Okay, we've got a request." Um, it sends it to the tokenizer to s- prepare, pre-process this request. That g- then goes into the scheduler that decides, like, okay, how many resources is this thing gonna consume? What resources are available? Um, and then that runs this core loop of creating, um, batches of requests to go in the model runners. So generally, most of the accelerators, like, benefit from operating on multiple requests

in parallel. Um, so especially GPUs, but also TPUs and, and many others. So you want to collect a bunch of requests together and run them. Um, if you're a database person, you might imagine, like, I'm go- I'm about to run a sequential scan, so I might as well grab like three or four queries that are all going to, like, read this entire table and do all of their, um, like, you know, do all of their queries at once. Um, 'cause the hard part is that, like, sequential scan of loading the entire table. Um, so there's a similar thing going on here.

Um, uh, so this core loop operates over and over again, start a batch, and outputs the, like, uh, the tokens and the probability distribution over tokens, the, the logits, um, that comes from the, the model runner or runners. Um, so that operates, um, sort of continually. Uh, and then as tokens are ready, uh, they can be passed to the detokenizer to be, like, turned back into a proper response to the outside world and then sent back to the server.

So yeah. Rel-- this is-- What's interesting about this stuff is that it is simple relative to what's going on inside databases, file systems, uh, like web servers, um, uh, partly because it's fairly new, but partly 'cause there's just this core bit at the middle that you just want to operate at maximum speed. Yeah. Question.

Is there communication between the tokenizer and the detokenizer?

The question was, is there communication between the tokenizer and the detokenizer? Um, to my knowledge, there is no direct communication between the tokenizer and the detokenizer in SGLang. Um, in vLLM, they actually live inside the same process, the, um, manager, the engine manager. And so, like, it'll be a little harder to determine if there's any communication there. Um, but in general, they actually don't need to communicate with each other. They just need to operate essentially the same, um, system for mapping from, like, inputs to, uh,

tokens or tokens to outputs. Yeah. Did that answer your question?

Yeah. I guess I got confused because I think, like, the, the mapping-- I'm not sure the map-- how simple the mapping is. Like, if you start to have ambiguity at some level, disambiguate it in some way-

Mm-hmm

... and it seems like you would need to respect the disambiguation and the de-tokenization.

Right. So the, the question was about basically maintaining coherence between the tokenizer and the detokenizer and how would you handle maybe, like, ambiguous, um, like, ambiguous inputs. Um, the answer is that, yeah, they, they run the same, like, underlying lib-- like, underlying libraries, like the-- usually, like, the tokenizers library from Hugging Face. Um, and they-- I would say the tokenization part is the one that is harder than the de-tokenization part. The

de-tokenization part is, like, a little bit more entirely under c- under the control of the person implementing the, the detokenizer. Um, yeah. Um, yeah.

So, say the software allows them to always make the same decisions?

Yes. So the question was, does the software allow the tokenizer and detokenizer to always make the same decisions? I would say yes. I'm try-- I'm, like, trying to make sure that I'm not overpromising here. Um, but I don't think you need communication between them to ensure that the same decisions are made. Yeah. Question. Sorry, the person... Yeah.

The tokenizers can be parallelized correctly in administrator.

Yes.

But in practice, you still see longer time per token on longer inputs. Um, why is that?

Right. So, well, first pointing out that the tokenizer can be, um, parallelized and generally is, like, multiple processes. Um, and yet you still see longer time to first token, uh, if you have longer input sequences. So why is that? Um, so the answer is that processing a longer sequence of tokens takes longer inside of this core loop. Um, so it's the, like, primary driver. Um, if you have a hundred thousand tokens, that will take, let's call it

ten times ten thousand tokens. Um, once you get to a certain, uh, size, it's basically linear in the input sequence size. Um, then in a, like, running inference engine, there's also effects from, like, you break up the input tokens into multiple pieces and run them as separate batches, so you can also get additional-- if there's, like, queuing and congestion, that adds more to the, um, uh, processing of a, uh, of a long batch, um, than to a short batch. You experience the queuing delay multiple times,

possibly. Um, but maybe the, uh, one underlying, um, point to clarify, like, producing the first token means producing the first output token, not the tokenization process. Um, and then finally, not a part of the question, but something I wanted to point out, it is interesting that you generally have multiple deta- tokenizer processes, um, and one detokenizer in the SGLang architecture at least. Um, and the answer there is that input tokens are-- come in at, like, a much higher rate than output tokens are

produced. Um, and that's related to the fact that prefill can operate at much, like, higher rates of total tokens per second than decode can in general. Um, and so you-- there isn't as much pressure on the detokenizer component of the system to operate, like, at the-- a-as quickly. Um, and it's also in some-- it's bottlenecking your response to the user, but it's not bottlenecking the core model forward pass resource. So it's also, like, off the hot path a little bit. Um, yeah.

How much is kind of the portion of the latency between the tokenizer and then the core loop, like this part of the code?

Yeah. So the question was how much of the latency comes from this part in the tokenizer versus this part in the core loop. This is gonna b-depend on your workload. Um, I would say there was a nice thing from, uh, who was that? Crusoe, I think, um, that pointed out that the existing tokenizers generally don't show up as a bottleneck if you have models that are, let's call it, like, twenty billion parameters total or, or above. Um,

and that, and if you-- but if you are operating a smaller model and you're operating on very large input context, then it can show up as a, as a bottleneck. So that's maybe a good rough guide, I would say. Like, I actually have not had to think about the tokenizer latency very much. It's generally been prefills and substantially decodes that are your-- the, like, latency bottleneck. And so decodes, if you're producing one thousand output tokens a second, each one is, like, one millisecond on average. Um,

and the, um... But up to, like, ten milliseconds, twenty milliseconds, um, or even more if you have a, um, lower decode throughput, like a large MOE that you're running at lower interactivity. And so that very-- the tokenizers are definitely millisecond scale or less and only happens once, um, per request. Yeah. Also notable that nobody talks about tokenizer caching. People talk about KV caching, not tokenizer caching, even though you could do it.

Um, I'm gonna keep going just to make sure we get through more stuff, but these have been great questions, and so if it's still relevant, please ask. Um, another important question to ask. Wait, why are we doing multiprocessing? The key reason is we want, we want separate threads of control for each subcomponent for our server I/O, for our tokenization and detokenization, for each model runner. Um, we want them to be able to sort of operate as independently of each other as possible. Um, they are-- many of them, the tokenizer and the detokenizer are doing relatively compute-intensive work relative

to most of what happens maybe in a, in a, in a web server. Um, the model runner is, um, like interfacing with the GPU or accelerator. It interacts with that in a way that's kinda similar to the way you would interact with a NIC, um, in like a web server, which is to say you offload work onto it. Um, and, uh, so that one, you, you're giving it its own event loop. Maybe in theory, that could be part of, like, a broader event loop. Um, uh, but it's, uh,

certainly, like, cleaner to have these separate threads of control and optimize their, uh, you know, optimize their latencies, and there's not sufficient, uh, interprocess communication to require them to, like, say, live in the same thread. Um, but why not threads, uh, for each of these instead of processes? Um, and one of the fundamental reasons is that Python is single-threaded per process. This has recently started to change. There's still a lot of work for compatibility and performance to be done there. Um, so that, uh, because Python has a global

interpreter lock for all the threads in the same process, you would, like, frequently block on that. Um, so it's-- so that is another reason to sort of split these up. Um, but that just, uh, begs the question, um, uh, or I guess begets the question, why Python? Um, and the answer is Python has excellent model or GPU support. Uh, so it's, like, easy to run code on accelerators with Python. So specifically for that model, um, uh, model forward pass component. Um, and so wait, but why--

wait, why does Python have excellent support here? Partly it's that it's a great language for researchers 'cause it's, like, ease of use and these, uh, models are still coming out of a sort of research-y, data science-y environment. But there's a deeper reason, which is that Python has really-- has kept, like, really deep C and C++ interop as one of its core design principles, and that allows for, like, much easier communication, uh, with these accelerators and, uh, with, like, accelerated code in general, um, than, uh, like, a maybe equally ergonomic

language like JavaScript. Um, but why not, like, rewrite it in Rust? Um, the model worker processes, again, they're just trying to, like, tell the GPU what to do. So you could-- like, the code on the GPU is written in a compiled language and, and, like, carefully optimized, like, to within an inch of its life. Um, but the model worker processes are just organizing that. And there is a few tricks, like CUDA graphs, um, that, like, really just pull all the work out of the worker processes and into sort of fast stuff running on the

GPU. Um, but there is the scheduler, on the other hand. So the scheduler just needs to stay faster than that work. Um, but as GPUs get faster and as the amount of work on the GPU goes down, the pressure on the scheduler component goes up. Um, so this is actually, like, a pretty good target for a rewrite it in Rust type moment, um, because this part is host side. It has to manipulate a small number of GPU resources, but it doesn't necessarily have to know stuff about models and PyTorch. Um, and so it's like a thinner rewrite. It's, like, specific

per engine, um, and, uh, uh, and host side only. So, like, in the sort of arc of future inference engines, I'd expect it to be a big target, um, making that faster now that the f-the sort of iterative development speed is slowing down a bit.

Okay, so now let's look inside of the, like, what, what is going on inside of these components. Um, um, get mostly, mostly focused on the, um, the model forward pass piece. So at the very-- We're going sort of like inside out. So at the very bottom, there are kernels that run operations fr- as part of the model architecture in general. Um, these are mostly deferred by the engine, which is to say, like, the engine doesn't implement all of the kernels for every operation it needs to

support. Um, it has backends that implement the, uh, these kernels, and it consumes them from external libraries of kernels. This is one reason why there isn't much differentiation between the engines on sort of raw GPU performance. Um, it's mostly about how good are you at getting out of the way of the GPU. Um, there are a few special kernels per engine like tree attention for tree-based speculators in SGLang. Um, and so there's like little bits where an engine will have a specific kernel, but, like, if it's useful, if it's popular, it

ends up in some kernel library, so it can be used in lots of places, and then they consume it from, like, an external library. Um, then, uh, the kernels are selected by the sort of model forward pass code that does control flow of-- and sort of like selection of which kernels to operate. Um, and this is, this is implemented in each engine in general, and it's implemented per model. Like, if you want to add support for a new model architecture, this needs to be implemented in each engine. Um, there's generally a reference

implementation that comes from a library like Transformers, um, but that is insufficiently optimized, um, the, like, s- the setup there. So there's, um... I guess in SGLang and vLLM, there is now a path where you can just use that Transformers implementation, and they'll maybe do a little bit, tiny bit of surgery to see what they can automatically kernel fuse or automatically optimize or rewrite. Um, but in general, uh, for most things people wanna run, you're gonna go through this hand, handwritten, as handwritten as software is these days,

uh, path for, um, like selecting which kernels to run. Um, then, uh, wrapping those model forward passes is our batch construction to select the inputs to that model forward pass. Um, this is kind of one of the primary sources of special sauce per engine. You need to set up this, um, this sort of like queuing setup. Um, so you might want to be able to, for example, not operate separate prefill and decode queues but operate one sort of

queue where you can mix prefill and decodes together. Um, that might have some performance implications. You might also want to be able to, like, dynamically switch whether you're running pure prefills, pure decodes, or mixes. Um, and then you want to do this. I think in the ideal operation of an inference server, there is no queuing. Um, so as somebody who operates these things, I, like, I treat queuing as a problem, um, but the engine should be able to kind of do it. Um, or may-maybe it's too extreme to say that queuing is a problem, but, um, you don't want

this to be a, like, yeah, a growing queue. Um, you want it to be flushed as quickly as possible. Um, and then lastly, wrapping that, you have, uh, detokenization and tokenization. Um, this is, like, mostly boring, I'll say. Um, it's, like, solved in the existing tokenization libraries outside of the engine. I think the exception to this being multimodal inputs and outputs. Um, we now have pretty good open weights multimodal input models. Um, and if you take a look at some traces, you'll see that there's, like,

still many opera- opportunities for optimization on this path. Um, tokenization of an image is just, like, a harder problem than tokenization of a, um, of a string of Unicode bytes, like conditioned on already having a, a tokenizer. There's more, more work to be done. You might need like FFmpeg, for instance, to like pull out raw frames to go into something that turns it into a tensor. Um, and so, yeah. So that, that's maybe the most interesting part there. Um,

the str- one way to sort of conceptualize the structure around the model forward codes is that the model forward code is, um, like going to call into a couple of different types of libraries or directly sort of write its own code to run on the GPU. Um, so a lot of it goes through like PyTorch code, which has its own kernel libraries and, and can run things on the GPU. Um, then there are like specific kernel libraries that you would draw from that aren't in the engine. And then finally, the

engine has its own, uh, set of like kernels that can directly, um, you know, run code on the GPU. Um, so picking that apart a little bit, the, um, uh, the kernels that exist in their backends, the core of language modeling architectures these days look something like this. Um, there's a sequence of layers, uh, that you take an input, you pass it, do some cross-token computation on the sequence, then you do some

per-token computation to enrich the sequence, um, and then you merge that back in with your representation, and you iterate this layer after layer. Um, and so this splits nicely into the two kind of classes of like really critical fast kernels that are available, especially via the backends. Um, so the cross-token computation is your attention or linear attention. Um, there's a bunch of different backends for attention. The FlashAttention kernels from Tri Dao and Jay Shah and others, uh,

FlashInfer from NVIDIA, uh, CUTLASS also from NVIDIA, um, the Triton, uh, implementations of these kernels, that's OpenAI's kernel authoring library. Um, the TRT-LLM, the engine has also split out its kernels, and they're available, um, as, as backends for attention. I think you'll see many of the same names that show up as attention backends also show up as backends for your matrix multiplications inside of your, um, your,

uh, per-token computation, which is your MLP layer. It's where the biggest, uh, GEMMs are. Uh, it's also, uh, in some architectures, this, this is a dense MLP, a classic neural network. In other architectures, it's these sparse mixture of experts, which basically just says, "Don't send every token to the same neural network. Per token, decide which smaller neural network to send it to." Um, so the, um- The dense MLPs, there's a, there's a nice, uh, library for this from, uh,

DeepSeek, DeepGEMM that gets used a lot in SGLang as kind of like the default. Um, in vLLM, there's some quantization-focused backends for this, the Marlin and bitsandbytes backends. On the, uh, for the grouped GEMM, you can split this out into first, like, there-- we have this like routing or communication problem of where do tokens go and how do I bring stuff together at the end. Um, so that can be done with a separate set of kernels like the, uh, for this all-to-all communication and

routing, like the-- another one from DeepSeek, DeepEP. Um, and that could be sort of factored out from the, like, matrix multiplication step, um, that ha- that can have performance penalties. And so you can also do that as like one big kernel, um, a, a mega kernel, uh, which is what is done in the DeepSeek V4 architecture.

Um, okay. So these, uh, with vLLM and SGLang, these are things that you can sort of set and control. In SGLang, it's, like, kinda pushed up into your face in the form of configuration at start with smart defaults. vLLM is a lot more focused on the, like, smart defaults side, and then you sort of like patch change things more with, um, environment variables. Um, at least last I checked, you know, these things change every, like, few days. Um, but yes.

Question.

Can you talk a bit about what happens when there are like thousands and millions of requests? Like, who decides how much KV cache to give to like each request, and how does everything happen? Like, there's been instances where like clouds stopped working when there was a lot of requests. So what happens on the inference side to deal with concurrency?

Yeah. So the question was basically about what's happening with concurrency with really large numbers of requests. Um, so that goes back-- So that's, uh, much higher level than the kernels and backends, um, that's happening sort of at our, like, batch construction and scheduler level. So basically, the scheduler is gonna do what-- the best that it can to decide like, um, which requests to run, em- generally emphasizing requests that already have KV cache allocated to them, um, 'cause then you don't need to recompute.

Um, the-- If you need to evict something from the cache to make space for another request, um, yeah, that will generally get like put out to CPU memory and then disk. Um, and I would say, yeah, to my point earlier about queuing, I think of that as essentially like a failure state of the engine, um, that we-- like I've deployed my engine improperly if I get to the case where I have KV cache thrashing or like things being flushed to disk. In the same way that there's like swap

memory in your operating system, like you can swap memory to disk, but at that point, your MacBook becomes unusable, right? Um, so it's, it's a useful feature to have, but almost like, um, so that you don't go hard down when you have a serving problem. Um, yeah.

Can I have one follow-up question? How does, how does serverless, uh, uh, inference engine deals with cold start problem? Like, for example, how is model compared to like AWS Institute or Google Cloud Run?

Yeah. So the question was about like serverless deployments and cold starts. I'm not gonna talk about that this much because I promised Swix I wouldn't do a vendor talk. Um, but please talk to me afterwards. There's a blog post called Truly Serverless GPUs that talks about how we solve that problem. Memory snapshotting, GPU memory snapshotting, custom file system, bunch of pieces to make the cold starts faster. Um, yes. Um, cool. All right. So let's talk about some of the key techniques that you need, uh, to implement to--

in- inside an inference engine to make sure, like, uh, your performance is good. Um, so attention is fundamentally quadratic, at least like the, uh, load-bearing part of the attention in, in every architecture that's popular. Um, and the, um, so the solution to that is to store things, um, after you've computed them, and you do like a little space-time trade-off for linear computational complexity in exchange for linear storage. Um, that runs into problems with how much can you

store in GPU memory because it's actually often slower to store this somewhere else, uh, and then load it back in as opposed to recomputing it. GPUs are fast. Um, so you want to be efficient with your use of GPU memory. Um, and then you often have requests that share prefixes. So for example, like all these commandments start with "Thou shalt not," um, and so you can reuse the computation of everywhere up to here. Um, as soon as a single token differs, you no longer can

share the prefix. So it's kind of this like tree or trie shaped structure. Um, but you want to be able to make use of that, especially if you have like a giant system prompt for like an agent system that's tens of thousands of tokens. Um, it's like quite nice to be able to share that across users. Um, and so the solution to this is like KV caching with pages or, or radices. Uh, so you store the information in this, um, in like a, a page cache data structure like you'd have in an operating system,

and then you just need to reconstruct these tensors sort of on the fly to pass them to attention kernels. So that, that job is now... It used to be the job of the engine to solve that problem. That's now incorporated into the attention kernels like FlashAttention-4, something that our team worked on, the, um, page size one, um, and sort of like radix, uh, form of KV caching that you can see on the right. Um, and, uh, yeah. So this is, this is moved a little bit. Uh, I put this in here 'cause it used to be like a defining feature of the engine,

but now it's kind of actually part of the kernels, and really it's the management of KV cache capacity and, um, and the like layout of the KV cache that's the problem of the engine.

Um, another problem I already talked about, how this, this scheduler can be this, um, host side bottleneck. What you want when you're running stuff on the GPU, like both at the sort of scheduler level and then inside of models, is that you want that just like at every moment, some kernel is running on the GPU, and things happen on the CPU to decide what that, uh, should run on the GPU, but they shouldn't ever block, uh, progress. But it's very easy to accidentally write some kind of host device sync. There does have to be information passed between them. The GPU can't just like

send the response directly to the user. Um, it wouldn't be that good at it if it did run an HTTP server. Um, and so you, you do need this CPU work. You just wanna make sure it doesn't block. Um, one of the key techniques for this is CUDA graph capture. So you take all of these kernels that get run in the forward pass of the model, and you track all of them, and you turn that into one big data structure, a DAG of operations of like this kernel runs and pr- and like, you know, mutates the data at this pointer.

Once that is done, this kernel can now run to mutate what was present at that same pointer and mix it with data from another pointer. So this big DAG of cu- of, uh, uh, of operations is your CUDA graph, and this, um, allows you to go from launching like each one of these GPU kernels having a little bit of CPU work to decide what to run, how to run it, uh, what, you know, which pointer to put. Um, you turn it into just one CPU side work to decide which graph to launch, and then all of it gets launched

from that single graph, which is what's depicted in this, um, uh, Perfetto trace. Uh, so this allows you to relax quite a bit about what is going on in the model, uh, forward pass component at least.

Um, another problem, decode here is sequential, um, and every time you run it, you have to load like either all of the parameters of the model or all of the active parameters in an MoE arc- architecture, um, and you need to do that over and over again, um, by default for each token. So you're, like on behalf of your users, you might be loading like a terabyte of data or a hundred gigabytes of data per token, and you wanna produce a token every couple milliseconds. Um, so now you're looking at that's petabyte per

second scale, uh, memory bandwidth. That's, uh, pretty hard to achieve. Um, so one solution is-- a key technique for solving this is speculative decoding, where you take, um, a separate model, the speculator, guess what the next couple of tokens will be, and then pass it through the target model. Um, and then the target model sort of grades all of those in parallel. Um, and if you apply the right sampling techniques, rejection sampling, um, then you can, uh, guarantee that, uh, the

output from the target model is the same as you would've gotten if you'd run it sequentially up to numerics. Um, the same output, uh, but you've now run it in parallel. You sort of turn your decodes into tiny pre-fills, which is nice. Um, and so the, um, I'll skip over a little any details about the algorithm. There's a great... We wrote a blog post about this, uh, just this past week or two. The key reason why this is really important is that as you improve the quality of the speculator, as you improve the number of tokens it

can guess in a row, um, you actually get basically a linear speed up in your decode throughput. So like many other optimizations that you can do on your inference engine are like the, it's like a game of inches, of like a percent, five percent, ten percent, like each time, and you have to do the like very like difficult performance engineering for each of those, you know, percentage points. Um, but then on the other hand, you could sit down and train a better speculator model, um, and

to first approximation, throw data and compute at it, um, and then get a two x speed up or a four x speed up or an eight x speed up, um, on top of the, the, the baseline. Um, so this is, this has emerged as one of the like key techniques to allow you to operate, say like a thousand tokens per second on, um, on NVIDIA hardware without having-- for actually fairly large models, uh, without having to go to, um, like a, a novel accelerator, like an LPU or Cerebrus.

Um, I'll actually get a... There's a lot of parallelism techniques, but in the interest of time, I'm gonna kinda skip over this one. Um, you want to be able to split work onto accelerators. Um, a lot of this comes from the underlying kernels. You need good kernels that can operate with this parallelism, but within the engine, in the sort of model forward pass code, which is written on a sort of per engine basis, um, you need to do all of the communications around that. Um, and so the communications is where a lot of the problems come in in these, uh,

parallelism, uh, techniques. Um, all right, so, uh, observability with our last couple minutes here. Um, inference services, like all services, will have some bugs in them. Um, you have everything from application level bugs, from models misbehaving, um, uh, which is really not the engine's problem, it's the app developer's problem, but they'll show up on your dashboard sometimes. Uh, model quality bugs, these are the problem of the engine. Are you actually doing your inference correctly? When you applied this

performance optimization, did you actually guarantee it didn't change model behavior? Um, one of the most common ones here is actually on model release, tokenizers are often slightly bugged. They both tokenize and apply chat templates, and there's often some like, uh, some issues there. Um, they're beaten down in a couple weeks after model release, but it's something you always have to be like on guard for. Um, and then performance bugs or engine performance issues, the kind of worst ones are regressions over time. As you operate a replica for an extended period, you might see performance degradation

from issues that only arise once the thing has been running for an extended period. Um, then you also might observe, especially in a heterogeneous cloud deployment environment, di- differences across replicas in their performance, um, that then need to be investigated. So the goal with these is with all kinds of like production debugging is observability. Um, if you can log enough information, uh, to debug just from those logs, then you don't have to spin up a development server and take a bunch of extra time. Um, and so this runs from application

level stuff, um, to, I think, focusing on the engine problems, model quality. You'll-- You want to be able-- You wanna be good at running evals for model quality as part of your deployment. Um, so you wanna hit your actual deployment with like evaluation benchmarking type scripts. Um, there's scripts in SGLang, there's scripts in vLLM, there's external benchmarking tools. Um, take those, uh, calculate some numbers, log traces somewhere, log as much data as you can about those so you can debug later. Um, and then in

prod log traces and try to get feedback from users on correctness that you can thread through all the way back to those requests. Um, for tokenizer bugs, hot tip, log the token IDs. Set your engine up so that the token IDs are logged as well. Um, and then finally, for performance, performance bugs can show up anywhere, so the more metrics that you log, uh, the better. Like, more metrics than you think you need, um, 'cause you never know which cross correlation will revea- will make the bug super easy to discover instead of super hard. Um, so rather than

going through those, I'm gonna talk a little bit about this. This is an endpoint dashboard on Modal and what-- like, how you might read it. Um, I was operating that, um, the, uh, art project, uh, that I showed at the beginning, and I just actually spun up like three different replicas of it, uh, to see what would happen, like under increased load. Um, so there was a traffic spike, which you can see down here, that turns into increasing, uh, first time to first token as things start to queue, uh, on the prefill side and, and don't

enter prefill, um, until, uh, the things have finished. Uh, and then also shows up as well as intertoken latency as, uh, decodes get first get a little bit slower and then a lot slower with this queuing as there's sort of contention on the underlying GPU. Um, and so the solution in this case with going back to the question about serverless deployments is this is detected as like increased load. You spin up new replicas. Those replicas come online. They start taking on some of this

request load. Um, and then the, um, like the amount queued, um, uh, decreases back down to baseline, and things are, um, you, your congestion is solved. So I think, um, that's the like primary way. I think in many cases, you know, we scale up additional resources to solve congestion and queuing. Um, so yeah. Um, when you need to go deeper, I'll skip over this, um, but there's the NSight Systems or Torch Profiler is one of your key

tools here. It allows you to see the whole engine at once. Um, it allows you to see all the processes that are running, all the work that's going on on the GPU, um, and pick out issues like, "Oh, this GPU is running, uh, like everything slower than the others." Turned out to be a NUMA awareness problem. Um, okay. So, uh, closing out here. Um, if you wanna learn more, miniSGLang and nano vLLM are a really great place to start. These are these simple implementations of the engine architecture, um, by the two

teams. They're great for humans. Um, they're incredible for agents. They can fit all the context in, you know, in their context window. Uh, Aleksa Gordic did a great walkthrough blog of vLLM, uh, and, uh, share these slides later. Um, also work through that in a notebook so you can play with it. And then actually, uh, you can point your coding agent at the coding base and ask questions. Cognition op- also operates this thing called DeepWiki, which basically scrolls these a bunch of repos, including SGLang and vLLM, and

writes nice architecture diagrams and makes this like wiki style, um, page where you can sort of read and ask, and then ask their, um, agent questions, which is a nice thing to pair with your own agent setup. Um, so that's everything. I'm out of time, so I'll just quickly say if you're interested in deploying and owning your own inference, we o- we have a, a new product on Modal to help you, like, get started with deployment really easily, get optim-- get an, uh, initial baseline of reasonably

optimized, uh, inference engine deployment that you can work off of with Modal endpoints. Um, so check that out. Uh, and if you're interested in working on deploying inference for hundreds of, of teams, uh, check out modal.jobs. Thank you.