AI Engineer World's Fair 2024
Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment
Read the talk
The Economics of LLM Inference: Prefill, KV Caches, and Engines Built for Real Traffic
LLM serving costs depend on what remains in GPU memory after a prompt arrives. Follow the path from tokenization and attention through workload measurement, batching, quantization, and hardware-specific deployment.
From a talk by Mark Moyou
Before you start: Familiarity with tokens, matrix multiplication, GPU memory, and basic model-serving terminology will help readers follow the prefill, attention, and deployment mechanics.
Inference economics begin with what stays on the GPU
A language-model deployment can become prohibitively expensive even when the model itself fits comfortably on a GPU. Every incoming prompt creates work and consumes memory, while each generated token adds context that must remain available for subsequent generation. At scale, the limiting question is not simply whether a model loads: it is how many active requests its remaining memory can sustain. Mark Moyou, an NVIDIA solutions architect, approaches inference economics through this distinction between resident model weights and the growing state required to serve users. 0:22
Real deployments often mix a commercial model API with an internally hosted open-source or fine-tuned model. Regardless of where a request goes, the underlying problem remains: the prompt enters the GPU workload, generation proceeds token by token, and previously generated context must remain available to produce a coherent continuation. 1:09
That retained context becomes the key-value cache, or KV cache, an important driver of deployment cost across both hosted APIs and self-managed GPUs. A request first has to be translated into the model’s vocabulary, then its entire prompt must be processed before token-by-token generation can begin. A heavily loaded service must perform that initial work for new users while continuing to generate responses for requests already in flight. Finally, generated model tokens must be converted back into human-readable text. 3:57
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From tokens to matrices to reusable attention state
Tokenization translates text into units from a model-specific vocabulary. Those units are chosen to represent the training corpus efficiently across natural languages, programming languages, and other text; they are not necessarily whole words. The vocabulary belongs to the model and tokenizer—for example, the Llama tokenizer Moyou uses—not to the application sending the request. 5:36
The inference pipeline has two distinct computational phases. During prefill, the system processes the complete prompt and computes attention over its tokens. During decoding, it generates subsequent tokens one at a time while retaining previously processed context on the GPU. These stages place different demands on hardware, and optimized implementations exploit different levels of GPU memory to accelerate them. 6:18
A useful rough estimate is that one token represents about four characters. Internally, each vocabulary entry has a numerical token ID, so incoming text becomes a sequence of integer lookups. The difference between a prompt ending in smart and one ending in smarter illustrates why word boundaries and token boundaries need not coincide: a small textual change can produce a different sequence of subword units. 6:49
Each token ID maps to an embedding vector: a numerical representation the model can compare and manipulate. Stack the vectors for an entire prompt together and the prompt becomes a matrix. Adding another token changes that matrix’s dimensions. GPUs are effective inference hardware because the workload repeatedly transforms and multiplies these large collections of vectors. 7:57
Attention determines which earlier tokens matter when interpreting context and choosing the next token. The prompt’s embedding matrix is multiplied by learned model-weight matrices to produce query, key, and value representations, conventionally abbreviated Q, K, and V. These projections describe the same token inputs in different mathematical spaces that the attention mechanism uses to relate context to the next generation step. 9:38
The keys and values are the reusable part. Saving them in the KV cache preserves the model’s relevant attention state instead of forcing the server to reconstruct the entire prompt every time another token appears. Making that cache smaller and faster directly affects both serving performance and the amount of context the GPU can hold. 12:09
The distinction becomes clearest with one request. Prefill processes the original prompt as matrix-matrix work. Once the first generated token is available, its embedding is processed against the existing model and cached context as vector-matrix work; its new keys and values extend the cache. Without that cache, each decode step would redo prior prompt processing. Production servers can batch these operations across many requests, but batching does not erase the underlying difference between a single request’s prefill and decode phases. 12:51
What did we calculate? What did we keep?
Moyou’s “Make me sound smart” prompt is already cached. Use word-sized tokens and an example continuation, “Here”, to follow one decode step through one attention head.
The current Q is new in both cases. These are vector counts, not latency estimates.
Four prompt positions cached. The new token has not been processed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Attention heads multiply the memory budget
Attention is not a single isolated calculation. An attention head contains its own query, key, and value projections, and multiple heads contribute to producing the next token. The particular Llama architecture Moyou uses as an example has 32 attention heads, meaning many sets of attention computations and associated cached state must be managed together; that count should not be generalized to every Llama variant. 14:11
At FP16, model weights require approximately two bytes per parameter. Moyou's simplified budget puts an eight-billion-parameter model on a 20 GB GPU:
Those remaining 4 GB must accommodate request context, KV-cache growth and other serving demands. Fitting the weights is not the same as serving concurrent requests. 14:35
Measure input and output lengths separately, then distinguish the timing questions:
ISLis input sequence length;OSLis output sequence length. 15:20- Time to first token: how long until prompt processing yields the first generated token?
- Inter-token latency: how long between subsequent tokens?
- Total generation time: how long from receiving the prompt to completing the answer? Growing memory pressure under concurrent load can change these measurements independently.
GPU utilization also depends on the type of workload. For some non-LLM deployments, colocating several models can increase throughput per hardware unit. LLM serving presents a different memory tradeoff: Moyou describes a typical production configuration as one model with remaining capacity devoted to active requests and their KV caches. More models are not automatically better if their weights displace the state needed for concurrent generation. 16:38
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Request shape determines what becomes expensive
A single benchmark shape cannot represent a production system whose users submit radically different prompts and expect different response lengths. The joint distribution of input and output sizes determines how much work occurs during prefill, how long decoding continues, and how much GPU memory remains occupied across overlapping requests. 17:23
The resulting request shapes expose different bottlenecks:
- Long input, short output: prefill must process substantial context, and the prompt occupies significant memory, but generation completes after relatively few tokens. 17:57
- Long input, long output: substantial prompt state combines with prolonged generation, filling the GPU and reducing the number of requests that can run concurrently.
- Short input, long output: the first token can arrive relatively quickly because prefill has less context to process, while decoding continues across a much longer response.
- Short input, short output completes the four-way classification: both the prompt and continuation are short. Actual traffic mixes all four shapes, so concurrent memory occupancy depends on their distribution.
Shrinking model weights can free space for additional tokens, but weight compression alone does not describe the complete optimization problem. The engine must also accommodate the observed request shapes, their cached context, and the traffic arriving concurrently. A two-dimensional histogram of ISL against OSL reveals which combinations actually occur rather than relying exclusively on advertised maximum context sizes. 19:16
That distribution informs the maximum input and output boundaries used to size and optimize an inference engine. It also provides a basis for understanding when capacity should grow, how requests compete for available memory, and why a deployment tuned for one workload can perform differently when the traffic mix changes. 20:17
A long prompt and a long answer cost different work.
Four teaching examples isolate input length from output length. The same prompt and response are reused across matching conditions; fragments are not tokenizer measurements.
short input / short output
Summarize the notes.
Owners assigned.
long input / short output
Summarize the meeting notes covering decisions, risks, owners, and next steps.
Owners assigned.
short input / long output
Summarize the notes.
Owners assigned; risks documented; decisions recorded; follow-ups scheduled.
long input / long output
Summarize the meeting notes covering decisions, risks, owners, and next steps.
Owners assigned; risks documented; decisions recorded; follow-ups scheduled.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diagnose latency under production load
Time to first token must be measured under meaningful concurrency. A single-request prefill measurement does not establish how attention processing behaves when the same server is busy admitting new prompts and generating tokens for existing users. The useful comparison is therefore between workloads with stated load conditions, not an isolated query and a saturated deployment treated as equivalent. 20:41
GenAI-Perf is the benchmarking package Moyou identifies for generating plots that expose these serving behaviors. One useful view examines time to completion across requests: a broad distribution invites investigation into whether different prompt shapes, batching decisions, or scheduling behavior explain why otherwise comparable requests finish at different times. 21:02
A second view plots token-to-token latency against token position. As a response grows, the retained context and memory pressure also grow; substantial variation between later token intervals can indicate that concurrency or scheduling is degrading generation. Stable spacing across positions is the operational behavior to investigate, rather than assuming that a fast first token guarantees a smooth response. 21:57
A third view plots time to first token against input length. Longer prompts require more initial attention work, but the slope reveals how sharply prompt-processing latency grows as the input expands. Comparing that relationship under realistic load makes it possible to distinguish an engine that handles growing context efficiently from one whose prefill cost rises too steeply. 22:29
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compile the model and optimize the serving loop
TensorRT-LLM, or TRT-LLM, is the NVIDIA inference and model-compilation framework Moyou introduces after establishing the workload and its measurements. Triton Inference Server provides a separate serving layer that can host models across different frameworks and hardware targets, reducing the need to maintain an independent model server for each deployment stack. NVIDIA Inference Microservice, or NIM, packages an enterprise-oriented path for deploying optimized configurations without reproducing every engine-building step manually. 23:07
One optimization changes numerical precision. Moving weights from FP16 to FP8 reduces their representation from 16 bits to eight bits, potentially freeing approximately half of the weight-storage memory for the same parameter count. Moyou describes accuracy preservation and faster execution as goals that must be evaluated for the model and supported hardware, rather than as unconditional results for every workload. He identifies Hopper and Ada Lovelace hardware when discussing FP8. 24:15
Other serving optimizations target the active workload directly:
- In-flight batching admits a new request as soon as another finishes, without waiting for every request in an existing batch to complete. 24:52
- Quantized KV cache stores cached attention state at reduced precision, shrinking the memory required by active contexts.
- Paged KV cache changes how cache memory is managed across requests, improving control over GPU memory allocation rather than merely compressing model weights.
- Tensor parallelism splits a model across multiple GPUs, typically within one node.
- Pipeline parallelism processes successive model partitions sequentially and can pass work between nodes for larger deployments.
These strategies solve different problems: batching improves utilization as requests arrive and finish, cache techniques change how much active context fits in memory, and parallelism distributes model execution across devices. Moyou describes an NVIDIA 340-billion-parameter model designed for inference on a single H100 node using FP8, illustrating that model placement depends on both hardware topology and numerical representation. 26:05
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy hardware-specific engines behind a serving layer
TensorRT-LLM extends the earlier TensorRT compilation approach with the additional operations required by the autoregressive inference loop. In the workflow Moyou describes, an engine is compiled for its target GPU rather than treated as a generic artifact that can simply be moved unchanged onto different hardware; the optimization depends on the characteristics of the deployment device. 26:36
The compiled engine still needs a process that accepts requests, manages batching, and exposes model execution to clients. Triton Inference Server fills that role: it loads models from a configured repository and describes execution in terms of tensor inputs and tensor outputs. The division of responsibility is straightforward: TensorRT-LLM produces the optimized inference engine, while Triton hosts and schedules requests against it. 27:25
Moyou’s deployment outlook moves from FP16 toward FP8, because lower-precision weights can leave more room for active tokens, and then toward the announced Blackwell generation’s FP4 capability. Each lower-precision deployment still needs its own quality and performance measurements. NVIDIA Inference Microservice rounds out that outlook by packaging model-specific and GPU-specific configurations as an enterprise deployment option. 27:52
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adapt engine boundaries to observed traffic
An ISL-by-OSL heat map becomes operationally useful when its observed distribution challenges the configuration used to build an engine. An organization might provision for prompts of 4,000 tokens, then discover that actual requests over the previous week were closer to 1,300 tokens. Those observations can justify rebuilding around the measured workload instead of optimizing exclusively for a theoretical maximum. 29:06
Moyou calls the possibility of varying configurations over time seasonal engines: daytime and nighttime traffic, or other recurring demand changes, might warrant differently sized engines and different scaling behavior. The heat map supplies the input and output bounds across the observed distribution; it does not, by itself, establish that dynamically replacing engines will improve every deployment. 29:55
For mixed request shapes, the typical starting point Moyou describes is one engine configuration capable of handling the full range of incoming traffic. Trying multiple configurations becomes an engineering experiment: check whether a replacement still serves the observed request distribution, then measure its cost and execution behavior. Changing traffic from agent-based systems could make those workload shifts more pronounced. 30:30
Longer context also raises a more fundamental question: can attention scale without making memory requirements prohibitive? Moyou describes approaches that train at shorter context lengths and interpolate to extend usable context, alongside attention implementations that exploit fast GPU memory and hardware interconnects such as NVLink to move data between GPUs. His expectation is that continued architectural and hardware work will address attention bottlenecks, rather than treating ever-larger context as free. 31:38
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore the inference framework used for model execution, batching and KV-cache optimizations on NVIDIA GPUs.
Serve the inference engine behind an endpoint that accepts and schedules model requests.
Inspect the benchmarking tool named in the talk for time to first token, inter-token latency, throughput and request-load analysis.
Further reading
Moyou's deployment walkthrough supplies environment checks, NIM launch scripts and example inference requests.
NVIDIA's technical overview expands the prefill/decode, KV-cache and model-parallelism mechanisms behind the talk.
Updates since the talk
A December 2025 maintenance notice directs new benchmarking projects to AIPerf while the recording demonstrates GenAI-Perf.
Related talks
- Low Level Technicals of LLMs
Connect implementation fidelity with KV-cache reuse and inference workload tradeoffs.
Read the complete timestamped transcript
- 0:00
[upbeat music]
- 0:12
It's very difficult to teach, um, extremely technical material in about twenty minutes. Initially, I had planned for at least a forty-five minute session, so I left some reading material for you at the end.
- 0:22
And, um, all of the resources, you can download slides and everything, so feel free to take screenshots or not. And so I work at NVIDIA, I'm a solutions architect, so I work primarily with retail clients, and it's my job to essentially work with those clients, understand sort of what their main challenges are.
- 0:38
This is data processing, computer vision, um, across all of the different use cases. And then now I'm focused on LLM inference. So my hope today is that, um, you get a better intuition of exactly what's happening with this particular workload and how you go about, uh, to some degree, sizing things, choosing different GPUs, et cetera, et cetera,
- 0:57
and more importantly, controlling the cost of a deployment. 'Cause that's oftentimes the, the thing that's going to really prevent you from taking this to, taking this to any meaningful scale, is that overall cost of a deployment.
- 1:09
Most folks that I've seen are doing some kind of hybrid. So you choose a big box API, you have some set of queries that go there, in addition to you have some set of queries that go to some open source, you know, hosted model or some fine-tuned model that you have internally.
- 1:25
So just reference, um, if you go to build.nvidia.com or ai.nvidia.com, everyone can get, uh, a thousand inference requests for free. So I typically recommend this to folks who are benchmarking, um, different types of open source models.
- 1:38
We have all of those models hosted. It's optimized if you're teaching a course, um, and you are trying to evaluate all of the different LLMs that are out there for your business.
- 1:47
There are also multimodal LLMs, um, speech LLMs. Every model that NVIDIA accelerates will be available there for you, and that's sort of a path to you, um, to either go optimize them yourselves or to work with us.
- 2:00
Um, you'll see things about, uh, NVIDIA Inference Microservice, and all of those things that you can take to enterprise. So we have sometimes the, um, I call it the rocky road, and then there's smooth roads.
- 2:11
Whatever path you want to take, we're here to support you.
- 2:15
Uh, in terms of agenda, very simple. I, I want you to understand the LLM inference workload, and then we'll move to, um, how you go about measuring a production deployment and some of the things you need to be watching.
- 2:26
It's a little more than, um, let's say, you know, the total time to generation, and really understanding what's happening on the GPUs as you sort of scale out. Even if you have a single GPU, I think it's very important for you to just have that intuition.
- 2:38
And then lastly, I'll show you some software that you can use, um, some open source packages that you can use, and then point to, uh, some paid offerings. Okay, we're gonna get into the LLM inference workload itself.
- 2:50
So the first part is, is really understanding what happens when you send a prompt onto the GPU. So I have this example here, I'm saying, "Okay, write me a presentation so I sound smart, I come to the AI engineer conference, and you guys are maybe gonna like the talk."
- 3:04
And essentially what I'm gonna do is I'm gonna put that on the GPU. So the moment that I send that prompt on the GPU, it stays on the GPU.
- 3:10
So think about that, and then from there, I'm going to generate one token at a time. So I'm generating the tokens, "LLM inference is hard," and I put the timestamps T1 through T4.
- 3:21
So in every single deployment, no matter how fast anyone claims they're, they're doing things, um, it's typically one token that's generated at a time. That's very important to understand.
- 3:31
The next thing is that in order for an LLM to give you a coherent answer, just like how you speak, you have to remember every single thing that you said before, and that you'll understand the mechanism of, um, how that-- how LLMs are able to do that.
- 3:45
So that's why I'm putting, "LLM inference is," in red and putting that back onto the GPU. So every token that I generate gets locked onto the GPU, and then you'll actually see what that looks like, um, in terms of vectors.
- 3:57
How many of you have heard of KV cache before? [coughs]
- 4:01
Okay, some of you. Um, typically I don't see maybe many leaders hear about this thing called KV cache. KV cache is this thing that really drives, to some degree, the cost.
- 4:11
Uh, so whe-whether or not you use some big box API or, um, you're using a single GPU, it's all the same sort of mechanisms, the same algorithm that everyone is trying to solve.
- 4:22
So in terms of steps, here the, the... I, I like to, as I said, sharpen your intuition. So the first thing, if we move from the left, my first job is to convert these text, whatever text that you send, we're gonna focus on LLM inference, into some words that the model understands.
- 4:37
So the model will have its own vocabulary, and it's my job to translate that. I'll give you the technical, uh, terms coming up after that. And the first thing that happens is I do some initial prompt processing.
- 4:48
So I have to compute the attention mechanism on the entire prompt. I repeat that. I have to compute the attention mechanism on the entire prompt per user. So if I have a million people hitting my service, and a million people send ten thousand tokens, that's a million times ten thousand attention mechanisms that I need to compute, also
- 5:08
while generating tokens for other people. So that's-- it's good for you to appreciate sort of that complexity that's happening. And once I finish processing that prompt, then I'm going to start generating one token at a time, and that typically happens very fast.
- 5:21
And then from there, every token that gets generated, that's in the LLM's vocabulary, I need to now de- tokenize that back into your language. So here's the technical terms that you'll see when you read the literature or you read, uh, super technical documents.
- 5:36
First, there's tokenization. Each model will have its own tokenizer. And, um, the thing to think about when you think of tokenizers, when they did pre-training, they had... they downloaded the Internet and some, right?
- 5:49
And they cleaned it up, et cetera, et cetera. So a tokenizer, and as you start thinking of the complexity across languages, coding languages, uh, regions, et cetera, et cetera, they try to get what is the minimal set- Of character groups that can represent this entire training data set efficiently, because it's really all about efficiency.
- 6:09
So for instance, the LLaMA tokenizer has one hundred and twenty thousand tokens, right? And I'll, I'll talk a bit more about that. So here's what it actually kinda looks like on a GPU.
- 6:18
So I tokenize, the LLM understands it. I go into this thing called prefill. Prefill is a stage where you compute the attention mechanism, and many people are doing advancements with attention mechanisms.
- 6:29
Um, I'll talk a bit more about that. So there are tons of different schemes. People leverage, um, all of the different types of memory hierarchies in GPUs to really accelerate this type of workload.
- 6:39
And then I start generating tokens one at a time. The red and the green just signify, "Hey, I'm storing those tokens on the GPU." The green is the latest one that I sent out.
- 6:49
So hopefully that makes sort of intuitive sense. All right, so the other thing I want you to visualize, um, I, I think it's nice to visualize what, what is the actual data that sits on the GPU.
- 7:02
So first, a, a token is approximately four characters. That's a nice way for you to think about it. Um, so from here I have, uh, two vectors. So the first vector is just showing token one through token V.
- 7:14
V is the total number of tokens that I have in my tokenizer. And the second vector below is just I have some numeric index. I don't want to keep using the token to reference itself.
- 7:24
I just use, use the number as a lookup. So my job when a prompt comes in is to convert that text into those token, what I'm gonna call token IDs.
- 7:34
Okay? So I, I have, "Make me sound smart," and "Make me sound smarter." You see two vector sets of tokens, and the key thing I want you to walk away with from, from that distinction is that an LLM's token is not a human word.
- 7:48
Sometimes it is, sometimes it's not. It's typically some subpart of words. You'll see weird symbols when you look at tokenizers from different models, um, but you want that first framing.
- 7:57
So now we have text, we hit to a vector, right? So from there, each one of those LLM tokens had a corresponding embedding vector. So embedding vector is everything.
- 8:08
We embed, uh, videos, we embed images, we embed text tokens. Think of it as a representation that, uh, an LLM can use to compare things and do math on.
- 8:19
So that's why we always want to com-- excuse me, convert into some vector representation, right? 'Cause some vector representation is just some high-dimensional coordinate space, and we're just rearranging objects.
- 8:30
That's to some degree what you're doing. Okay, so from those token IDs, I went to the actual, um, embedding vectors themselves. So if you look, "Make me sound smart," now becomes a matrix, right?
- 8:42
"Make me sound smarter," becomes a matrix with an extra column. So in reality, what you're doing every time you submit a prompt, I don't care what LLM you submit it to, who you submit it to, this is what you're doing, right?
- 8:55
You are converting your text, now images as well, they get converted to some, uh, either tokens or something like that. That'll be another interesting talk to do diffusion models, et cetera, et cetera.
- 9:05
But you're really putting this large matrix on the GPU. So why that-- The next question you should ask is, "Okay, um, why are GPUs good for this workload?" 'Cause they process matrices really, really fast.
- 9:17
So that's sort of the, the advantage and the thing hopefully that, that makes a lot more sense to you. Now, the next thing I want to talk about is how the LLM is going to process these tokens.
- 9:27
And now keep in mind if any-- Well, I'm not even gonna ask you to raise your hand. I'm a hundred percent sure each of you has used an LLM.
- 9:33
If you have not, I'm not sure what's happening. [laughs]
- 9:38
Uh, the other one is the attention mechanism. I, I truly think, um, it's one of the things that you should understand. If we ever drift away from it, that's fine, but the fundamentals of that mechanism and, and seeing sort of the innovations around that I think can help anyone, any business leader, et cetera, et cetera, just because
- 9:56
you are able to speak a different kind of language, um, in this generative future. So as you think of the attention mechanism, the intuition that you should have is just that mechanism of relating tokens.
- 10:07
How do I distinguish in a sentence what is important, right? And then for the next token that's gonna be generated, hey, what tokens that I said before were really important for me to make a next good decision for that next token?
- 10:20
So that's the intuition, and now we're gonna... we won't necessarily touch too much on the math, but I want you to see sort of what's happening on the GPU.
- 10:27
So once again, the prompt comes in. I'm just gonna do a short one, "Make me sound smart." I'm gonna generate this token called LLM, right? Uh, we saw these same matrices that I said before, so remember my text now turns into a matrix heading onto the GPU.
- 10:42
And, and the main thing I want you to understand or visualize here is actually how
- 10:48
an LLM memory works. So now when you're speaking, you've recorded everything that I've said for the last, uh, ten minutes in your brain. Somewhere it's, it's stored. So now you're gonna see how the LLM is storing what it is that you just said.
- 11:02
So from there, um, a lot of folks will hear about these query, key, and value matrices. This is what the actual model weights look like. So when you look at a model weights file, if you go on Hugging Face, there's typically a JSON file that will show you all of the different pieces of model files, and you'll
- 11:17
see this thing called Q, K, and V. So I have these model weights. So now I've went from text to a matrix. I'm going to matrix multiply against the weights of the models.
- 11:27
So now I get these three output models. So think of these weight matrices that I showed here, think as, um, when you're doing a projection, what you're doing is you're taking, um, some coordinates and you're putting it into a different space.
- 11:41
That's really what you're doing when you do vector matrix math. So now when I do this matrix multiplication, this query, key, and value matrix, so if you look at different tutorials on attention, you'll see these things pop up a lot.
- 11:52
So hopefully that'll help you to read it a lot more. This is now the LLM's interpretation of each of those tokens that you sent in, right? And now the job is how do I now take, uh, these query, key, and value matrices and sort of interpret it- To try to generate the next best token.
- 12:09
And this is just happening constantly over and over every single token that's happening. But the key thing I want you to walk away on this slide is where I drew the key and the value.
- 12:19
Right? When people talk about KV cache optimization, every LLM performance engineer is just literally trying to make that thing as fast and small as possible. And I'll, I'll-- that'll make a little more sense as to what that does to your cost.
- 12:32
But ultimately, these key and value matrices, this is like your LLM's memory. So it'll make a little more sense coming up. I know I didn't show a ton of the math.
- 12:39
I show some tutorials afterwards, so you can go read more about that. Um, my intention here is for you to visualize key and value. So every time you see a prompt, I just want you thinking, "Crap, key and value is on my, on my GPU."
- 12:51
Okay? The next. So here's the real value of the KV cache. So remember we said that whenever I generate a token, I'm going to push it back into the GPU, right?
- 13:04
So every token I generate, it goes back into the GPU, and then I have to compute an attention mechanism. So this is what's happening. This new token I generated, LLM, I get its vector representation, as you see in blue, but now I, I do that vector matrix math now.
- 13:19
So before I did matrix-matrix math, that's my first prompt first comes in. I generated my first token. Now I'm doing vector matrix math. You know, people will batch this across all requests, but I'm just showing you a single request so you can see it.
- 13:34
Now, the value of the KV cache is if I were to-- if I didn't have the KV cache, I would have to reprocess all of that work I did on the prompt that I did before.
- 13:46
So this is the benefit of your KV cache, is to now I'm just gonna compute attention on this newest token. How does this new token relate to everything that I said before?
- 13:54
That's the thing that's really happening, uh, intuitively. So if I have this KV cache, my generation is gonna be fast. Okay? And it, it's really up to the, the what's called a batch manager on the GPU to make sure that I'm just pushing out as many tokens as possible.
- 14:11
Okay. So if you look at, uh, an LLM, these groups of three matrices are called an attention head. There are more matrices than that, but these are the main ones.
- 14:20
Um, Llama has thirty-two attention heads, so I just kinda want you to appreciate what an LLM really looks like, right? So I have thirty-two sets of these matrices. I have thirty-two of those KV caches happening at the same time, and now I have to combine all of that to then generate the next token.
- 14:35
So there's, like, an incredible amount of work that happens in a very short space of time to give you a coherent token. Okay. A good mental model for you to keep in your head, I'm gonna speed up a little bit, is to, um, if you see the number of parameters, multiply that by two, and that is your
- 14:52
FP16 gigabyte memory on the GPU. So if you have, let's say, an L4, I think is twenty gigs, um, and I have a Llama 8B, that's automatically sixteen gigs FP16.
- 15:03
So I only have four gigs left for my KV cache. So on the GPU, it's either the model weights or tokens. That's it. There's nothing else on the GPU.
- 15:12
And I have, I have a thing to read on that. This is a really good blog. It shows you all of the different, um, optimizations that you can do and...
- 15:20
Okay, now let's talk about measuring. So if you ever see this thing called ISL or... Do I have it here? Oh, sorry. ISL or OSL, that's input sequence length, output sequence length.
- 15:32
So now I want you to see what some advanced monitoring might look like. If any of you are DevOps folks, these are things that you wanna record. The first thing that we measure is time to first token.
- 15:41
So how long does it take me to generate, um, the-- process the prompt and then generate my first token? And that's typically a measurement of how good your attention mechanism processing is.
- 15:52
That's really what you're trying to suss out. So that's time to first token. Inter-token latencies, so after I've generated my first token, every single token after that, I'm looking at those individual spaces.
- 16:03
So everything that's going to happen there, uh, think about when the system is under load. I have, you know, a thousand requests coming into my system. I'm generating a thousand sets of different tokens.
- 16:13
And the more memory I occupy, typically that slows down processing. So if you start to see drift in this metric, then I'll show you some plots that you can look at, and then time to total generation, how long did it take me to initially get the prompt, fully finish the answer, right?
- 16:27
Super intuitive. Like I said, ISL, OSL, that's all that means when you see them on the plots coming up. Okay, uh, this is a very important paradigm for you to understand in your mind.
- 16:38
Uh, so I, I worked with a lot of folks on, you know, com-- maybe, uh, REXX deployments or deployments of other types of models. So on the GPU, if you're only deploying one model on a GPU outside of LLM inference, uh, in my opinion, I think you're wasting the GPU.
- 16:52
You can put multiple models on the GPU to actually increase your throughput. That's why it was really created. Um, so this is a slide... Excuse me. This figure is just showing I can have multiple models.
- 17:03
I have some space for data, and that's how I increase my throughput per unit hardware. However, on the LLM inference side, it's very different. I have one model. You know, folks can fit multiple models on a GPU.
- 17:14
That's cool, but that's not a real production use case. You'll typically have a single model. The remaining space that you have is all for KV cache and generating all those tokens.
- 17:23
So I just put four different requests, and I, I just kinda want you to see the boxes that are happening. Okay? Uh, I would say this is the most important slide in the entire presentation because this is the thing that will determine both your cost and performance.
- 17:36
So there are four different querying patterns that happen, and this is something that you must measure in your deployment, 'cause oftentimes you might read benchmarks, and it'll just say, all right, they'll cherry-pick one or two of these.
- 17:48
But in reality, in your production system, you might have several of these different patterns that are occurring. So let's take a look at the first one, long input, short output.
- 17:57
So a long input means it's gonna take me technically longer to compute the attention mechanism. So my prefill stage will be longer. I occupy more memory from my prompt.
- 18:07
Does that make sense intuitively? Hopefully, it, it's grabbing you. But then on the generation side, I don't generate much tokens, so there's not much Those tokens are not picking up a lot of memory, and they will tend to finish fast.
- 18:19
So the second, the second one, or the maybe the most costly use case is, um, so I have clients that will message me and say, "Hey, my data scientists are putting two bigger prompts on my GPUs, so now they're killing my deployment."
- 18:31
'Cause if everyone went and put the maximum context length, I can only fit so many requests on the GPU. So that's something for you to think about. You'll have to manage that internally with your deployments.
- 18:42
So that's why I'm putting, you know, okay, the GPU is really full 'cause a long text, uh, excuse me, long input, a long output. The next one, short-long, you know, your time to first token be really fast.
- 18:51
I don't have much to compute the attention mechanism on, but hey, I'm generating a ton of tokens. That's really, really fast. So hopefully, as you start measuring these types of different query patterns, um, you'll see different results.
- 19:03
I just put, you know, what a random sampling set might actually look like on the GPU, 'cause not everyone will send the same length of input and output. Um, so that will-- it'll be good for you to just sort of visualize and track these statistics.
- 19:16
More importantly, why we're doing that, uh, internally, I'm, I'm gonna steal the time here, Peter. Uh, more importantly, why we're doing that or why we're tracking these things is that the whole goal is to build...
- 19:28
I have a big model. My goal is to shrink it as much as I can, but to keep it as accurate as possible. So the more that I shrink, the faster it runs, the more GPU memory I have for what?
- 19:39
Tokens. All right? So that's how you really try to improve your cost. This is why I'm, I'm sort of proposing to you to build inference engines. So what I-- all I'm showing here is a 2D histogram of input sequence length versus output sequence length.
- 19:53
'Cause the question that you'll have to answer is, "Hey, how long are my actual prompts?" Someone might say, "Okay, here's the max prompt length that you can ingest, or the max prompt you can out, you know, excuse me, get on the output."
- 20:05
And all of the big box model providers have to estimate this when they go into costing or providing a service to you, right? 'Cause they have to host all of that machinery under the hood now that you understand what's happening.
- 20:17
So we use this stat- to statistically determine what is the max input sequence length and the max output sequence length across all of my users. And this would give you a really good indication of, um, how you can size your engines.
- 20:31
We use that to actually build more optimized engines. In addition, um, it'll just give you good view as to maybe, uh, what you call it, scaling out and things like that.
- 20:41
The next one is time to first token analysis. Remember, time to first token is measuring my performance of the attention mechanism under load. So someone might show attention mechanism at one query.
- 20:51
Woohoo. Show me attention mechanism under load. When this thing is fully maxed out twenty-four/seven, that's when you really need to start, um, measuring these types of things. So this is something you can look at.
- 21:02
These are sort of experimental plots. Um, there's a package called GenAI Perf that will be released open source. It, it's out already. I have a link to it there.
- 21:10
This is where you can-- it'll generate these plots for you. But I, I'm just showing you what the engineers are looking at internally to measure the performance of, of the compute platform.
- 21:19
Next, time to completion analysis. How long did it take me to go from start to finish across every single request? Naturally, the wider that box plot, you have to intuitively ask, "What's happening?
- 21:31
Why did this person's prompt take longer than another?" So you can investigate either batching issues, scheduling issues, different things like that. I'll, I'll take questions in the end. Um, or I have to move really fast.
- 21:41
Sorry there, Peter. Okay, I'm gonna speed up here. Uh, token-to-token latency. Peter, how much time I got?
- 21:47
Oh, you're fine. Yeah, we'll definitely have time for questions.
- 21:49
Okay, cool. I, I'm gonna steal... I'm definitely over, so sorry. I just wanna-- I realize I may have gone a little too fast, so forgive me for that.
- 21:56
No, you've got five minutes.
- 21:57
Cool. All right. Time, uh, token-to-token latency. So that is, I'm generating tokens, so I'm looking at that spacing versus token position. So the longer a sequence gets, remember, my memory grows, so typically that means our system is under more load, it has more throttling that might happen under high load of requests.
- 22:17
So if I see a large variation in token-to-token latency as the sequence gets longer when I'm generating, that means I'm not very performant, right? So we look at that to see, I try to make sure that that's constant.
- 22:29
No matter how much tokens I'm generating, that means I'm, I'm really proficient. Okay? Uh, last one would be time to first token versus number of input tokens. So time to first token, remember, is computing the attention mechanism, okay, versus number of input tokens.
- 22:46
So if I have a bigger prompt, my attentions will take longer. But I-- if that plot goes up like, uh, from your perspective, it goes up like this in terms of sequence length, that's not really good performance.
- 22:57
We really look at that slope, and we try to get that slope almost, you know, as, as low as possible. So if you send me this long sequence, I can get that thing done really fast.
- 23:07
Okay? Okay. Uh, in terms of software, uh, you'll see this thing called TRT-LLM, uh, Triton as an open source inference server. So you can deploy models on CPU, on GPU, computer vision, Raxas, Python, PyTorch, TensorFlow.
- 23:21
It's, uh, it'll host all of the different types of models, so there's one way that your deployment team deploys. All the data scientists are happy 'cause they don't have to do conversion.
- 23:29
You're happy as a deployment person 'cause you don't have to manage a TorchServe versus TSServe and, uh, Flask, and all of it is done through one. It's written in C++, blazingly fast.
- 23:39
And then the other thing you'll see in NVIDIA, you'll see a lot more coming out of NVIDIA's NVIDIA Inference Microservice, 'cause building these engines, getting them deployed, optimizer scale is not easy.
- 23:49
So we've sort of made that easy for you as an enterprise offering, but you guys can try it out, uh, for free. Okay. So TRT-LLM, let me just give you a high level.
- 23:57
Lots of stuff on this slide, but the main thing I want you to walk away with, um, uh, is this is the model compilation package for LLMs on NVIDIA GPUs.
- 24:05
This, if you want to get best performance from NVIDIA GPUs, please make sure you use TRT-LLM. Um, and naturally, uh, once we're investing more in NIM, you'll see some more things come out.
- 24:15
So you'll see performances on A100 and H100 really focus on FP8 GPUs. So FP8 will be Hopper and Ada Lovelace. Okay, so FP8, I'll, I'll talk a bit more about that, what the advantage there is.
- 24:28
But mainly is if I go from FP16, FP8 is this, half my memory, almost the same accuracy. And so we measured the accuracy, and we publish the accuracy. So now I have this much more space for tokens, but more importantly, this model is that much faster.
- 24:44
Okay, so I want you to understand where the sort of industry is going. This is why Hopper, the world ate Hopper for breakfast and lunch and dinner because of FP8.
- 24:52
It gave folks that cost, um, benefit to do this thing a lot faster, okay? Um, in-flight batching, it just means I don't have to wait for all the requests to finish to start a new request.
- 25:04
The moment your request finishes, I can in-inject a new request while others are going. Okay? Tons of features here. I, I put the features. Um, so some ones to, to focus on are quantized KV cache, so I can actually represent my KV cache in different, uh, excuse me, precision.
- 25:21
So that means I'm actively shrinking that memory, making it more performant. Um, you have paged KV cache, that's just you managing, uh, your GPUs a lot better in terms of all of that memory.
- 25:30
So there are tons of things you can do. Tensor parallelism, the thing to remember about tensor parallelism, if you want to increase latency, you use tensor parallelism. Split the model up across multiple GPUs.
- 25:41
That's typically done within a node. I repeat that. That's typically done within a node. You don't like to do tensor parallelism across a node. You'll see pipeline parallelism go across a node.
- 25:53
Pipeline parallelism is more sequential, so I process this chunk. So in a multi-node model, like huge models, this box will finish and pass off to the next box. But most folks will typically just work...
- 26:05
Most models will work within a single node, like, um, NVIDIA's 340B model that they just released. It was designed to do inference on a single H100 node, but that's an FP8.
- 26:16
Okay. So those are some of the things. In terms of models that you have access to, um, we optimize those models, and we give you a lot of scripts where you can go do that on your own, or you can sort of take our software, um, and take an easy path.
- 26:28
You-- Either way, we support you. So here are some of the models that are there. All of the Llamas, Mistral, Mixtrals, we work with all those teams behind the scenes.
- 26:36
So typically, before any foundation model comes out, we, we work with those teams to, to get them deployed. Okay, what does it mean for TensorRT? Uh, so you might have seen TensorRT before, which was a deep learning compilation package for NVIDIA GPUs.
- 26:50
Lots of folks in computer vision, et cetera, et cetera, have used that. We took the best practices from there and added all of the extra things that need to happen in the LLM inference loop.
- 27:00
So that's what TRT-LLM is really about. Um, so mainly focus on LLM inference. Uh, here's a good visual. An engine that's built to a specific GPU cannot be moved to another GPU.
- 27:12
So you always have to compile to that GPU. That's why it's that performant, 'cause we really leverage, um, all of the, the actual hardware on that system to rewrite the algorithm, rewrite that model to that specific piece of hardware.
- 27:25
Okay? Um, TRT-LLM on Triton. So TRT-LLM will give me an inference engine. I need something to host that inference engine and accept requests, batching, et cetera, et cetera. So we have Triton.
- 27:37
Triton works very simply. It's literally a folder where you specify it, it works on tensor in and tensor out. So it'll tell you what are my inputs coming in and out, and then it'll basically understand how to interpret that file, or you can host any other different models.
- 27:52
That's one-- That's a thing I do a lot with folks. Just two more slides. This is where the future of inference is going. So a lot of folks do FP16 inference today.
- 28:01
Um, a lot of folks are moving towards FP8 just because, hey, I know half the model size, almost twice the speed, more space for tokens. It just makes more sense from a cost perspective.
- 28:13
That's why folks like that. And then, uh, you saw Blackwell was announced. That's the major innovation. I get FP4. So that's where things are really gonna get interesting. I'll end with, um, NVIDIA Inference Microservice.
- 28:25
So we've made this thing really easy. We've gone and actually found the best configurations for all of these models on each piece of GPU, and we're slowly rolling out, um, all of the models 'cause it, you know, it'll just take some time to optimize the world, [chuckles] essentially.
- 28:38
And yeah, you can use this to download all the slides. I put papers, um, tons of other things you, for you to read. So yeah, hopefully your, your intuition has sharpened. [laughs] [audience applauding]
- 28:53
Shall we just conclude with the... 'Cause there was someone had a question.
- 28:56
Sure. Yeah, I think-
- 28:57
Where was the question?
- 28:58
Yeah.
- 28:59
Oh, hang on. I'm gonna come over and, uh, point my mic at you.
- 29:06
Thank you. Uh, so hi. Um, sorry. My question is actually on the heat map that you shared.
- 29:12
Yeah, yeah.
- 29:12
Do you mind, um, walking through the heat map and how to interpret it? 'Cause it was a little small, couldn't really-
- 29:17
Yeah, sorry about that. Yeah. So the heat map-
- 29:18
Thanks
- 29:19
... all I'm looking at is, um, so when you go to build an engine, you build an engine to the max input sequence length and the max output sequence length.
- 29:27
So we actually change how that matrix math is happening under the hood based on those settings. So you might say, "All right, uh, my users are only gonna send four thousand tokens."
- 29:38
But in reality, they might have been sending thirteen hundred over the past week that you measured. So now you can stay with statistical certainty that, hey, the majority of people that we're serving, um, during this time, these were their querying patterns, so I can rebuild an engine for that period of time.
- 29:55
What gets super interesting, this is a topic I'm very interested in, is seasonal engines. So during the day, you have different querying patterns. So you'll scale down, you'll scale up, and so you might have different engines built for different types of querying patterns based on traffic and stuff like that.
- 30:10
So hopefully that, that may have answered your question, yeah. But it's just saying, you know, looking at the bounds of what's the minimum number of tokens that came in, the max, uh, min, min out and max out, and just looking at that over the entire distribution.
- 30:25
Yes, sir. Oh, yeah, yeah, right. [laughs]
- 30:30
When it comes to those, uh, inference strategies you talked about, like Lilo and Lyso, um, how do you-- what kind of strategies do you have to manage, like, which ones are used at, like ...
- 30:39
'Cause obviously each session is going to be pretty generic. You don't know which one to use at first.
- 30:43
Correct.
- 30:43
Um, do you split those between GPUs or do you stick with one? Does it switch between-
- 30:47
So typically we'll, we'll go to ... You try to find what's one configuration that'll manage the, the plethora of types of requests that you have coming in. So we, we're typically at a, a one engine per all the different querying types, and I think you'll start seeing ...
- 31:03
I'm giving you a little bit of future ways to think about it on the DevOps side, because that's something you'll have to test, right? If I look at this querying pattern that came into my system with this engine, if I switch the engine, does it still satisfy the querying pattern?
- 31:17
And how much cost does it save? How much faster is it? So that's more of a, an engineering ex- exercise that you'll have to deploy. Sorry, I, I didn't have a- [speaking faintly]
- 31:25
Yeah, yeah. So I, I just ... I'm very interested in the seasonal side just because, okay, querying patterns will change. Um, especially when agents come, it'll just be ...
- 31:34
That's gonna get super interesting when agents are just throwing stuff. Yes, sir?
- 31:38
Um, so a question about how you measure quality of attention. Um, is it, is it correct intuition to think that attention is a fundamentally scarce resource, in the sense of it's about paying attention to one thing at the expense of other contexts?
- 31:52
So then how do ... Or like, can you scale attention mechanisms infinitely the way we can
- 31:57
context lengths?
- 31:57
Yeah. So, so what people do in order to scale the attention mechanism is, here's another interesting fact that, um, why folks don't train huge context models, 'cause it's actually ...
- 32:08
Now you've seen the bigger my, uh, prompt, the more memory I need. So imagine what that does to a huge, I don't know, 10,000, 100,000 GPU deployment. It might make it a million GPUs just to do that context length.
- 32:21
So people will train to a small context length and then interpolate in that value to, to give you that length of context length, and then you're sort of bound to what attention mechanism you were using.
- 32:33
Design there, there's things like flash attention that will just do everything in the L1 cache really, really fast. So it depends on the speed of some of the different ...
- 32:41
It also depends on the GPU as well. So that's why, um, if you look at Blackwell that was announced by, by Jensen, they literally have connected, I think, 72 different GPUs on one NVLink.
- 32:53
So NVLink connects GPUs together. That's how we can move data insanely fast. So now we've connected like 72 GPUs on one. That's, that's just to show you, um, like mixture of experts trying to compute attention across all of these different things.
- 33:06
But that's a, actually a really good question. Yep.
- 33:08
So, so we're not gonna be bottlenecked then by attention mechanisms as-
- 33:11
No, I, I don't necessarily think so. And like the entire industry is, you know, going after that problem. That's why everybody wants to maybe see something other than attention and, ah, you know, there's so much excitement there.
- 33:22
Yeah. Okay. Unfortunately, I have to call time on that. Yeah, yeah. Sorry about that. But that's been fantastic. Thank you, Mo. Great. Um- [clapping] [outro music]