Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

Talk transcript

1. 0:00

   [upbeat music]

2. 0:12

   It's very difficult to teach, um, extremely technical material in about twenty minutes. Initially, I had planned for at least a forty-five minute session, so I left some reading material for you at the end.

3. 0:22

   And, um, all of the resources, you can download slides and everything, so feel free to take screenshots or not. And so I work at NVIDIA, I'm a solutions architect, so I work primarily with retail clients, and it's my job to essentially work with those clients, understand sort of what their main challenges are.

4. 0:38

   This is data processing, computer vision, um, across all of the different use cases. And then now I'm focused on LLM inference. So my hope today is that, um, you get a better intuition of exactly what's happening with this particular workload and how you go about, uh, to some degree, sizing things, choosing different GPUs, et cetera, et cetera,

5. 0:57

   and more importantly, controlling the cost of a deployment. 'Cause that's oftentimes the, the thing that's going to really prevent you from taking this to, taking this to any meaningful scale, is that overall cost of a deployment.

6. 1:09

   Most folks that I've seen are doing some kind of hybrid. So you choose a big box API, you have some set of queries that go there, in addition to you have some set of queries that go to some open source, you know, hosted model or some fine-tuned model that you have internally.

7. 1:25

   So just reference, um, if you go to build.nvidia.com or ai.nvidia.com, everyone can get, uh, a thousand inference requests for free. So I typically recommend this to folks who are benchmarking, um, different types of open source models.

8. 1:38

   We have all of those models hosted. It's optimized if you're teaching a course, um, and you are trying to evaluate all of the different LLMs that are out there for your business.

9. 1:47

   There are also multimodal LLMs, um, speech LLMs. Every model that NVIDIA accelerates will be available there for you, and that's sort of a path to you, um, to either go optimize them yourselves or to work with us.

10. 2:00

    Um, you'll see things about, uh, NVIDIA Inference Microservice, and all of those things that you can take to enterprise. So we have sometimes the, um, I call it the rocky road, and then there's smooth roads.

11. 2:11

    Whatever path you want to take, we're here to support you.

12. 2:15

    Uh, in terms of agenda, very simple. I, I want you to understand the LLM inference workload, and then we'll move to, um, how you go about measuring a production deployment and some of the things you need to be watching.

13. 2:26

    It's a little more than, um, let's say, you know, the total time to generation, and really understanding what's happening on the GPUs as you sort of scale out. Even if you have a single GPU, I think it's very important for you to just have that intuition.

14. 2:38

    And then lastly, I'll show you some software that you can use, um, some open source packages that you can use, and then point to, uh, some paid offerings. Okay, we're gonna get into the LLM inference workload itself.

15. 2:50

    So the first part is, is really understanding what happens when you send a prompt onto the GPU. So I have this example here, I'm saying, "Okay, write me a presentation so I sound smart, I come to the AI engineer conference, and you guys are maybe gonna like the talk."

16. 3:04

    And essentially what I'm gonna do is I'm gonna put that on the GPU. So the moment that I send that prompt on the GPU, it stays on the GPU.

17. 3:10

    So think about that, and then from there, I'm going to generate one token at a time. So I'm generating the tokens, "LLM inference is hard," and I put the timestamps T1 through T4.

18. 3:21

    So in every single deployment, no matter how fast anyone claims they're, they're doing things, um, it's typically one token that's generated at a time. That's very important to understand.

19. 3:31

    The next thing is that in order for an LLM to give you a coherent answer, just like how you speak, you have to remember every single thing that you said before, and that you'll understand the mechanism of, um, how that-- how LLMs are able to do that.

20. 3:45

    So that's why I'm putting, "LLM inference is," in red and putting that back onto the GPU. So every token that I generate gets locked onto the GPU, and then you'll actually see what that looks like, um, in terms of vectors.

21. 3:57

    How many of you have heard of KV cache before? [coughs]

22. 4:01

    Okay, some of you. Um, typically I don't see maybe many leaders hear about this thing called KV cache. KV cache is this thing that really drives, to some degree, the cost.

23. 4:11

    Uh, so whe-whether or not you use some big box API or, um, you're using a single GPU, it's all the same sort of mechanisms, the same algorithm that everyone is trying to solve.

24. 4:22

    So in terms of steps, here the, the... I, I like to, as I said, sharpen your intuition. So the first thing, if we move from the left, my first job is to convert these text, whatever text that you send, we're gonna focus on LLM inference, into some words that the model understands.

25. 4:37

    So the model will have its own vocabulary, and it's my job to translate that. I'll give you the technical, uh, terms coming up after that. And the first thing that happens is I do some initial prompt processing.

26. 4:48

    So I have to compute the attention mechanism on the entire prompt. I repeat that. I have to compute the attention mechanism on the entire prompt per user. So if I have a million people hitting my service, and a million people send ten thousand tokens, that's a million times ten thousand attention mechanisms that I need to compute, also

27. 5:08

    while generating tokens for other people. So that's-- it's good for you to appreciate sort of that complexity that's happening. And once I finish processing that prompt, then I'm going to start generating one token at a time, and that typically happens very fast.

28. 5:21

    And then from there, every token that gets generated, that's in the LLM's vocabulary, I need to now de- tokenize that back into your language. So here's the technical terms that you'll see when you read the literature or you read, uh, super technical documents.

29. 5:36

    First, there's tokenization. Each model will have its own tokenizer. And, um, the thing to think about when you think of tokenizers, when they did pre-training, they had... they downloaded the Internet and some, right?

30. 5:49

    And they cleaned it up, et cetera, et cetera. So a tokenizer, and as you start thinking of the complexity across languages, coding languages, uh, regions, et cetera, et cetera, they try to get what is the minimal set- Of character groups that can represent this entire training data set efficiently, because it's really all about efficiency.

31. 6:09

    So for instance, the LLaMA tokenizer has one hundred and twenty thousand tokens, right? And I'll, I'll talk a bit more about that. So here's what it actually kinda looks like on a GPU.

32. 6:18

    So I tokenize, the LLM understands it. I go into this thing called prefill. Prefill is a stage where you compute the attention mechanism, and many people are doing advancements with attention mechanisms.

33. 6:29

    Um, I'll talk a bit more about that. So there are tons of different schemes. People leverage, um, all of the different types of memory hierarchies in GPUs to really accelerate this type of workload.

34. 6:39

    And then I start generating tokens one at a time. The red and the green just signify, "Hey, I'm storing those tokens on the GPU." The green is the latest one that I sent out.

35. 6:49

    So hopefully that makes sort of intuitive sense. All right, so the other thing I want you to visualize, um, I, I think it's nice to visualize what, what is the actual data that sits on the GPU.

36. 7:02

    So first, a, a token is approximately four characters. That's a nice way for you to think about it. Um, so from here I have, uh, two vectors. So the first vector is just showing token one through token V.

37. 7:14

    V is the total number of tokens that I have in my tokenizer. And the second vector below is just I have some numeric index. I don't want to keep using the token to reference itself.

38. 7:24

    I just use, use the number as a lookup. So my job when a prompt comes in is to convert that text into those token, what I'm gonna call token IDs.

39. 7:34

    Okay? So I, I have, "Make me sound smart," and "Make me sound smarter." You see two vector sets of tokens, and the key thing I want you to walk away with from, from that distinction is that an LLM's token is not a human word.

40. 7:48

    Sometimes it is, sometimes it's not. It's typically some subpart of words. You'll see weird symbols when you look at tokenizers from different models, um, but you want that first framing.

41. 7:57

    So now we have text, we hit to a vector, right? So from there, each one of those LLM tokens had a corresponding embedding vector. So embedding vector is everything.

42. 8:08

    We embed, uh, videos, we embed images, we embed text tokens. Think of it as a representation that, uh, an LLM can use to compare things and do math on.

43. 8:19

    So that's why we always want to com-- excuse me, convert into some vector representation, right? 'Cause some vector representation is just some high-dimensional coordinate space, and we're just rearranging objects.

44. 8:30

    That's to some degree what you're doing. Okay, so from those token IDs, I went to the actual, um, embedding vectors themselves. So if you look, "Make me sound smart," now becomes a matrix, right?

45. 8:42

    "Make me sound smarter," becomes a matrix with an extra column. So in reality, what you're doing every time you submit a prompt, I don't care what LLM you submit it to, who you submit it to, this is what you're doing, right?

46. 8:55

    You are converting your text, now images as well, they get converted to some, uh, either tokens or something like that. That'll be another interesting talk to do diffusion models, et cetera, et cetera.

47. 9:05

    But you're really putting this large matrix on the GPU. So why that-- The next question you should ask is, "Okay, um, why are GPUs good for this workload?" 'Cause they process matrices really, really fast.

48. 9:17

    So that's sort of the, the advantage and the thing hopefully that, that makes a lot more sense to you. Now, the next thing I want to talk about is how the LLM is going to process these tokens.

49. 9:27

    And now keep in mind if any-- Well, I'm not even gonna ask you to raise your hand. I'm a hundred percent sure each of you has used an LLM.

50. 9:33

    If you have not, I'm not sure what's happening. [laughs]

51. 9:38

    Uh, the other one is the attention mechanism. I, I truly think, um, it's one of the things that you should understand. If we ever drift away from it, that's fine, but the fundamentals of that mechanism and, and seeing sort of the innovations around that I think can help anyone, any business leader, et cetera, et cetera, just because

52. 9:56

    you are able to speak a different kind of language, um, in this generative future. So as you think of the attention mechanism, the intuition that you should have is just that mechanism of relating tokens.

53. 10:07

    How do I distinguish in a sentence what is important, right? And then for the next token that's gonna be generated, hey, what tokens that I said before were really important for me to make a next good decision for that next token?

54. 10:20

    So that's the intuition, and now we're gonna... we won't necessarily touch too much on the math, but I want you to see sort of what's happening on the GPU.

55. 10:27

    So once again, the prompt comes in. I'm just gonna do a short one, "Make me sound smart." I'm gonna generate this token called LLM, right? Uh, we saw these same matrices that I said before, so remember my text now turns into a matrix heading onto the GPU.

56. 10:42

    And, and the main thing I want you to understand or visualize here is actually how

57. 10:48

    an LLM memory works. So now when you're speaking, you've recorded everything that I've said for the last, uh, ten minutes in your brain. Somewhere it's, it's stored. So now you're gonna see how the LLM is storing what it is that you just said.

58. 11:02

    So from there, um, a lot of folks will hear about these query, key, and value matrices. This is what the actual model weights look like. So when you look at a model weights file, if you go on Hugging Face, there's typically a JSON file that will show you all of the different pieces of model files, and you'll

59. 11:17

    see this thing called Q, K, and V. So I have these model weights. So now I've went from text to a matrix. I'm going to matrix multiply against the weights of the models.

60. 11:27

    So now I get these three output models. So think of these weight matrices that I showed here, think as, um, when you're doing a projection, what you're doing is you're taking, um, some coordinates and you're putting it into a different space.

61. 11:41

    That's really what you're doing when you do vector matrix math. So now when I do this matrix multiplication, this query, key, and value matrix, so if you look at different tutorials on attention, you'll see these things pop up a lot.

62. 11:52

    So hopefully that'll help you to read it a lot more. This is now the LLM's interpretation of each of those tokens that you sent in, right? And now the job is how do I now take, uh, these query, key, and value matrices and sort of interpret it- To try to generate the next best token.

63. 12:09

    And this is just happening constantly over and over every single token that's happening. But the key thing I want you to walk away on this slide is where I drew the key and the value.

64. 12:19

    Right? When people talk about KV cache optimization, every LLM performance engineer is just literally trying to make that thing as fast and small as possible. And I'll, I'll-- that'll make a little more sense as to what that does to your cost.

65. 12:32

    But ultimately, these key and value matrices, this is like your LLM's memory. So it'll make a little more sense coming up. I know I didn't show a ton of the math.

66. 12:39

    I show some tutorials afterwards, so you can go read more about that. Um, my intention here is for you to visualize key and value. So every time you see a prompt, I just want you thinking, "Crap, key and value is on my, on my GPU."

67. 12:51

    Okay? The next. So here's the real value of the KV cache. So remember we said that whenever I generate a token, I'm going to push it back into the GPU, right?

68. 13:04

    So every token I generate, it goes back into the GPU, and then I have to compute an attention mechanism. So this is what's happening. This new token I generated, LLM, I get its vector representation, as you see in blue, but now I, I do that vector matrix math now.

69. 13:19

    So before I did matrix-matrix math, that's my first prompt first comes in. I generated my first token. Now I'm doing vector matrix math. You know, people will batch this across all requests, but I'm just showing you a single request so you can see it.

70. 13:34

    Now, the value of the KV cache is if I were to-- if I didn't have the KV cache, I would have to reprocess all of that work I did on the prompt that I did before.

71. 13:46

    So this is the benefit of your KV cache, is to now I'm just gonna compute attention on this newest token. How does this new token relate to everything that I said before?

72. 13:54

    That's the thing that's really happening, uh, intuitively. So if I have this KV cache, my generation is gonna be fast. Okay? And it, it's really up to the, the what's called a batch manager on the GPU to make sure that I'm just pushing out as many tokens as possible.

73. 14:11

    Okay. So if you look at, uh, an LLM, these groups of three matrices are called an attention head. There are more matrices than that, but these are the main ones.

74. 14:20

    Um, Llama has thirty-two attention heads, so I just kinda want you to appreciate what an LLM really looks like, right? So I have thirty-two sets of these matrices. I have thirty-two of those KV caches happening at the same time, and now I have to combine all of that to then generate the next token.

75. 14:35

    So there's, like, an incredible amount of work that happens in a very short space of time to give you a coherent token. Okay. A good mental model for you to keep in your head, I'm gonna speed up a little bit, is to, um, if you see the number of parameters, multiply that by two, and that is your

76. 14:52

    FP16 gigabyte memory on the GPU. So if you have, let's say, an L4, I think is twenty gigs, um, and I have a Llama 8B, that's automatically sixteen gigs FP16.

77. 15:03

    So I only have four gigs left for my KV cache. So on the GPU, it's either the model weights or tokens. That's it. There's nothing else on the GPU.

78. 15:12

    And I have, I have a thing to read on that. This is a really good blog. It shows you all of the different, um, optimizations that you can do and...

79. 15:20

    Okay, now let's talk about measuring. So if you ever see this thing called ISL or... Do I have it here? Oh, sorry. ISL or OSL, that's input sequence length, output sequence length.

80. 15:32

    So now I want you to see what some advanced monitoring might look like. If any of you are DevOps folks, these are things that you wanna record. The first thing that we measure is time to first token.

81. 15:41

    So how long does it take me to generate, um, the-- process the prompt and then generate my first token? And that's typically a measurement of how good your attention mechanism processing is.

82. 15:52

    That's really what you're trying to suss out. So that's time to first token. Inter-token latencies, so after I've generated my first token, every single token after that, I'm looking at those individual spaces.

83. 16:03

    So everything that's going to happen there, uh, think about when the system is under load. I have, you know, a thousand requests coming into my system. I'm generating a thousand sets of different tokens.

84. 16:13

    And the more memory I occupy, typically that slows down processing. So if you start to see drift in this metric, then I'll show you some plots that you can look at, and then time to total generation, how long did it take me to initially get the prompt, fully finish the answer, right?

85. 16:27

    Super intuitive. Like I said, ISL, OSL, that's all that means when you see them on the plots coming up. Okay, uh, this is a very important paradigm for you to understand in your mind.

86. 16:38

    Uh, so I, I worked with a lot of folks on, you know, com-- maybe, uh, REXX deployments or deployments of other types of models. So on the GPU, if you're only deploying one model on a GPU outside of LLM inference, uh, in my opinion, I think you're wasting the GPU.

87. 16:52

    You can put multiple models on the GPU to actually increase your throughput. That's why it was really created. Um, so this is a slide... Excuse me. This figure is just showing I can have multiple models.

88. 17:03

    I have some space for data, and that's how I increase my throughput per unit hardware. However, on the LLM inference side, it's very different. I have one model. You know, folks can fit multiple models on a GPU.

89. 17:14

    That's cool, but that's not a real production use case. You'll typically have a single model. The remaining space that you have is all for KV cache and generating all those tokens.

90. 17:23

    So I just put four different requests, and I, I just kinda want you to see the boxes that are happening. Okay? Uh, I would say this is the most important slide in the entire presentation because this is the thing that will determine both your cost and performance.

91. 17:36

    So there are four different querying patterns that happen, and this is something that you must measure in your deployment, 'cause oftentimes you might read benchmarks, and it'll just say, all right, they'll cherry-pick one or two of these.

92. 17:48

    But in reality, in your production system, you might have several of these different patterns that are occurring. So let's take a look at the first one, long input, short output.

93. 17:57

    So a long input means it's gonna take me technically longer to compute the attention mechanism. So my prefill stage will be longer. I occupy more memory from my prompt.

94. 18:07

    Does that make sense intuitively? Hopefully, it, it's grabbing you. But then on the generation side, I don't generate much tokens, so there's not much Those tokens are not picking up a lot of memory, and they will tend to finish fast.

95. 18:19

    So the second, the second one, or the maybe the most costly use case is, um, so I have clients that will message me and say, "Hey, my data scientists are putting two bigger prompts on my GPUs, so now they're killing my deployment."

96. 18:31

    'Cause if everyone went and put the maximum context length, I can only fit so many requests on the GPU. So that's something for you to think about. You'll have to manage that internally with your deployments.

97. 18:42

    So that's why I'm putting, you know, okay, the GPU is really full 'cause a long text, uh, excuse me, long input, a long output. The next one, short-long, you know, your time to first token be really fast.

98. 18:51

    I don't have much to compute the attention mechanism on, but hey, I'm generating a ton of tokens. That's really, really fast. So hopefully, as you start measuring these types of different query patterns, um, you'll see different results.

99. 19:03

    I just put, you know, what a random sampling set might actually look like on the GPU, 'cause not everyone will send the same length of input and output. Um, so that will-- it'll be good for you to just sort of visualize and track these statistics.

100. 19:16

     More importantly, why we're doing that, uh, internally, I'm, I'm gonna steal the time here, Peter. Uh, more importantly, why we're doing that or why we're tracking these things is that the whole goal is to build...

101. 19:28

     I have a big model. My goal is to shrink it as much as I can, but to keep it as accurate as possible. So the more that I shrink, the faster it runs, the more GPU memory I have for what?

102. 19:39

     Tokens. All right? So that's how you really try to improve your cost. This is why I'm, I'm sort of proposing to you to build inference engines. So what I-- all I'm showing here is a 2D histogram of input sequence length versus output sequence length.

103. 19:53

     'Cause the question that you'll have to answer is, "Hey, how long are my actual prompts?" Someone might say, "Okay, here's the max prompt length that you can ingest, or the max prompt you can out, you know, excuse me, get on the output."

104. 20:05

     And all of the big box model providers have to estimate this when they go into costing or providing a service to you, right? 'Cause they have to host all of that machinery under the hood now that you understand what's happening.

105. 20:17

     So we use this stat- to statistically determine what is the max input sequence length and the max output sequence length across all of my users. And this would give you a really good indication of, um, how you can size your engines.

106. 20:31

     We use that to actually build more optimized engines. In addition, um, it'll just give you good view as to maybe, uh, what you call it, scaling out and things like that.

107. 20:41

     The next one is time to first token analysis. Remember, time to first token is measuring my performance of the attention mechanism under load. So someone might show attention mechanism at one query.

108. 20:51

     Woohoo. Show me attention mechanism under load. When this thing is fully maxed out twenty-four/seven, that's when you really need to start, um, measuring these types of things. So this is something you can look at.

109. 21:02

     These are sort of experimental plots. Um, there's a package called GenAI Perf that will be released open source. It, it's out already. I have a link to it there.

110. 21:10

     This is where you can-- it'll generate these plots for you. But I, I'm just showing you what the engineers are looking at internally to measure the performance of, of the compute platform.

111. 21:19

     Next, time to completion analysis. How long did it take me to go from start to finish across every single request? Naturally, the wider that box plot, you have to intuitively ask, "What's happening?

112. 21:31

     Why did this person's prompt take longer than another?" So you can investigate either batching issues, scheduling issues, different things like that. I'll, I'll take questions in the end. Um, or I have to move really fast.

113. 21:41

     Sorry there, Peter. Okay, I'm gonna speed up here. Uh, token-to-token latency. Peter, how much time I got?

114. 21:47

     Oh, you're fine. Yeah, we'll definitely have time for questions.

115. 21:49

     Okay, cool. I, I'm gonna steal... I'm definitely over, so sorry. I just wanna-- I realize I may have gone a little too fast, so forgive me for that.

116. 21:56

     No, you've got five minutes.

117. 21:57

     Cool. All right. Time, uh, token-to-token latency. So that is, I'm generating tokens, so I'm looking at that spacing versus token position. So the longer a sequence gets, remember, my memory grows, so typically that means our system is under more load, it has more throttling that might happen under high load of requests.

118. 22:17

     So if I see a large variation in token-to-token latency as the sequence gets longer when I'm generating, that means I'm not very performant, right? So we look at that to see, I try to make sure that that's constant.

119. 22:29

     No matter how much tokens I'm generating, that means I'm, I'm really proficient. Okay? Uh, last one would be time to first token versus number of input tokens. So time to first token, remember, is computing the attention mechanism, okay, versus number of input tokens.

120. 22:46

     So if I have a bigger prompt, my attentions will take longer. But I-- if that plot goes up like, uh, from your perspective, it goes up like this in terms of sequence length, that's not really good performance.

121. 22:57

     We really look at that slope, and we try to get that slope almost, you know, as, as low as possible. So if you send me this long sequence, I can get that thing done really fast.

122. 23:07

     Okay? Okay. Uh, in terms of software, uh, you'll see this thing called TRT-LLM, uh, Triton as an open source inference server. So you can deploy models on CPU, on GPU, computer vision, Raxas, Python, PyTorch, TensorFlow.

123. 23:21

     It's, uh, it'll host all of the different types of models, so there's one way that your deployment team deploys. All the data scientists are happy 'cause they don't have to do conversion.

124. 23:29

     You're happy as a deployment person 'cause you don't have to manage a TorchServe versus TSServe and, uh, Flask, and all of it is done through one. It's written in C++, blazingly fast.

125. 23:39

     And then the other thing you'll see in NVIDIA, you'll see a lot more coming out of NVIDIA's NVIDIA Inference Microservice, 'cause building these engines, getting them deployed, optimizer scale is not easy.

126. 23:49

     So we've sort of made that easy for you as an enterprise offering, but you guys can try it out, uh, for free. Okay. So TRT-LLM, let me just give you a high level.

127. 23:57

     Lots of stuff on this slide, but the main thing I want you to walk away with, um, uh, is this is the model compilation package for LLMs on NVIDIA GPUs.

128. 24:05

     This, if you want to get best performance from NVIDIA GPUs, please make sure you use TRT-LLM. Um, and naturally, uh, once we're investing more in NIM, you'll see some more things come out.

129. 24:15

     So you'll see performances on A100 and H100 really focus on FP8 GPUs. So FP8 will be Hopper and Ada Lovelace. Okay, so FP8, I'll, I'll talk a bit more about that, what the advantage there is.

130. 24:28

     But mainly is if I go from FP16, FP8 is this, half my memory, almost the same accuracy. And so we measured the accuracy, and we publish the accuracy. So now I have this much more space for tokens, but more importantly, this model is that much faster.

131. 24:44

     Okay, so I want you to understand where the sort of industry is going. This is why Hopper, the world ate Hopper for breakfast and lunch and dinner because of FP8.

132. 24:52

     It gave folks that cost, um, benefit to do this thing a lot faster, okay? Um, in-flight batching, it just means I don't have to wait for all the requests to finish to start a new request.

133. 25:04

     The moment your request finishes, I can in-inject a new request while others are going. Okay? Tons of features here. I, I put the features. Um, so some ones to, to focus on are quantized KV cache, so I can actually represent my KV cache in different, uh, excuse me, precision.

134. 25:21

     So that means I'm actively shrinking that memory, making it more performant. Um, you have paged KV cache, that's just you managing, uh, your GPUs a lot better in terms of all of that memory.

135. 25:30

     So there are tons of things you can do. Tensor parallelism, the thing to remember about tensor parallelism, if you want to increase latency, you use tensor parallelism. Split the model up across multiple GPUs.

136. 25:41

     That's typically done within a node. I repeat that. That's typically done within a node. You don't like to do tensor parallelism across a node. You'll see pipeline parallelism go across a node.

137. 25:53

     Pipeline parallelism is more sequential, so I process this chunk. So in a multi-node model, like huge models, this box will finish and pass off to the next box. But most folks will typically just work...

138. 26:05

     Most models will work within a single node, like, um, NVIDIA's 340B model that they just released. It was designed to do inference on a single H100 node, but that's an FP8.

139. 26:16

     Okay. So those are some of the things. In terms of models that you have access to, um, we optimize those models, and we give you a lot of scripts where you can go do that on your own, or you can sort of take our software, um, and take an easy path.

140. 26:28

     You-- Either way, we support you. So here are some of the models that are there. All of the Llamas, Mistral, Mixtrals, we work with all those teams behind the scenes.

141. 26:36

     So typically, before any foundation model comes out, we, we work with those teams to, to get them deployed. Okay, what does it mean for TensorRT? Uh, so you might have seen TensorRT before, which was a deep learning compilation package for NVIDIA GPUs.

142. 26:50

     Lots of folks in computer vision, et cetera, et cetera, have used that. We took the best practices from there and added all of the extra things that need to happen in the LLM inference loop.

143. 27:00

     So that's what TRT-LLM is really about. Um, so mainly focus on LLM inference. Uh, here's a good visual. An engine that's built to a specific GPU cannot be moved to another GPU.

144. 27:12

     So you always have to compile to that GPU. That's why it's that performant, 'cause we really leverage, um, all of the, the actual hardware on that system to rewrite the algorithm, rewrite that model to that specific piece of hardware.

145. 27:25

     Okay? Um, TRT-LLM on Triton. So TRT-LLM will give me an inference engine. I need something to host that inference engine and accept requests, batching, et cetera, et cetera. So we have Triton.

146. 27:37

     Triton works very simply. It's literally a folder where you specify it, it works on tensor in and tensor out. So it'll tell you what are my inputs coming in and out, and then it'll basically understand how to interpret that file, or you can host any other different models.

147. 27:52

     That's one-- That's a thing I do a lot with folks. Just two more slides. This is where the future of inference is going. So a lot of folks do FP16 inference today.

148. 28:01

     Um, a lot of folks are moving towards FP8 just because, hey, I know half the model size, almost twice the speed, more space for tokens. It just makes more sense from a cost perspective.

149. 28:13

     That's why folks like that. And then, uh, you saw Blackwell was announced. That's the major innovation. I get FP4. So that's where things are really gonna get interesting. I'll end with, um, NVIDIA Inference Microservice.

150. 28:25

     So we've made this thing really easy. We've gone and actually found the best configurations for all of these models on each piece of GPU, and we're slowly rolling out, um, all of the models 'cause it, you know, it'll just take some time to optimize the world, [chuckles] essentially.

151. 28:38

     And yeah, you can use this to download all the slides. I put papers, um, tons of other things you, for you to read. So yeah, hopefully your, your intuition has sharpened. [laughs] [audience applauding]

152. 28:53

     Shall we just conclude with the... 'Cause there was someone had a question.

153. 28:56

     Sure. Yeah, I think-

154. 28:57

     Where was the question?

155. 28:58

     Yeah.

156. 28:59

     Oh, hang on. I'm gonna come over and, uh, point my mic at you.

157. 29:06

     Thank you. Uh, so hi. Um, sorry. My question is actually on the heat map that you shared.

158. 29:12

     Yeah, yeah.

159. 29:12

     Do you mind, um, walking through the heat map and how to interpret it? 'Cause it was a little small, couldn't really-

160. 29:17

     Yeah, sorry about that. Yeah. So the heat map-

161. 29:18

     Thanks

162. 29:19

     ... all I'm looking at is, um, so when you go to build an engine, you build an engine to the max input sequence length and the max output sequence length.

163. 29:27

     So we actually change how that matrix math is happening under the hood based on those settings. So you might say, "All right, uh, my users are only gonna send four thousand tokens."

164. 29:38

     But in reality, they might have been sending thirteen hundred over the past week that you measured. So now you can stay with statistical certainty that, hey, the majority of people that we're serving, um, during this time, these were their querying patterns, so I can rebuild an engine for that period of time.

165. 29:55

     What gets super interesting, this is a topic I'm very interested in, is seasonal engines. So during the day, you have different querying patterns. So you'll scale down, you'll scale up, and so you might have different engines built for different types of querying patterns based on traffic and stuff like that.

166. 30:10

     So hopefully that, that may have answered your question, yeah. But it's just saying, you know, looking at the bounds of what's the minimum number of tokens that came in, the max, uh, min, min out and max out, and just looking at that over the entire distribution.

167. 30:25

     Yes, sir. Oh, yeah, yeah, right. [laughs]

168. 30:30

     When it comes to those, uh, inference strategies you talked about, like Lilo and Lyso, um, how do you-- what kind of strategies do you have to manage, like, which ones are used at, like ...

169. 30:39

     'Cause obviously each session is going to be pretty generic. You don't know which one to use at first.

170. 30:43

     Correct.

171. 30:43

     Um, do you split those between GPUs or do you stick with one? Does it switch between-

172. 30:47

     So typically we'll, we'll go to ... You try to find what's one configuration that'll manage the, the plethora of types of requests that you have coming in. So we, we're typically at a, a one engine per all the different querying types, and I think you'll start seeing ...

173. 31:03

     I'm giving you a little bit of future ways to think about it on the DevOps side, because that's something you'll have to test, right? If I look at this querying pattern that came into my system with this engine, if I switch the engine, does it still satisfy the querying pattern?

174. 31:17

     And how much cost does it save? How much faster is it? So that's more of a, an engineering ex- exercise that you'll have to deploy. Sorry, I, I didn't have a- [speaking faintly]

175. 31:25

     Yeah, yeah. So I, I just ... I'm very interested in the seasonal side just because, okay, querying patterns will change. Um, especially when agents come, it'll just be ...

176. 31:34

     That's gonna get super interesting when agents are just throwing stuff. Yes, sir?

177. 31:38

     Um, so a question about how you measure quality of attention. Um, is it, is it correct intuition to think that attention is a fundamentally scarce resource, in the sense of it's about paying attention to one thing at the expense of other contexts?

178. 31:52

     So then how do ... Or like, can you scale attention mechanisms infinitely the way we can

179. 31:57

     context lengths?

180. 31:57

     Yeah. So, so what people do in order to scale the attention mechanism is, here's another interesting fact that, um, why folks don't train huge context models, 'cause it's actually ...

181. 32:08

     Now you've seen the bigger my, uh, prompt, the more memory I need. So imagine what that does to a huge, I don't know, 10,000, 100,000 GPU deployment. It might make it a million GPUs just to do that context length.

182. 32:21

     So people will train to a small context length and then interpolate in that value to, to give you that length of context length, and then you're sort of bound to what attention mechanism you were using.

183. 32:33

     Design there, there's things like flash attention that will just do everything in the L1 cache really, really fast. So it depends on the speed of some of the different ...

184. 32:41

     It also depends on the GPU as well. So that's why, um, if you look at Blackwell that was announced by, by Jensen, they literally have connected, I think, 72 different GPUs on one NVLink.

185. 32:53

     So NVLink connects GPUs together. That's how we can move data insanely fast. So now we've connected like 72 GPUs on one. That's, that's just to show you, um, like mixture of experts trying to compute attention across all of these different things.

186. 33:06

     But that's a, actually a really good question. Yep.

187. 33:08

     So, so we're not gonna be bottlenecked then by attention mechanisms as-

188. 33:11

     No, I, I don't necessarily think so. And like the entire industry is, you know, going after that problem. That's why everybody wants to maybe see something other than attention and, ah, you know, there's so much excitement there.

189. 33:22

     Yeah. Okay. Unfortunately, I have to call time on that. Yeah, yeah. Sorry about that. But that's been fantastic. Thank you, Mo. Great. Um- [clapping] [outro music]