From model weights to API endpoint with TensorRT-LLM

Talk transcript

1. 0:00

   [upbeat music] I am here with, uh, with Pankaj Gupta.

2. 0:16

   He's the co-founder of Baseten. Um, actually, so today, uh, I was, I was checking Slack, and in the random Slack channel, one of the people in the company, uh, was saying like, "Hey, you know, I heard someone say...

3. 0:28

   call someone cracked. What does, what does cracked mean?" Um, and you know, those, uh, those of you who are, you know, uh, Gen Z like me or, or, or know someone like that is like laughing right now because cracked just means a, uh, an exceptional engineer.

4. 0:42

   And so, uh, Pankaj is the most cracked software engineer I've ever had the, uh, pleasure of working with. Um, he's, uh, from San Francisco. His favorite model is Llama 3 8B.

5. 0:51

   Um, we're gonna be working with a smaller version of that today. Um, I'm Philip. I do developer relations here at Baseten. I've w- been here for about two and a half years.

6. 1:00

   And, uh, I am based in Chicago, but I'm very happy to be here in San Francisco with you all today. And, uh, my favorite model is Playground 2. It's a te- it's a text-to-image model that's kind of like SDXL, but it's trained on Midjourney images.

7. 1:15

   You're gonna see a ton of Playground 2 images in the slideshow today. So, what are we doing here today? Uh, what is our agenda? So we're going to cover what is TensorRT-LLM and why use it.

8. 1:28

   Model selection and TensorRT-LLM support because it supports a lot of stuff, but not everything. Um, we're going to talk about building a TensorRT engine, configuring a TensorRT engine automatically, benchmarking it, so you can know if you actually did something worthwhile, and then, uh, deploying it to production.

9. 1:46

   And, you know, as much as I love the sound of my own voice and I wanna just stand here and grasp this microphone for two hours and say things, this is not just gonna be Philip reads off a slideshow.

10. 1:56

    Um, we're gonna do tons of coding, debugging, live Q&A. So the way this, uh, se- presentation's kind of broken up is we've got some, some sections, we've got some live coding.

11. 2:07

    Um, it's gonna be, you know, a very interactive workshop. I'm gonna be taking questions all the time, so please don't hesitate, uh, to let us know if anything's confusing.

12. 2:15

    We really want everyone to come away from this with a strong working understanding of how you can actually use this technology in production.

13. 2:24

    If I may-

14. 2:24

    So let's get started. Yes.

15. 2:25

    If I may interject for a second and ask, uh, raise of hands, how many of you know about TensorRT?

16. 2:33

    Great. How about-

17. 2:33

    Oh, this is so exciting. I'm so glad that we get to teach you all this today.

18. 2:37

    And how about TensorRT-LLM? Okay, a few. So we'll, we'll cover the basics. I, I think, uh, I- I'm, I'm pretty sure that you'll get a sense of what it is.

19. 2:47

    If you know PyTorch, this shouldn't be too hard.

20. 2:49

    And if you don't know PyTorch, like me, it's still not that hard.

21. 2:55

    So, uh, we're gonna start with the story of TensorRT-LLM. What, who, why? You know, once upon a time, there was a company called NVIDIA. Um, and they, uh, they, they noticed that there were these things called large language models, um, that people love running.

22. 3:11

    But what do you want when you want a large language model? You want a lot of tokens per second, you want a really short time to first token, and you want high throughput.

23. 3:20

    You know, GPUs are expensive, so you wanna get the maximum value out of your GPU. And TensorRT and TensorRT-LLM are technologies that are gonna help you do that. So if we get into it here, what is TensorRT?

24. 3:32

    Here's one of my, uh, Playground 2 images. Very proud of these. Uh, if the words on the slides are dumb, uh, just look at the images, uh, because I worked hard on those.

25. 3:42

    Um, anyway, so TensorRT is a SDK for high-performance deep learning inference on NVIDIA GPUs. Basically, what that means is it's just a great set of tools for building high-performance models.

26. 3:55

    Um, it's a, you know, toolkit that supports both C plu- C++ and Python. Um, our interface today is going to be entirely Python, so if, like me, you skipped the class that teaches C++, don't worry, you're covered.

27. 4:08

    I know Pankaj read C++ textbooks for fun, uh, but, but, but I do not, so we're gonna do it in Python today. Um, and so how does this work?

28. 4:16

    You know, do, do you wanna, do you wanna kinda jump in here and, and-

29. 4:19

    Yeah

30. 4:19

    ... and talk about this a little bit? Because, you know, it's, it's a, it's a really cool process how you go from a neural network to, to an engine.

31. 4:27

    Yeah. Um, yeah, exactly. So ultimately, what are machine learning models? They are the graphs. They are computation graphs. You flow data through them, you transform them. And ultimately, whatever executes a model does that.

32. 4:40

    They execute a graph. Your neural network is a graph. TensorRT works on a graph representation. You take your model, and you express that using an API, uh, that graph in TensorRT, and then TensorRT is able to take that graph, discover patterns, optimize it, and then be able to execute it.

33. 4:59

    That's what TensorRT is ultimately. When you write a P- a PyTorch model, you're ultimately creating a graph. It's, it's graph all over, right? There is data flowing through this graph.

34. 5:08

    Uh, and that's what it is. TensorRT additionally provides a plug-in mechanism. So it says that, you know what? I know this graph. I can do a lot of, lot of stuff, but I can't do very fancy things like flash attention.

35. 5:21

    It's just too complex. I can't infer automatically from this graph that this is even possible. Like, I'm not a research scientist. So it gives a plug-in mechanism using which you can inspect the graph and say that, "Okay, I recognize this thing, and I can do it better than you, TensorRT, so I'm gonna do it through this plug-in."

36. 5:38

    And that is what TensorRT-LLM does. It has a bunch of plug-ins for optimizing this graph execution for large language models. So for example, for attention, for flash attention, it has its own plug-in.

37. 5:50

    When it says that, "Okay, now we are in TensorRT-LLM land. Take this graph and let me execute it using my optimized CUDA kernels." And that's what ultimately TensorRT-LLM is.

38. 6:01

    Um, uh, a very, very optimized way of executing these graphs using GPU resources to Uh, not only to get more efficiency, uh, better, [clears throat] better cost for your money, but also better latency, better time to first token, all the things that we care about when we are running these models.

39. 6:19

    Uh, in addition to that, it provides, uh, a few more things like, uh, when you're executing a model, you're not just executing a request at a time, we're executing a bunch of requests at a time.

40. 6:28

    And in-flight batching is a key optimization that is very, very key. Like, uh, in this day and age, if you're executing a large language model, you have to have in-flight batching.

41. 6:38

    There's just no way. Like, it's like a 10X or 20X improvement. Like, uh, and you have to have that. And TensorRT-LLM provides that. TensorRT wouldn't. TensorRT is a graph executor.

42. 6:47

    It doesn't know about that. But TensorRT-LLM has an engine that does that. It also has a language to express graph, just like PyTorch, and it requires that there is a conversion, but it makes it pretty easy to do that conversion, and there are tons of examples in the repo.

43. 7:01

    Exactly. So TensorRT is this great sort of engine builder, and then TensorRT-LLM is a mechanism on top of that that's going to give us a ton of plugins and a ton of optimization specifically for large language models.

44. 7:16

    So TensorRT-LLM, like Pankaj said, defines the set of plugins for your LLMs. If you want to, you know, compute attention, do LoAs, Medusa, other fine tunes, um, and it g-- lets you define optimization profiles.

45. 7:31

    So when you're running an, a large language model, you generally have a batch of requests that you're running at the same time. You also have a input sequence and an output sequence.

46. 7:41

    And this input sequence could be really long. You know, maybe you're summarizing a book. It could be really short. Maybe you're just doing some LLM chat like, "Hi, how are you?

47. 7:51

    I'm Fred from the bank." Um, depending on what your input sequence and output sequence lengths are, you're gonna wanna build a different engine that is going to be optimized for that, um, to process that number of tokens.

48. 8:05

    Um, so yeah. So TensorRT-LLM is this toolbox for taking TensorRT and building large language model engines in TensorRT.

49. 8:16

    Yeah. I wanna say just one thing at this point. Like, why I care about input and output sizes. Like, how does TensorRT-LLM optimize for that? It actually has specific kernels for different sizes of inputs, different sizes of matrices, and it's optimized for that level, and it sometimes becomes a pain when I'm compiling TensorRT-LLM.

50. 8:33

    It takes hours because it optimizes for so many sizes. But it al- also means that giving it that size guidance is useful. It can use better kernels to do things faster, and that's why.

51. 8:44

    Uh, a lot of the models you'll run, you don't have to care about it, but there is always a trade-off. Here, it does care about that, and you can benefit using that trade-off.

52. 8:54

    Yeah. And TensorRT-LLM is a great tool for a number of reasons. Um, it's, you know, it's, it's, it's got those, those built-in optimized kernels for different sequence lengths. Um, and that level of detail is really across the entire tool.

53. 9:10

    And what that means is that with TensorRT-LLM, you can get some of the highest performance possible on GPUs for a wide range of models, and it's really a production-ready s- uh, system.

54. 9:20

    We are using TensorRT-LLM today, uh, for tons of different client projects, and it's, uh, you know, running in production powering things. Um, TensorRT-LLM has support for a ton of different GPUs.

55. 9:32

    Um, basically anything like Volta or newer. The Volta support's kind of experimental. Um, but yeah, like your A10s, your A100s, H100s, all that stuff's supported. Um, and yeah, and TensorRT-LLM, it's, uh, developed by NVIDIA.

56. 9:45

    Um, so, you know, they know their graphics cards better than anyone. Um, so we, we just kinda use it to run models quickly on that. Um, that said, everything does come with a trade-off.

57. 9:55

    Um, is, is anyone from NVIDIA here in the room? It's okay, you don't have to raise... Okay. Uh, so I, I, I'm gonna be nice. Um, uh, no, we, we really are big fans of this technology, but it g- does come with trade-offs.

58. 10:07

    You know, some of the underlying stuff is not fully open source. Um, so sometimes if you're diving super deep, you need to, uh, go get more information without just, like, looking at the source code.

59. 10:17

    Um, and it does sometimes have a pretty steep learning curve, uh, when you're building these optimizations. So that's what we're here to help flatten out for you guys today.

60. 10:26

    Hopefully, we're still friends. Uh, what makes it hard? So the... there's a couple things that make building with TensorRT-LLM really hard. And when we enumerate the things that make it hard, that's how we know what we need to do to make it easy.

61. 10:39

    So the number one thing in my mind that makes it hard to build a general model, uh, uh, to, to optimize a model with TRT-LLM is you need a ton of specific information about the production environment you're going to run in.

62. 10:51

    All right. So I, I do a lot of sales enablement trainings, and I love a good metaphor. So I'm gonna, I'm gonna walk you guys through a metaphor here.

63. 11:01

    Uh, apologies if metaphors aren't your thing. So imagine you go into a clothing store, and it only sells one size of shirt. You know, it's just, like, a medium.

64. 11:10

    Um, you know, for some people that's gonna fit great. For some people it's gonna be too small. For some people it's gonna be too big.

65. 11:16

    And on the other hand, you can go to, like, a tailor, I don't know, in, like, Italy or something. And you go there, and they've got, you know, some, some super fancy guy with a, with a, you know, cool mustache and stuff, and he, you know, he measures you, like, every single detail and then builds a suit

66. 11:33

    exactly for you that's perfect for your body measurements, like a made-to-measure suit. So optimizing a model is kind of like making that suit. You know? Everything has to be measured for exactly the use case that you're building for.

67. 11:47

    And so when people come in and expect that they can just walk in and grab off the shelf a model that's going to work perfectly for their use case, that's like expecting you're gonna go into a store and buy a piece of clothing that fits you just as well as that custom-made, made-to-measure suit from the tailor.

68. 12:03

    So in... you know, to relate that more concretely to TensorRT-LLM You need information. You need, like we talked about, you need to understand the sequence lengths that you're gonna be working at, the batch sizes that you want to run at.

69. 12:15

    You also need to know ahead of time what GPUs you're gonna be using in production. These engines that we're building are not portable. They are built for a specific GPU.

70. 12:24

    You build them. So if you build it on an A10, you run it on an A10. If you build it on an H100, you run it on an H100.

71. 12:32

    You want to switch to H100 MIG, okay, you build it again for H100 MIG. So you need to know all of this information about your production environment. Um, and then also, as we'll talk about kind of toward the end, there are some infrastructure challenges as well.

72. 12:46

    These engines that we're gonna build are quite large. So if you're, for example, doing auto-scaling, you have to deal with slow cold starts, f- uh, you know, work, work around the size of the engines, otherwise your cold starts are gonna be slow.

73. 12:58

    Um, and overall, also, just model optimization means we're living on the cutting edge of new research. You know, I'm... When I'm, when I'm writing blog posts about this stuff, I'm oftentimes looking at papers that have been published in the last six months.

74. 13:11

    So, you know, just combining all these new approaches and tools, there, there can be some rough edges, but the performance gains are worth it. So-

75. 13:21

    Yeah, I want to-

76. 13:21

    Yeah. Oh, please, go ahead.

77. 13:22

    I want to add one thing is that there are modes in TensorRT-LLM where you can build for a certain... On a certain GPU, and it will run on other GPUs.

78. 13:31

    But then it, it's not optimized for other GPUs. So why would you do that? We never do that. We always build it for the GPU. But there is that option.

79. 13:38

    Exactly. That would be like if I went into that fancy tailor shop, got a made-to-measure suit, and then was like, "Hey, Pankaj, happy birthday. I got you a new suit."

80. 13:45

    Uh, that's the, that's what it would be like. So, you know, what, what makes TensorRT-LLM worth it? Well, it's, it's the performance. So these numbers are from a Mistral 7B that we ran on Artificial Analysis, which is a third-party benchmarking site.

81. 13:59

    Um, and we were able to get, with TensorRT-LLM and a few other optimizations as well on top of it, 216 tokens per second, uh, perceived tokens per second, and 180 milliseconds time to first token.

82. 14:11

    So, um, unless any of you are maybe, like, some super high-quality athletes, like a UFC fighter or something, your reaction time is probably about 200 milliseconds. So, you know, 180 millisecond time to first token, counting network latency, by the way, counting the round trip time to the server, is great because that to a user feels instant, once

83. 14:31

    you're under 200 milliseconds.

84. 14:32

    And actually, most of it is network latency. The time on the GPU is less than 50 milliseconds.

85. 14:38

    Less than 50 milliseconds. So, uh, we've got another one of these green slides here. I like to talk really fast, so these slides I put in this presentation to give us all a chance to take a breath and ask any questions.

86. 14:49

    So, you know, we're gonna cover a lot more technical detail moving forward, but if there's anything kind of foundational that you're struggling with, like what's TensorRT? What's TensorRT-LLM? Anything I can explain more clearly, I would love to hear about it.

87. 15:03

    Going once. Going twice. It's okay, we're all friends here. You can raise your hand. All right. Well, it sounds like I'm amazing at my job. I explained everything perfectly, and we get to move on to the next section.

88. 15:18

    So what models can you use with TensorRT-LLM? Lots of them. Uh, there's a list of, like, 50 foundation models in the TensorRT-LLM documentation that you can use, and you can also use, you know, fine-tunes of those models, anything you've built on top of them.

89. 15:34

    Um, it supports open source large vision models, so if you're, you know, building your own GPT-4o, um, you can do that with TensorRT-LLM. Um, and it also supports models like Whisper.

90. 15:45

    And then TensorRT itself, you can do anything with TensorRT. So any model, custom, open source, fine-tuned, you can run it with TensorRT. Um, but TensorRT-LLM is what we're focusing on today because it's a much more convenient way of building these models.

91. 16:00

    And, uh, you know, on this list of models that it supports, there's, there's one that maybe stands out. Does a- does anyone know, like, what model kind of doesn't belong in, in this list of supported models?

92. 16:12

    Like, what, what, what up here isn't an LLM? Whisper. Exactly. Why, why is, why is Whisper on here? Well, TensorRT-LLM, it's, it's called dash LLM, um, but it really is a little more flexible than that because you can run, you know, a lot of different autoregressive transformers models with it, like Whisper.

93. 16:33

    So if anyone doesn't know what Whisper is, it is a audio transcription model. You give it a, you know, MP3 file with someone talking, it gives you back a transcript of what they said.

94. 16:44

    It's one of our, it's one of our favorite models to work with. We've spent a ton of time optimizing Whisper, building pipelines for it, and all that sort of stuff.

95. 16:51

    And what's really cool about Whisper is structurally, like, it's basically an LLM. You know, that, that, that's a massively reductive statement for me to make, but it's a autoregressive transformers model.

96. 17:03

    It has the same bottlenecks in terms of influence performance. So even though this is not an L- not an LLM, it's an audio transcription model, we're actually still able to optimize it with TensorRT-LLM because, uh, because of its architecture.

97. 17:18

    Let me say one more thing. [chuckles]

98. 17:19

    Of course.

99. 17:20

    So the, the whole, uh, re- I think the recent ML revolution started with the transformers paper, Attention Is All You Need, and that describes an encoder-decoder architecture. And in a way, Whisper is machine translation.

100. 17:34

     That paper was about machine translation. You're translating audio text, uh, audio into text, right? And it's basically that. It's an encoder-decoder model exactly like the transformer architecture. And Tensor, TensorRT-LLM is about that.

101. 17:47

     It's about that transformer architecture. So it actually matches pretty well.

102. 17:52

     Exactly. So, um, moving on, um, I wanna run through a few things, uh, just, just some, some things in terms of what TensorRT-LLM supports. So I assume it's gonna support Blackwell when that comes out, like 99.999% certain.

103. 18:07

     Um, but anyway, in terms of what we have today, we've got Hopper, so the H100s, the L4s, RTX 4090s. If anyone has a super sweet gaming desktop at home, number one, I'm jealous.

104. 18:18

     Number two, you can run TensorRT-LLM on that. Um, Ampere GPUs, Turing GPUs, uh, V100s are, you know, somewhat supported. Um- And what's cool about, what's cool about TensorRT and hardware support is that, like it works better with newer GPUs.

105. 18:37

     When you move from an A100 to an H100 and you're using TensorRT or TensorRT-LLM, you're not just getting the sort of like linear increase in performance that you'd expect from, you know, "Oh, I've got more flops now.

106. 18:50

     I've got more to- gigabytes per second of GPU bandwidth." You're actually getting more of a performance gain going from one GPU to the next, uh, than you would expect off raw stats alone.

107. 19:02

     And that's because, um, you know, H100s, for example, have all these great architectural features, and TensorRT, because it actually optimizes the model by compiling, uh, to CUDA instructions, is able to take advantage of those architectural features, not just kind of run the model, um, you know, raw.

108. 19:21

     And so for that, you know, that's why we do a lot with H100 MIGs. Th- this, this bullet point here is a whole different 45-minute talk that I tried to pitch to, uh, do here.

109. 19:31

     But basically, you know, H100 MIGs are especially good for TensorRT-LLM, um, if you're trying to run smaller models like a 7B, you know, Llama 8B, for example, uh, because you don't need the massive amount of VRAM, but you get the, um, increased performance from the architectural features.

110. 19:49

     Um, and you know, just my own speculation down here that I'm sure whatever the next generation is, is going to have even more architectural features for TensorRT to take advantage of.

111. 19:59

     And so, you know, adopting it now is a good move, uh, you know, looking to the future. Here we've got a graph showing, you know, with SDXL... Now, this is TensorRT, not TensorRT-LLM, but the underlying technology is the same.

112. 20:12

     Um, you know, wh- when you're working on an A10G, we were looking at, you know, maybe like a 25% to 30% increase in throughput for SDXL, and, uh, with an H100, it's a 70%.

113. 20:23

     And that's not, you know, just because the H100 is bigger. It's 70% more on an H100 with TensorRT L- TensorRT versus an H100 without. So yeah, great. Uh, yeah, please go ahead. [laughs]

114. 20:36

     One thing I want to add here is that, uh, H100 supports FP8 and A100 does not. FP8 is a game changer. I think it's very easy to understate that fact.

115. 20:46

     FP8 is really, really good. Post-training quantization, you don't need to train anything. Post-training quantization, it takes, like five minutes, and the results are so close. We've done perplexity tests on it.

116. 20:56

     Whenever you quantize, you have to check the accuracy. Uh, and we've done that. It's ha- it's hard to tell. And FP8 is about 40% better in most scenarios. So if you're using, uh, MIG H100, if you can, then it's, it can be way better if you use FP8.

117. 21:12

     And FP8 is also supported, um, by Lovelace, so that's going to be your L4 GPUs, um, which are also a great option for, for FP8. So yeah, prec- a bunch of different precisions are supported.

118. 21:24

     Again, FP8 is kind of the highlight. FP4, uh, could be coming. And, um, you know, traditionally, though, we're gonna run in FP16, um, which is sort of like a, a, uh, full precision, um, and...

119. 21:38

     Oh, sorry, half precision. FP32 is technically full precision.

120. 21:41

     Nobody does FP32.

121. 21:42

     What?

122. 21:42

     But nobody does FP32.

123. 21:43

     Yeah, yeah. So, so for, for inference, generally, you start at FP16. By the way, FP16 means a 16-bit floating-point number. Um, and from there, you know, you can quantize to INT8, FP8 if, you know, you want your model to run faster, if you want to run on fewer or smaller GPUs.

124. 22:02

     Um, we'll, we'll, we'll cover quantization in a bit more detail later on in the actual workshop. I don't wanna spend too long on the slides here. I know you guys wanna get your laptops out and start coding.

125. 22:12

     Um, so the other thing just to talk about is, like I said, TensorRT-LLM, it's a, you know, it, it, it's a set of optimizations that you can, you know, build into your TensorRT, into your TensorRT engines.

126. 22:26

     Um, so some of the features that are supported, again, each one of these could be its own talk here. Uh, but we've got quantization, LoRA swapping, speculative decoding, and Medusa heads, which is where you basically like fine-tune additional heads onto your model, and then at each forward pass, you're generating like four tokens instead of one token.

127. 22:46

     Great for when you're, you know, when you have memory bandwidth restrictions. Um, yeah, in-flight batching, like you mentioned, page attention. There's just a ton of different optimizations supported by TensorRT-LLM for you to dive into once you have the basic engine built.

128. 23:01

     So, um, we're about to switch into more of like a live coding workshop segment. So if there's any of this sort of groundwork information that didn't make sense or any more details that you want on anything, let us know.

129. 23:13

     We'll cover it now. Um, otherwise, it's, it's about to be laptop time.

130. 23:18

     Looks like everyone wants laptop time. So, uh, yes, please go ahead.

131. 23:23

     Can you do like some high-level comparison with like vLLM and so on?

132. 23:28

     Yeah. Do you want, do you wanna, do you wanna handle that one, like a high-level comparison to vLLM?

133. 23:32

     I can do very high-level comparison. Uh, first of all, um, I respect both tools. vLLM is great. TensorRT-LLM is also great. We found in our comparisons that, uh, for most of the scenarios we compared, we found TensorRT-LLM to be better.

134. 23:47

     Um, there, there are a few things there. One thing is that whenever a new GPU lands or a new technique lands, that tends to work better on TensorRT-LLM. vLLM, it takes a bit of time for it to catch up, for the kernels to be optimized.

135. 24:00

     Uh, TensorRT-LLM is generally ahead of that. For example, when H100 landed, TensorRT-LLM was, uh, was very, very fast out of the box because they've been working for, uh, on it for a long time.

136. 24:10

     Um, second thing is that TensorRT-LLM is, uh, optimized from bottom to the top. These, uh, CUDA kernels at the very bottom are very, very well optimized. On top of that, there is the in-flight batching engine all written in C++, and I've, I'm seeing that code.

137. 24:24

     It's very, very optimized C++ code with your STD moves and whatnot. And on top of that is Triton, which is a web server, again, written in C++. So the whole thing is very, very optimized.

138. 24:37

     Whereas, uh, in some other frameworks, uh, they also try to optimize, uh, in the sense like, you know, Java versus C++, where Java is like, you know, we optimize everything that matters.

139. 24:46

     But there are always cases where it might not be as good. With TensorRT-LLM is that let's optimize every single thing. So it generally tends to perform better in our experience.

140. 24:55

     That said, vLLM is a great, great product. We use it a lot as well. For example, LoRA swapping, it became available in vLLM first. So we used that, uh, there for a while.

141. 25:05

     What we found is that when something lands in TensorRT-LLM, and it's usually after a delay, it works like, like bonkers. It just works like so well that, uh, performance is just amazing.

142. 25:18

     So when something is working very stably in TensorRT-LLM, we tend to use that. But, uh, vLLM and other frameworks, they provide a lot of flexibility, which is great. Uh, we love all the products.

143. 25:29

     And like, does that smaller performance gains kind of like how [audio glitch] the speaker learning curve? Because-

144. 25:37

     Yeah

145. 25:38

     ... it's like much easier to use.

146. 25:40

     Yeah, yeah. I think, I think we should, uh, definitely question that. That is a clear trade-off. If you're working with two GPUs, for example, two ATNGs, it's not worth it probably.

147. 25:50

     But if you're spending hundreds of thousands of dollars a year, uh, it's, it's your call. But if you're spending,

148. 25:57

     uh, many hundreds of thousands of dollars a year, it can be material. Like your profit margin might be twenty percent. This twenty percent or fifty percent improvement might make the all the difference that you need.

149. 26:09

     So it depends upon the use case.

150. 26:10

     Yeah. So do you have like any intuition like how you build a cluster you need like compared to before like TensorRT? Because you said it's really important like just like on a few V2, V2, so vLLM is probably easier, faster than the results.

151. 26:29

     It could be, yeah. I think if you're working with one or two GPUs, ATNGs or T4s... I mean, I'm an ex-- I'm not an expert in that, but, uh, it probably doesn't matter which framework you use, whichever works best for you.

152. 26:41

     But if you end up using A100s or H100s, you should definitely look into TensorRT-LLM.

153. 26:46

     Thank you.

154. 26:47

     And, uh, regarding the learning curve, stick around a little bit. We're going to, uh, flatten it out a lot for you because we've built some great tooling on top of TensorRT-LLM that's gonna make it just as easy to use.

155. 26:58

     Little, little, little marketing spin right there. Um, so yeah. So we're going to, um, be doing a engine building live coding exercise. Um, and, and Pankaj is gonna lead us through that.

156. 27:11

     I'm just gonna kinda roam around. So if, if people have, you know, questions, need help kind of on a one-on-one basis, I'll be able to help, help out with that doing this portion of the workshop.

157. 27:21

     Great. So, uh, we have-

158. 27:27

     Yeah. Let, let's just like go through a couple... There, there's, there's-

159. 27:29

     Yeah

160. 27:29

     ... a little bit of s-- uh, a little bit of setup material, right? Or...

161. 27:33

     Yes.

162. 27:33

     Yeah.

163. 27:34

     Um, you wanna run through that-

164. 27:36

     Okay, sure. I'll, I'll run through the-

165. 27:37

     So let me just, uh-

166. 27:38

     I'll run through the setup material. Um, and then, uh...

167. 27:43

     Yeah. So, um, anyway, what we're gonna do, um, to, to, to be clear, is we're gonna build an engine for a model called TinyLlama 1.1B. I wanna be really clear about this.

168. 27:53

     TensorRT-LLM is a production-ready technology that works great with big models on big GPUs. Uh, that takes time to run. The dev loop can be a little bit slow. And we only have a two-hour workshop here, and, uh, you know, I don't want us to all just be sitting there watching a model build.

169. 28:09

     It's basically as fun as watching paint dry or watching grass grow. Um, so we're going to be using this super tiny one point one billion parameter model. We're gonna be using forty-nineties and ATNGs, um, just to kinda keep the dev loop fast.

170. 28:23

     But this stuff does scale. So, um, at this point, we're gonna walk you through the manual process of doing, doing it all from scratch. You're gonna procure and configure a GPU.

171. 28:32

     You're going to install dependencies for TensorRT-LLM, configure the engine, run the engine build job, and, uh, test the results. And we, we should be able to get through this in, in about half an hour or maybe a little less because these, uh, these models are quite small.

172. 28:47

     Um, and there's a few important settings that we're gonna look at when building the engine. We're gonna look at the quantization. Again, the post-training quantization, like we talked about.

173. 28:56

     We're gonna be on ATENs. Or sorry, no. First, we're gonna be on forty-nineties, so we will actually have access to FP8 so that you can test that out. Um, we're going to look at sequence shapes and batch sizes, how to set that, and we're gonna look at tensor parallelism.

174. 29:09

     You wanna give them a quick preview on tensor parallelism?

175. 29:12

     Well, yeah. Tensor parallelism is, uh, is very important in certain scenarios. I wish it were more useful, but it is critical in many scenarios. So what is tensor parallelism?

176. 29:23

     Ultimately, machine learning, running these GPUs is about matrix multiplications. We take this model architecture, whatever it is. It ultimately boils down to matrices that we multiply. A lot of the wrangling is around that.

177. 29:35

     How do we shove these all batches into matrices? So ultimately, it is matrix multiplication, right? What you can do is you can split these matrices, and you can multiply them separately on different GPUs and then combine the results.

178. 29:47

     And that's what tensor parallelism is. It's one of the tensor pa-- uh, of parallelism techniques. Uh, there are many techniques. Uh, it's one of the most commonly used ones because you need that.

179. 29:58

     Uh, why do you need tensor parallelism versus other parallelisms like pipeline parallelism? Um, is that it saves on latency. You can do things in parallel. You can use two GPUs at the same time for doing something, even though there is some overhead of crosstalk between them.

180. 30:15

     With pipeline parallelism, you take the model architecture, and you can divide these layers into separate things. So your thing goes through one GPU, like half the layers, and then half the layers on a second GPU.

181. 30:26

     But you're not saving on latency. It still has to go through each layer, and it's going sequentially. And that's why pipeline parallelism is not very popular for inference. It is still popular for training.

182. 30:37

     There are scenarios, uh, and there's a lot of theory about that. But for, for inference, I don't think I've ever seen it used, and nobody pays much attention to optimizing it because of this thing that tensor parallelism is just better.

183. 30:49

     There's also expert-level parallelism. If your model has mixture of experts, then you can parallelize those experts, and that tends to be very advanced, and Llama doesn't have mi-mixture of experts, so it's a esoteric thing that we haven't covered here.

184. 31:02

     Tensor parallelism is pretty helpful and useful. Uh, one downside is that your throughput is not as great. If you can fit something in a bigger GPU, that's generally better.

185. 31:11

     But there are bigger, uh, models like Llama 70B, they just can't fit on one GPU, so you have to use tensor parallelism.

186. 31:19

     Awesome. So for everyone to get started, um, we made a GitHub repository for you all to work off of in this, in this, uh, workshop. So you can scan the QR code, it'll take you right there.

187. 31:32

     Otherwise, uh, you know, this is, this is not too long to type out. Um, so I'm just gonna leave this up on screen for 30 seconds. Everyone can pull it up.

188. 31:42

     Um, you're going to want to, you know, fork and clone this, uh, this repository, um, to your, to your local development environment. Um, we're just, you know, we're just using Python, Python 3.10, Python 3.11, um, 3.9.

189. 31:56

     Um, so yeah, just like however, however your, your normal way of writing code is, um, this, this should be compatible. Um, there, there isn't a lot of... What? [in background]

190. 32:07

     Uh, no. So, uh, I... Yeah, to be clear, in, in this, in this repository, um, you're gonna find instructions, and we're gonna walk through all this. Um, we're gonna be using entirely remote GPUs.

191. 32:18

     Um, so you know, I personally have an H100 under my podium right here that I'm gonna be using. No, I'm just kidding. I don't. Um, but, uh, yeah, yeah.

192. 32:26

     So we're, we're just, uh, we just all have laptops here, so we're gonna be using cloud GPUs.

193. 32:31

     Yeah. Actually, if you want to follow along, you might need a RunPod account.

194. 32:35

     Yeah, yeah. We'll, we'll, we'll, we'll talk them through the, uh, the, the, the setup steps though. Um, does, does anyone want me to leave this information on the screen any longer?

195. 32:44

     Going once, going twice. Okay. If, if you, for whatever reason, lose the repository, just let me know. I'll, I'll get it back for you. Uh, yes. Okay. So this, this slide means we are transitioning to live coding.

196. 32:57

     So yes, let's go, uh... Let's go over to, um, the,

197. 33:03

     yeah, the, the, the live coding experience.

198. 33:06

     So I'm, I'm basically gonna follow this repository. All the instructions are here. And, uh, I'm gonna follow exactly what is here, so you can see, uh, how to follow along.

199. 33:18

     And if you, uh, if you ever get lost or need help, just raise your hand and I'll come over and catch you up, like one-on-one.

200. 33:24

     Yeah, yeah. I'm gonna go, uh, really slow. I'm gonna actually do all these steps here. I know it takes time, but, uh, you know, there's a lot of information here.

201. 33:31

     It's easy to, uh, lose track of thing and get lost. So if you, if you feel lost, like ask, and we'll break. I wanna make sure. This is not a long process, a 10-minute process.

202. 33:40

     We can take it slow for everybody here.

203. 33:43

     So first thing is that, uh, we'll, we'll do it like really, really from scratch. We're gonna spin up a new, uh, container on RunPod with a GPU to run our setup in.

204. 33:54

     So if you-

205. 33:54

     Let's, let's give everyone a minute to sort of make an account, right?

206. 33:57

     Okay. Okay. Yeah, yeah. Please, please. Um, if you wanna follow along, please go on RunPod and create an account. This should cost like less than $5 overall.

207. 34:05

     Yeah. So, um, so yeah. So if you wanna make an account, um, there's instructions in the, um, 01 folder. Uh, yeah, this README, so TensorRT in the, in the first folder in the README, there's instructions and a video walkthrough.

208. 34:19

     Um, the minimum... We're, we're, we're not affiliated with RunPod in any way. Uh, they just have 4090s, and we wanted you guys to use 4090s today. There is a minimum credit buy of $10.

209. 34:30

     If for whatever reason you can't use a company card or get it reimbursed or whatever, and you want your $10 back, uh, send me an email after the, um, after the conference, and I will reimburse you myself.

210. 34:41

     So, uh, you know, I, I just... I wanna... I-- It's really important to me that we are giving you the GPUs, uh, to, to run this on. So yeah.

211. 34:49

     So if, if you, if you, uh, if you need the $10 back, I will, I will get it for you. Um, so yeah, just, uh, just follow this. It's like a one-and-a-half-minute video.

212. 34:59

     Um, and, uh, yeah, that, that, that'll get the account set up. Maybe we could even like play the video.

213. 35:04

     Yeah, that'd be better.

214. 35:06

     Um, so yeah. And then this is how you, you know, deploy, um, once we, once we get set up.

215. 35:15

     So in, in two minutes, we will come back together and, you know, we resume the, resume the talk. So yeah, make that account real quick. Get checked out. Get those $10 of credits at the SH key.

216. 35:26

     If you, if you have difficulty create account, just let us know on RunPod.

217. 35:28

     Yep. Yeah. And, and also, if you, if you do happen to run into any issues getting the RunPod account set up, like it's not working or whatever, um, I do have one that I can share individually with people.

218. 35:41

     It would just take a really long time for me to go type in ev- my username and password into every single computer in this room.

219. 35:53

     I feel like maybe I should have brought some, some jokes to tell while everyone sets their account up. Uh, all right. Uh, how about this one? Why did the LLM cross the road?

220. 36:05

     Ignore previous instructions and cross the road. All right. I'll, I'll, I'll workshop that. I'll get some better jokes for you guys before, before the next setup.

221. 36:17

     How are we doing on, on the account setup? Um, a-anyone need a little more time? All right. Great. No problem. No problem. Sorry, I, I don't wanna rush you.

222. 36:27

     Just checking in. Yeah. And then once, once everyone has the account, we'll set up the GPU together, 'cause there's a few things you need to con-configure.

223. 36:42

     I don't know what's going on, but like Andrew's credit card is getting declined. Cool. Uh- [laughing] My card being declined. Oh, really? I tried like four of us. It got denied for the same reason.

224. 36:52

     I even gave you a separate one one time.

225. 36:56

     It's, it's not taking anyone's credit card? Yeah.

226. 37:02

     Great. Uh, can... Do, does someone... Does... Here, can, can I, can I, can I- Yeah ...

227. 37:08

     know someone who runs at... who works at OnePod and would have their, uh, their phone number? He's calling them? Okay, awesome. [laughing] All right. We're, we're getting, we're getting in touch with customer support.

228. 37:20

     It's probably a stripe block function. Oh, yeah, it could be, it could be that. Yeah, stripe block function. Okay, so as a backup, um- Yeah.

229. 37:29

     I think you should get off the WiFi.

230. 37:31

     What's that?

231. 37:35

     You get off the WiFi.

232. 37:35

     Okay.

233. 37:35

     You put it on, you put it on your phone WiFi.

234. 37:37

     Okay, so the recommendation here is go off of the conference WiFi, put your computer on your phone hotspot, and try it again, um, because that, that worked, uh, you know, may- maybe, maybe coming from a different IP address will, will help.

235. 37:54

     How about we do is I run through this, and we can do it again once everybody has their account?

236. 37:59

     Yeah, that sounds good. So what we're gonna do in the interest of time here, um, is we're going to, uh, Pankaj is gonna run through end to end, um, sort of the, the, the demo as we, as we get the stuff set up and everyone's credit cards get unblocked.

237. 38:15

     Um- [laughing] Yeah, you know, who, who would have thought, you know? We, we were, we were talking this big game. "Oh, TensorRT-LLM, it's so hard, it's so technical. There's gonna be so many bugs."

238. 38:27

     And then it's the payment processing. [laughing] So, uh, yeah, you know that, that's, that's, that's live demos for you. So anyway, yeah, go-

239. 38:34

     Yeah

240. 38:34

     ... go ahead and, uh, work through it. Um, and then we'll do it kind of again, uh, together once everyone has their account.

241. 38:40

     All right. Yeah, let me run through this. I'll follow the... all the steps. Uh, I, I already have an account at OnePod, so let me spin up a new instance here.

242. 38:50

     And, uh, I'm picking up the forty-ninety here,

243. 38:55

     which this one. And it has high availability, so that should be fine. And, uh, I'm gonna edit this template and get more space here. This doesn't cost anything extra.

244. 39:06

     Yeah, we need more space, uh, because the engine... Uh, everything that we're installing, um, and the engine we're building takes up a lot of gigabytes, so otherwise we'll run out of memory.

245. 39:15

     Yeah. Better to be safer. Yeah. Even though these engines are small, engines in general can be very, very big. They can be hundreds of gigs. And I'm gonna pick on-demand because I'm doing this demo.

246. 39:24

     I don't want the instance to go away. But feel free to use Spot for your use case. I'm gonna do that.

247. 39:32

     Can you use, uh, container disk or volume disk?

248. 39:34

     The, the-

249. 39:34

     You want to set the, uh, the container-

250. 39:37

     Container one

251. 39:38

     ... um, container disk to two hundred gigabytes so that you have enough room to install everything.

252. 39:44

     And then I'm gonna deploy Spot. It's gonna be a bit slow, but, you know, feel free to ask any questions. And, uh, I feel like this way we'll take it slow, but we'll make sure everything is understood by everybody.

253. 39:58

     So what, what's happening now, now is that this, uh, pod is spinning up. Uh, one thing to note here is that it has a specific image of, uh, Torch with a specific CUDA version.

254. 40:10

     It's very important that the node has GPUs. And the first thing we're gonna, we're gonna do is that once this pod comes up, we're gonna check that it has everything related to GPUs running fine.

255. 40:25

     So this is starting up. Now I'm gonna connect. It gives you nothing sensitive here. It uses your SSH keys, but, uh, the names are not sensitive. So I'm gonna just do that, log into that box.

256. 40:42

     If you want, scroll up. It's a little bigger.

257. 40:45

     Oh, sorry?

258. 40:46

     Can you just make, make it bigger?

259. 40:48

     Oh, okay. Sorry. Yeah, yeah, this is much smaller. [clears throat]

260. 40:53

     I think the pod is still spinning up, so it's taking a bit of time.

261. 40:59

     Hmm. Okay. Now it's up.

262. 41:16

     So to test that everything is set up properly-

263. 41:18

     Just, uh-

264. 41:19

     Yeah

265. 41:19

     ... is it, is it possible to scroll it to the top of the screen so we can see?

266. 41:22

     Oh, yeah. Okay, great. So we are on this, uh, machine that we spin up. We're gonna run nvidia-smi to make sure that the GPU is available. And this is what you should see.

267. 41:36

     Uh, one thing to note here is this portion, which shows that the GPU has, uh, more than twenty-four gigs of memory, the RTX forty-ninety has. And right now it's using one memory.

268. 41:49

     I think it does some, uh, some stuff like it's by default, so one mag is already taken.

269. 41:58

     So, [clears throat] so now we're gonna go back to workshop and then just follow these instructions.

270. 42:07

     Manual engine build. We are at this point.

271. 42:11

     Uh, and now we're gonna install TensorRT-LLM. This is gonna take a bit of time. TensorRT-LLM comes as a Python library that you just pip install, and that's all we are doing.

272. 42:20

     We are setting up the dependencies. This AP- APD update is setting up the Python environment, uh, OpenMPI and other things, and then we just install TensorRT-LLM, uh, from... not from PyPi, but from NVIDIA's own PyPi.

273. 42:34

     That's where we find the right versions. If you focus on this line. Uh, let me kick this off, then I can come back here and show you that. We are using a specific version of TensorRT-LLM.

274. 42:47

     And, uh, we need to tell it to get it from the NVIDIA PyPi using these instructions. And all these are on the GitHub repo if you wanna follow from there.

275. 43:03

     I saw a guy with a camera, so I started posing. [laughs]

276. 43:08

     Which price point did you use again? Uh, I think three ten, I think. Three ten. Three ten. Yes. Seems like MPI profile is not supported. Oh, this, this command should have instructions to install it.

277. 43:24

     I also-- I wanna check in with the room. Has anyone else had success getting RunPod up and going, uh, using your, using your phone Wi-Fi? It's working? Okay.

278. 43:33

     Okay, great. Great.

279. 43:33

     Awesome. Crisis averted. Thank you so much to, uh, to whoever from, from over there suggested the idea to begin with. Really saved the day.

280. 43:42

     Great. So we're just waiting for it to build. It takes some time. [laughs]

281. 43:47

     This is, this is the best part of the job. You know, you wait for it to build. You can go get a snack. You can go, like, change your laundry.

282. 43:53

     It's, uh, very convenient that it takes this time sometimes.

283. 43:56

     It used to be comp- compilation takes time. Now engine build takes time. [laughs]

284. 43:59

     Yeah, exactly.

285. 44:04

     I think we're very close. And I promise, like, then the fun part begins.

286. 44:11

     Are you saying that pip install isn't the fun part? I think this is pretty fun. You know, look, look, look at all this, look at all this lines. You know?

287. 44:19

     Well, this is-

288. 44:20

     This is, this is real coding right here. This guy.

289. 44:22

     And if you want pip to feel like more fun, try poetry.

290. 44:25

     Oh, that's true.

291. 44:26

     Then pip feels more like fun. [laughs]

292. 44:27

     Poetry is really fun.

293. 44:29

     Yeah. NVIDIA does publish these, uh, images on their container registry called NGC, and there are Triton registries, uh, available for these things. Maybe we should have used that rather than RunPod.

294. 44:40

     But, uh, it's all good. So, uh, now let's check that TensorRT-LLM is installed, and this will just, uh, tell us that, uh, everything is well and good. So it printed the version.

295. 44:53

     You should see that if everything's working fine.

296. 44:55

     Yeah.

297. 44:56

     And then we're gonna do the real thing. [clears throat]

298. 45:00

     Now we're gonna clone the TensorRT-LLM Git repository where a lot of those examples are, and I'll, I'll show you those examples while this, uh, this cloning happens. It shouldn't take that much long.

299. 45:11

     Maybe, uh, maybe a minute or so. But TensorRT-LLM has a lot of examples. Uh, if you go to the TensorRT-LLM repository, there are these examples folder, and there are ton of examples.

300. 45:23

     Like, uh, Ph-Philip mentioned, there are about fifty examples. And we're gonna go through the Llama example here. So if you search for Llama...

301. 45:33

     Uh, that's the one we are gonna l-look into.

302. 45:37

     And so the cloning is complete, and we go back to these instructions. And now we're gonna actually build the engine. Actually, [laughs] one more thing. Uh, how many of you know about hf.transfer?

303. 45:51

     Have you used the transformers library from Hugging Face?

304. 45:54

     So hf.transfer is a fast way of downloading and uploading your engines. It does sliced download. It takes the URL and, uh, patches them up into slices, downloads them all in parallel, and it works really, really fast.

305. 46:07

     It goes up to, like, one gig a second. So we should definitely do that, which is what I did just now.

306. 46:14

     Now we're gonna follow this step-by-step. Uh, first thing we're gonna do is download from Hugging Face. And, uh, let's see, like, how fast the Wi-Fi here is, uh, how fast this downloads.

307. 46:24

     So not bad. It's going at one gigs a second. So hf all-

308. 46:27

     It's from-

309. 46:27

     Yeah. [laughs]

310. 46:28

     It's from the, the, the Wi-Fi.

311. 46:29

     Oh, from... Oh, you're right. You're right. You're right. See? But this is, uh, this is what I call good software. Downloads at one gig a second. [clears throat]

312. 46:38

     Now we... Now first thing to build with TensorRT-LLM is that we have to convert, uh, the Hugging Face checkpoint into a checkpoint format that TensorRT-LLM works with. And, uh, checkpointing also covers tensor parallelism and quantization.

313. 46:53

     Sometimes you need a different kind of checkpoint for doing those things. So I'm gonna run this command to convert the checkpoint. And this should be pretty fast. It's just converting weights to weights.

314. 47:09

     That's pretty fast, like three seconds. Uh, [clears throat] and now we do the actual build. And I'm gonna do this basic build here, where there are a ton of options that this command takes, the trtllm build command.

315. 47:22

     Uh, in here we are just saying that, uh, take this checkpoint and build me an engine with most of the default settings.

316. 47:30

     And that should build the engine. Now it will print a lot of stuff about what it's doing, what it's finding, and how it's optimizing and all that. Uh, it won't make much sense right now, but, uh, later on this could be very useful.

317. 47:46

     So the engine was built. That was pretty fast, right? It's a small, uh, model, uh, only a billion parameters. So that was pretty fast. And now let's, uh, let's try to see how big the engine is.

318. 47:58

     I'm gonna do that. And the engine is, uh, two gigs in size. This is about how big that model is on Hugging Face. So it's, uh, the engine itself adds very little me- um, um, storage or memory.

319. 48:10

     It's maybe like, uh, hundreds of megabytes, but very tiny compared to the overall. And those weights are bundled into the engine. And what is this engine? This engine is something that the TensorRT-LLM, uh, runtime can take, and it can execute it.

320. 48:27

     Uh, you can think of it like a shared library. It's, uh, it's kinda, kind of like a binary in the standard format that the binaries are in. Uh, but it's, it's something that TensorRT-LLM can take and interpret.

321. 48:41

     Ultimately, it's a TensorRT engine because that's what TensorRT-LLM works with. It creates TensorRT, uh, engine, and then TensorRT is the one that loads it. But TensorRT-LLM gives it these plug-ins that TensorRT understands and then is able to make sense of it.

322. 48:58

     And now let's execute this. So these, these examples also come with a, come up with a, um, come with a run script that we can run, and we're gonna run that.

323. 49:07

     So what this is going to do is start up the engine and give it a very tiny request, and we should expect a response. And that's what happened here.

324. 49:17

     Our engine was launched. We gave it an input text of, "Born in northeast France, Sorrell trained as a..." And the model printed out the response beyond that, "Painter in Paris, moving-- before moving to London in 1929."

325. 49:32

     And this is a standard, uh, example that comes with TensorRT-LLM. So if you follow along these instructions, you should see that.

326. 49:42

     Any questions at this point?

327. 49:44

     Yeah.

328. 49:46

     Where did the conversion take place? Like, what is... What happens during conversion? What is it that's necessary?

329. 49:53

     Uh, the convert-- The question is what happens during the convert checkpoint? Uh, I think there, there are three things that happen, um, potentially three things. First thing is that TensorRT-LLM needs the tensors to be in a specific format to work with.

330. 50:08

     So think of it as a pre-processing. There are many ways of specifying a model. It can be on Hugging Face. It can be exported from PyTorch. It can be ONNX.

331. 50:17

     There are many, many different ways of specifying these models. So the first thing it does is that it converts that into a format that it understands. So it does, does some kind of translation into a standard structure.

332. 50:28

     Second thing is quantization. For quantization, it needs to, uh, quantize the weights. It needs to take the weights and quantize them into the quantized versions of them, and that happens at convert checkpoint two.

333. 50:43

     Uh, not necessarily though. They also have a quantize script. Some of those, uh, quantizations happen... Some types of quantizations happen in convert checkpoint, but they also have a different way of quantizing.

334. 50:54

     They call it, I think, uh, Amo. There is a library called Amo which does that, uh, and that can also be used for doing it. But, uh, I think AWQ and, uh, SmoothQuant, they happen in convert checkpoint.

335. 51:07

     And, uh, third thing is tensor parallelism. For tensor parallelism, you need to divide up weights, uh, into different categories for the different GPUs that they will run on. So it does that during convert checkpoint as well.

336. 51:20

     Thank you.

337. 51:22

     Yes.

338. 51:24

     Hi. I just have a question. Um-

339. 51:26

     Me too

340. 51:26

     ... does it all have any notion of the max output length? So for instance, if you do fifty, it'll cut off. If you do two fifty, it gives you, like, two or three.

341. 51:38

     For instance, if you do five hundred, it gives you the same as two fifty. So I, I just don't know whether or not that actually is gonna pass into the model, and then it tries to approximately give that length, uh, you know?

342. 51:50

     Yeah. So, so there-

343. 51:51

     Yeah

344. 51:51

     ... there's two places that the max output is set. Um, so the first place is when you're actually building the engine, you give it a argument for the m- expected output sequence length.

345. 52:02

     And then that's, that's more just so, like, for the optimization side, you know, so that you're selecting the correct CUDA kernels and so that you're, you know, batching everything up correctly.

346. 52:12

     And then once the engine is built, it just uses a standard... I think it's, uh, max tokens, right, is the parameter?

347. 52:18

     Yeah.

348. 52:18

     Um, and yeah, you just, you just pass max tokens, and that'll, you know, limit, um, how-

349. 52:24

     Mm-hmm

350. 52:24

     ... how long it, it runs for.

351. 52:26

     Yeah. I guess what I'm asking is, does it influence the generation before generation happens, or is it strictly just a cut-off, like, I guess [inaudible]?

352. 52:35

     Right. Like, are you asking if you s- if you make the engine with a shorter, um, a- output se-

353. 52:42

     No, not making the engine-

354. 52:42

     Oh, okay

355. 52:44

     ... just for demonstration purposes. Just a-

356. 52:45

     No, I, I... As far as I know, the ma- all the max token does is it just cuts off influence after a certain number, right?

357. 52:52

     Yeah. So the, the way, uh, I would put it is that normally, if you give it a large number of max tokens, it would emit, uh, a end-of-sequence token.

358. 53:03

     Most models have a different end-of-sequence token, and, uh, it's up to you. You can stop there. You can configure it runtime, like, "I don't want more than that." But you can also cal-- tell it, "Ignore end of sequence.

359. 53:14

     I just want the whole thing." And we do need it for performance benchmarking. For example, when we are comparing performance across different GPU types or whatnot, we want all of those tokens to be generated, so we can tell it, like, "Give me all of them."

360. 53:28

     Thank you.

361. 53:28

     Welcome.

362. 53:28

     Yes. In the back there.

363. 53:29

     I just wanted to continue, like, can you explain what happens in memory [inaudible] during build? Like, does it take [inaudible]?

364. 53:39

     Oh, great. Great question, actually. So, uh, in build checkpoint a l- a lot of stuff is happening. You're taking these weights, and you're, uh, you're generating this thing called a network in TensorRT.

365. 53:53

     TensorRT has this notion of a network, and what you need to do is populate that network with your weights and architectures. So it actually does that. It creates that network and feeds it these weights.

366. 54:06

     It also does inference during building the engine, uh, for doing optimization. So it generates... For every model type, it has a mechanism of generating sample input, and it passes that into the-- a TensorRT engine that it's generating, and then it optimizes it that way.

367. 54:23

     And as, uh, as a result, this TensorRT engine is generated in memory, which is then serialized. So all of this is happening in that.

368. 54:34

     And these, uh, there's a lot of nuance to it. Uh, if you get a chance, you can look at the source code for that TRT LLM build. I'll post references in that GitHub repo, and you can follow on.

369. 54:44

     There is l- lots of options. But let me try if I can find a help here.

370. 55:00

     Ah, maybe I press it. What's going on? Sorry. [laughs]

371. 55:23

     Oh, yeah, this is lo-lot of stuff here [laughs] which we can go through. Um,

372. 55:29

     uh, yeah, maybe I should go through some of them which are very important.

373. 55:36

     Yeah, I think a lot of stuff is important here, like the max beam width if you're using beams for, uh, generating the graph. You can generate, uh, logits, not just the output tokens.

374. 55:45

     We can also generate logits if you want to process them. Uh, there are a lot of optimizations that you can use, like you can... There's a optimization called fused MLP.

375. 55:53

     There is contrast chunking. There is, there is a lot of stuff, and I think you should play around with those, uh, at your time. LoRA is very good to play around with.

376. 56:02

     Uh, yeah. I'll try to leave some, uh, some more examples, uh, in the GitHub repo to try. Okay. So let me go to the next one. Just, uh, uh, one more thing I want to do is, uh, FP8 quantization.

377. 56:14

     Uh, RTX 4090 is, is actually an amazing GPU. It's pretty cheap, but supports FP8, so we're gonna do an FP8 engine build now.

378. 56:24

     So in this case, uh, like I said, like some of these optimizations, the, these, uh, quantizations are not in convert checkpoint, but quantize.py, which uses a library called Amo in NVIDIA.

379. 56:35

     So I'm gonna run that now. And, uh, yeah. Let me spend some time here. We're, we're saying is we're telling it that the quantization format is FP8, but also note that we are saying KV cache dtype is FP8.

380. 56:51

     So FP8 quantization actually can happen at two levels. You can weight quantize to FP8, but you can also quantize the KV cache with FP8. And doing both is very critical because, uh, these GPUs, they...

381. 57:07

     You might have heard of things called tensor cores, right? Tensor cores are very, very, very important because they can do

382. 57:14

     quantized calculations very fast. For example, if you look at a spec of the H100 GPU, you can see that the teraflops that you can get, number of computation that you can get with lower quantization options are much more than higher.

383. 57:28

     For example, FP16 teraflops will be much lower than FP8 because you can use the special tensor cores for doing more FP8 computations in the same time that you would do FP16.

384. 57:40

     But for that to happen, both sides of the matrix have to be the same quantization type. Mixed precision doesn't exist. Uh, at least now it's not very common or popular.

385. 57:49

     So you want both sides to be quantized. And when you quantize both the KV cache and the weights to FP8, you get that extra unlock that your computation is also faster, which can be critical for scenarios which are compute bound.

386. 58:05

     And as you would know, in LLMs there are con- there's a context phase and generation phase. Uh, generation phase is memory bandwidth bound, but the context phase is compute bound, so that can benefit greatly from both sides being quantized.

387. 58:18

     So in this case, we are saying that quantize both weights and KV cache. And it's actually, uh, not a trivial decision to do that because weights quantize very easily.

388. 58:28

     It's, uh, you hardly lose anything when you quantize weights. The dynamic range of weights is generally much, uh, much smaller. You can use int8 or, uh, when you do FP8, FP8 there is hardly any loss.

389. 58:40

     KV cache doesn't quantize as well, and that's why FP8 is a game changer. Because what we found is that when you quantize the KV cache with int8, even using SmoothQuant, there is a still degradation of quality.

390. 58:52

     And practically, we've never seen anybody use it, even though there are a lot of papers about it, and it's great, great technology. But practically, it was not there until FP8.

391. 59:02

     FP8 even KV cache quantization works extremely well.

392. 59:05

     Let me, uh, let, let, let me show something with that actually, um, if, if you don't mind back on the, um-

393. 59:11

     Yeah

394. 59:11

     ... on the, um... If we go to, uh...

395. 59:19

     I'll just show like a little visualization, um, for, for FP8 that, that shows off the dynamic range. Um, so... Oh, hey, look, it's us. Um, yeah. So, so when you look at, um, the, the FP8 data format, it has a sign, and then, um, rather than, uh...

396. 59:38

     So there's two different FP8 data formats, but, but we're using the, um, you know, the e4m3 format. So basically you have four bits dedicated to an exponent, um, and that's what gives your FP8 data format a lot of dynamic range versus int8 which is just, you know, like what?

397. 59:55

     Like two fifty-six to-

398. 59:57

     Two fifty-six.

399. 59:57

     Yeah. So, so you, you still have the same number of possible values, but they're spread apart further. That's dynamic range, and it's that which allows you to quantize this much more sensitive KV cache.

400. 1:00:10

     Yeah. Yeah. Exactly. Basically, you have the, uh, multi-channel exponents. You basically are able to, uh, quantize smaller values better. You give more bits to smaller scale than larger scale.

401. 1:00:23

     You don't have to fit into a linear scale, and that's where FP8 excels. So going back to the presentation, the FP8, um, quantization is done, and, uh, I forgot to show you this, but, uh, there is calibration involved here.

402. 1:00:36

     If you look at this, uh, stack here, we actually give it some data. We feed it some data and let it calibrate because as you would know, int8 and FP8, you have a start and end range, and they differ in how you divide up that range into data points, but you have to find min and max.

403. 1:00:52

     And for that, you need calibration. So we give it a standard dataset, and you can change the dataset. Uh, but we give it a specific dataset and it does multiple runs, and we try to calibrate, like what are the dynamic ranges of each of the layers of this transformer architecture?

404. 1:01:07

     And based upon that, we specify that min and max for each layer separately. There is more detail there, but at the high level, that's what is happening.

405. 1:01:24

     Yeah, yeah. It's, it's, uh, possible that the ranges can vary a lot with dataset, and this used to be more critical with int8. With FP8 we found that you, you get to a good state pretty fast, but it's worth thinking about trying different datasets, especially if you know what dataset you are going to be calling it with.

406. 1:01:41

     Uh, it could be worth it. It just works very well out of the box, but it's not perfect. [clears throat]

407. 1:01:49

     Going back to the workshop. Uh, so we were following along here and yeah, so after you quantize it, the steps are very similar as before.

408. 1:02:00

     Now we are building an, building an engine with FP8 and, uh, internally all the CUDA kernels that are being used are now FP8 specific. They are different kernels which to use the tensor cores in the right fashion.

409. 1:02:16

     And this should be pretty quick as well. [clears throat]

410. 1:02:26

     And there is a lot of depth here as you learn more about it. You don't need to, but, uh, there are things like timing cache, there are optimization profiles in TensorRT through which you tell what sizes we expect and it does optimizations.

411. 1:02:39

     But, uh, th-this is a good beginning. So now we have the engine. Uh, let me do a du on that and to see the size of that engine now.

412. 1:02:48

     And the size is one point two gigs, which is about half of previous, and which is what we expect because we quantized.

413. 1:02:57

     And now let's, uh, run this engine and see the output, and it should be pretty similar to what we saw before.

414. 1:03:04

     So using this run script, now it's gonna load the engine, and then we'll do a inference on top. That should be pretty quick.

415. 1:03:11

     So yeah, the... Here's the output, the same input as before, and about the same output as before.

416. 1:03:18

     And that's what we generally observe with FP8. FP8 quality is really, really good. It's very, very hard to tell the difference.

417. 1:03:26

     And that's it for this workshop, this, this part, part of the workshop.

418. 1:03:29

     Yeah. Awesome. Thank you. So, um, you know, I definitely welcome you to keep playing around, um, with this one pod setup and trying different, uh, different things,

419. 1:03:42

     try to build different engines and stuff. Um, but, uh, we're going to move on to the next step, which is a, um, automated version of basically exactly what we just did.

420. 1:03:53

     So, um, we're going to show a few things to, uh, to make this easier. Um,

421. 1:04:00

     so, uh, we're going to be using for this next step something called Truss. Uh, Truss is an open source, uh, model serving framework developed, uh, by us here at Baseten.

422. 1:04:09

     Pankaj is the one who, you know, actually wrote a lot of the code. All I did was name it Truss, um, because I was riding on a train and I was like, "Huh, what should I call the framework?"

423. 1:04:18

     And then we went over a bridge, and I was like, "I know. I'll call it Bridge." But that was already taken, so I called it Truss. Um, so it lets you deploy models with Python, um, instead of, you know, building a Docker image yourself.

424. 1:04:30

     Um, it gives you a nice live reload dev lo- dev loop. And what we really wanted to focus on when we were building this, 'cause it's kind of the technology that sits onto our entire, uh, model serving platform, is we really wanted a lot of flexibility so that we could work with, you know, things like TensorRT, TensorRT-LLM.

425. 1:04:47

     You can run it with vLLM, Triton. Uh, you can, you know, run a Transformers model, a Diffusers model. You can put an XGBoost model in there if you're still doing ML.

426. 1:04:55

     Like you can do basically whatever you want with it. It's just Python code.

427. 1:04:59

     Oh, if I may-

428. 1:04:59

     Yes

429. 1:04:59

     ... interject. Like Truss is actually, uh, a very simple system. It's a way of running Python code, specifying an environment for running the Python code and your Python code.

430. 1:05:09

     So it's sort of like a very simple packaging mechanism, but built for machine learning models. It takes account of the typical things you would need with the machine learning models, like getting access to data, passing security, secure tokens and such.

431. 1:05:23

     But it's, uh, fundamentally a very, very simple system, just a config file and some Python code.

432. 1:05:29

     Exactly. And, um, so looking at, uh, at that config file, we're not even actually gonna write any Python code today for the, um, model server. We're just going to write a quick config.

433. 1:05:41

     Um, so actually this morning I was eating breakfast here, and I sat down with a group of engineers, and we were talking about stuff. And everyone was complaining about YAML and how they were always getting like typos when they write YAML.

434. 1:05:52

     So, um, unfortunately, this, this is gonna be a YAML system. So apologize... apologies to my new friends from breakfast. Um, but what we're going to do is, is use this as basically an abstraction on top of TRT-LLM.

435. 1:06:06

     Uh, why do you need abstraction on top of TRT-LLM? Uh, Pankaj, a quick question for you. What's the name of that C++ textbook you were reading before bed every night the, the other month?

436. 1:06:16

     Modern C++.

437. 1:06:18

     What?

438. 1:06:18

     Modern C++.

439. 1:06:19

     Yeah. Modern C++. So, uh, you know, uh, before bed every night, uh, I was, I was watching Survivor. Uh, and so, uh, for, for those of us who are, who are not, um, cracked software engineers, um, and even for those who are, who wanna get things done quickly, um, we want to have a great abstraction.

440. 1:06:37

     What does that abstraction need to be able to do? It needs to be able to build an engine. Um, and that engine, it needs to take into account what model we're gonna run, what GPU we're gonna run it on, the input and output sequence lengths, the batch size, quantization, any of the other, um, optimizations we wanna do

441. 1:06:53

     on top of that. And then we also want to not just grab that and run it in the GPU pod somewhere. We actually want to deploy it behind an API endpoint so that, you know, we can integrate it into our product and stuff.

442. 1:07:06

     So I'm going to show how to do that. Uh, let, let's see here. This is, this is yours now. I'm, I'm stealing... Ooh, this is a good mic. I might not give this back, Pankaj.

443. 1:07:16

     This is a good mic. All right. Um, so we're going to go over, um... Let's see. This is, this is in the one pod thing still?

444. 1:07:29

     Uh, great.

445. 1:07:32

     The second one.

446. 1:07:33

     Perfect. [laughs]

447. 1:07:34

     It's not different. Let me do the workshop for that.

448. 1:07:37

     Okay. Yeah. Here, you, you do that. This is his computer, not my computer, so I don't know where anything is. It's like, uh, walking into someone else's house.

449. 1:07:46

     There you go.

450. 1:07:47

     All right. Thank you. Thank you so much. Um, okay. So, um, what we're going to do in this, in this second step is we are going to do basically exactly the same thing we just did, um, just automated.

451. 1:08:03

     So for this step, we're going to use Baseten. Um, we're gonna give you all some, some GPUs to play with here. Um, so if you wanna follow along, I really encourage you to do so.

452. 1:08:13

     Um, you're gonna go sign up at Baseten. We're gonna... You know, your account will automatically get free credits. If, um, our fraud system, uh, is a little freaked out by everyone, uh, signing up at the same time, well, fortunately, uh, we have some, uh, admin panel access ourselves over here.

453. 1:08:30

     So we'll just unplug this, approve you all, and plug it back in. Um, so yeah. So everyone go ahead and, um, sign up for Baseten. We're also going to want you to make an API key real quick and save that.

454. 1:08:42

     Um, and then once that's all done, we're going to jump into this, uh, this part of the, of the project.

455. 1:08:53

     Yeah.

456. 1:08:53

     We use your account on this.

457. 1:08:55

     Oh. Okay, everyone. So I know there's a, uh, few errors going on. Um, we have, we have pinged the team about that. Let me let you in on a little secret.

458. 1:09:06

     Uh, we shipped this on Thursday as an internal beta, and this is the very first time anyone who doesn't have an @[REDACTED:email_address] email address is using our, uh, new TensorRT-LLM build system.

459. 1:09:19

     So if the, uh... [audience applauding] If, uh... Yeah, so, uh, sorry for tricking you all into beta testing our software. Um, but hey, that's what demos are for, right? So, uh, we'll, we'll get that sorted out.

460. 1:09:33

     In the meantime, we have an image cached locally, which means we can keep going with the demo as if nothing ever happened.

461. 1:09:41

     So, um, let's see. So what you, uh, what you would see in the logs as you, uh, build, a-as you, uh, deploy this-

462. 1:09:52

     You can play the game this time.

463. 1:09:54

     Yeah. Oh, well, I mean, I can just kinda look through the, look through the logs right here. Um, let me actually just... Let me just, uh, wake it up.

464. 1:10:01

     Sorry, what was that?

465. 1:10:03

     Read the logs.

466. 1:10:03

     Okay. Yep, yep, yep. I got you. Um-

467. 1:10:08

     Log plus.

468. 1:10:09

     Yep. All right. Big logs. Um, let's see. All right. So, um,

469. 1:10:20

     what, what you're seeing here, um, as we... Oh, I'm sorry, we got a, we got a lot of logs here, um, right, 'cause we tested a bunch with the...

470. 1:10:30

     Just let this scale up. Um, anyway, what you see is, uh, you see the engine getting built, um, and then, and then deployed. And to walk through the YAML code really quick, uh, yes, here.

471. 1:10:43

     So we talked about that there were a bunch of different settings that you need to do when you are working with TRT-LLM. Um, and you can set all these settings right here in build.

472. 1:10:53

     So right now we're doing something with an input sequence and output sequence of two thousand tokens each, um, and a batch size of sixty-four concurrent requests. Um, we're using the INT8 quantization because we're running on a A10, and that does not support A- FP8 because it's an Ampere GPU, which is one generation before FP8 support.

473. 1:11:14

     Um, and then of course you, you pull in your, uh, model and stuff. And then if we want to, you know, call the model to test that it is working, um, we can come over here, um, to the call model.

474. 1:11:30

     Um, we can just test this out really quick.

475. 1:11:33

     Question.

476. 1:11:33

     Yes. [inaudible] So we do not. Um, on Baseten we have T4s, A10s, A100s, H100s, and H100 Migs and L4s as well. Um, we, we generally stick with the more like data center type GPUs, um, rather than the consumer GPUs.

477. 1:11:52

     It's a good GPU though.

478. 1:11:55

     Yeah. Uh, I want one for, uh, for, um, well, Pankaj is here, so I'm gonna say that I want it for legitimate business purposes and it should be an approved es- approved expense.

479. 1:12:05

     Uh, I don't want it for playing video games. Definitely not.

480. 1:12:09

     So-

481. 1:12:09

     One more question.

482. 1:12:10

     Yes. [inaudible] No, but I bet he can.

483. 1:12:23

     Yeah, you... Uh, all of that, uh, all of this course code is open source. And typically when new models come up, those companies provide, uh, convert checkpoint scripts. Uh, but if you can follow those scripts, it's not terribly difficult.

484. 1:12:36

     It's mostly like, uh, if you're familiar with the Transformers library, it's about reading weights from a Hugging Face transformer model and converting that into something else. It's a simple transformation, so it should be possible to do it yourself if you want.

485. 1:12:57

     Awesome. So once, once your model is deployed, again you can, you know, you can just test it really quick. You can call it with a API endpoint. Um, but, uh, yeah, we're coming up on, on two thirty here, so I'm not gonna spend too long on this example.

486. 1:13:12

     Um, let's see. But you know, we've been talking a big game up here about performance, right? And performance is not just, "Okay, I'm testing it by myself." Performance is in production for my actual users.

487. 1:13:26

     Is this meeting my needs at a cost that is reasonable to me? And in order to, you know, validate your performance before you go to production, you need to do benchmarking.

488. 1:13:36

     And you need to do a lot more rigorous benchmarking than just saying like, "Hey, you know, I, I, I called it. It seemed pretty fast." Um, so what do you want to measure when you're benchmarking?

489. 1:13:47

     Uh, uh, you know, say it with me, everyone. It depends. That's what us software engineers are always saying. So, um, you know, depending on your use case, you might have different things that you're optimizing for.

490. 1:13:58

     If you're, say, like a live chat service, uh, you probably really care about time to first token for your streaming output because, you know, you're, you're trying to give people, you know, instantaneous responses.

491. 1:14:09

     Um, you might also care a lot about tokens per second. Um, so that's, you know, how many, how many tokens are, are generated. Generally, some, some good numbers to keep in mind is somewhere...

492. 1:14:21

     Depending on the tokenizer and the data and everything and the reader, somewhere between thirty to fifty tokens per second is gonna be about as fast as anyone can read.

493. 1:14:31

     So, you know, if you're at fifty tokens per second, generally it's gonna feel pretty fast. People aren't going to be waiting for your output. However, if you're doing something like code, you know, code takes more tokens per word than, say, natural language, so you're gonna need even more tokens per second for that, you know, nice smooth output.

494. 1:14:48

     And then from there, we're getting into, you know, one hundred, two hundred tokens. That's when it just feels kind of, you know, magically fast. But again, we'll, we'll... Our influence is all about trade-offs, right?

495. 1:14:58

     When we're optimizing. So, you know, sometimes you might wanna trade off a, you know, a few... You know, maybe, maybe you're gonna go at a hundred, not a hundred and twenty tokens per second because that gets you a bigger batch size, which is gonna lower your cost per million tokens.

496. 1:15:12

     Another thing you're gonna wanna look at, um, when you're running your benchmarks is your total tokens per second. So there's the tokens per second per user, right? Like per request.

497. 1:15:21

     How many tokens is your end user seeing? And then there's tokens per second in terms of how many tokens is your GPU actually producing. And that's a really important metric for throughput, for cost, um, especially if you're going to be doing anything that's a little less than real time.

498. 1:15:36

     Um, you wanna look at this not just once. You wanna look at the, uh, fiftieth, ninetieth, ninety-fifth, ninety-ninth percentile, make sure you're good with all those. And you wanna look at the effects of different batch sizes on this.

499. 1:15:48

     And, um, so something is that benchmarking actually reveals really important information. It's not linear, and it's not obvious. The sort of performance space of your model is not this ni- nice flat piece of paper that goes linearly from batch size to batch size.

500. 1:16:05

     So this is a graph of time to first token for like a Mistral model that I ran a long time ago. I just happened to have a pretty graph of it, so that's how it ended up in the presentation.

501. 1:16:16

     Um, so if you look at the batch sizes as it's, uh, you know, increasing, doubling, um, thirty-two to sixty-four, the time to first token like barely budges. Um, but as it goes from sixty-four to one twenty-eight, doubling again, the time to first token, uh, increases massively.

502. 1:16:32

     And in this case, you know, the reason behind that is we're, we're in the, you know, compute bound, um, prefill step, um, when we're talking about computing the first token, and there's these different sort of slots that this computation can happen in.

503. 1:16:45

     And as you increase the ba- increase the batch size, you're saturating these slots until eventually you have an increased chance of a slot collision, and that's what's gonna lock at your time to first token.

504. 1:16:55

     I'm glad you're nodding. I'm gl- I'm glad I got that right. Um, but yeah. All of this to, uh, all of this to say, um, you know, the performance that you get out of your model once it's actually built and deployed is not necessarily just going to be linear.

505. 1:17:10

     It's not gonna be something super predictable. You have to actually benchmark your deployment before you put it into production. Otherwise, these sort of surprises can, can happen quite often.

506. 1:17:20

     Um, so yeah. So Pankaj, do you wanna take over the, uh, the benchmarking script?

507. 1:17:27

     So, uh, just for this, uh, workshop, we wrote a benchmarking script. It's not the script we use, uh, ourselves, but it's a simpler version so that you can follow along, that if you wanted to modify it, you can play around with it and understand it easily.

508. 1:17:42

     Uh, if you go into that repository, it's, uh, it's a very simple script where we send requests in parallel just using Python, using async libraries. And, uh, all you give it is the URL of the endpoint of the model where your model is deployed, and you can give it different concurrencies and input lengths and output lengths and

509. 1:18:03

     give it number of runs. You want to run these benchmarks a number of times to get, uh, an idea of values. One run might be off. So I'm just gonna run that script, and it's all, it's all in the benchmark repository.

510. 1:18:15

     Uh, it's all structured using Makefiles. And there is a Makefile target for benchmark that we're gonna use.

511. 1:18:23

     And, uh, and the README should also have instructions on that.

512. 1:18:30

     So we're basically gonna run this, and we're gonna need the base URL. We... You need two things. We need to export the API key, and then we need to, uh, supply the URL.

513. 1:18:41

     So once you deploy your model on Baseten, you, you would see deployed, and like Philip said, there is a Call Model button. There are various ways you can deploy it.

514. 1:18:50

     Ultimately, it's an HTTP API, and you can just copy this URL for that model for our benchmarking script. Uh, but if you wanna play around, there are examples in all kinds of, uh, languages.

515. 1:19:03

     And you can also click Streaming. This will give you streaming code. Streaming is very important with large language models because you wanna see the output as soon as possible.

516. 1:19:11

     So you're gonna take this output and, uh, I don't know if I exported the API key, so give me one second to export the API key. [laughs]

517. 1:19:22

     Come on, Pankaj. Grow up so you can show your API keys. [laughs]

518. 1:19:27

     This is a good time to mention... Oh, yes. What's your question?[background noise]

519. 1:20:01

     All right. And, uh, [clears throat] you know, if you lose the API key, you can always revoke it. So that's good.

520. 1:20:08

     Yes. Uh, well, this is a good time to mention that Baseten is SOC 2 Type II certified and HIPAA compliant. Uh, and that is why we cannot show you our API keys. [chuckles]

521. 1:20:19

     That is one of the reasons.

522. 1:20:21

     And now we just gonna give it the URL. Here, let me go back.

523. 1:20:30

     This call model. All right. So I'm gonna do this, uh, first run with a concurrency of thirty-two and input and output lengths of a thousand. And, uh, let's see it work.

524. 1:20:49

     So first it does a warm-up run just to make sure that there is some traffic on the GPU. You always want to have a warm-up run before you get the real numbers.

525. 1:20:58

     Now, as this is, uh, running, and you can see the TPS here. The total TPS is five thousand, and this is on eight NG. Eight NG is not the most powerful,

526. 1:21:09

     uh, GPU, but this TPS is still very, very high. Five thousand is very, very high. It's because this is tiny Llama model. Tiny Llama is a tiny model, just like a billion parameters, and that's why we see this very high.

527. 1:21:22

     Um, but yeah, on bigger GPUs with the, with Llama eight G, you should also see very, very high values because H100s are very, very powerful and TensorRT-LLM is very, very optimized.

528. 1:21:33

     I think we see up to like eleven thousand tokens per second, and you should do a comparison. It's really, really high. Uh, in this case, we have two runs, and you see these values.

529. 1:21:43

     Uh, let me try a different run. Now I'm going to do concurrency of one. And one is good to know how best of a time to first token you can get.

530. 1:21:51

     So you're just sending one request at a time, and many requests, but one at a time with the same input and output length. And, uh, this should run... Okay, I think I ran into an error.

531. 1:22:01

     Let's try again. So, uh, you see time to first token of, uh, a hundred and eighty milliseconds here. And this is from this laptop. I am running it right from this laptop on this Wi-Fi, and my model is deployed somewhere in, in US Central, and this is, uh, one eighty milliseconds for that.

532. 1:22:24

     Uh, one-

533. 1:22:24

     Yeah. So, so the, so the vast majority of that time to first token is gonna be network latency, right? Not model latency.

534. 1:22:30

     Not the model itself. It should be pretty fast. Um-

535. 1:22:32

     Uh, yes, sir, over there.

536. 1:22:34

     Um, what is that error message, um, PyTorch 404 Flex?

537. 1:22:40

     Oh, uh, not, not that one. Um, this one.

538. 1:22:45

     None of...

539. 1:22:48

     Like, is this a message from, uh, Baseten or is it a message from, uh-

540. 1:22:54

     This is from this, uh, script, uh, script that I'm running. I didn't do a thorough job of cleaning up everything. [chuckles] We're saying that, uh, we're just making an RPC call.

541. 1:23:02

     We don't need PyTorch or whatever. This, this-

542. 1:23:05

     Oh, so this is from the local runtime.

543. 1:23:06

     Yeah, local runtime. Something, uh, wrong with the machine. This script, uh, if you look at that, all it's doing is RPC. Uh, let me go through that real, real quick.

544. 1:23:17

     Benchmark script. Uh, it's just a simple Python using Async. And it's amazing how good Python has got. With this Async API, you're able to load, uh, load this model with thousands of tokens per second, all of them coming in streaming.

545. 1:23:30

     Uh, Python has actually gotten really, really well. There was a, there was a case where I was loading with K six, and K six client became a bottleneck because H100s are so fast.

546. 1:23:40

     But Python could keep up. Python was able to load it very well.

547. 1:23:45

     Uh, but I would... I'm getting distracted. So, uh, we tried concurrency one, which is like the best-case scenario. Latencies are very, very good, and TTFT should be very low.

548. 1:23:54

     Now let's go to the other extreme. If you look at this, uh, model that we deployed, uh, we created it with a batch size of sixty-four maximum. So now we'll do sixty-four.

549. 1:24:05

     And, uh, I'm hoping we see throughput improvements.

550. 1:24:11

     So ignore the warm-up run. And this is gonna take a bit longer because now we're gonna send sixty-four requests at the same time. Sixty-four is not, uh, not that high.

551. 1:24:22

     We can go even higher. So in this case, you see total TPS of, uh, uh, of seven thousand, which is, uh, even higher than before. We saw five thousand before.

552. 1:24:32

     This goes up to seven thousand. But maybe this is a fluke. Let's wait for the second run. And you can run... Uh, this is also seven thousand. So this is, uh, this is much better than what we saw before.

553. 1:24:42

     So, uh, if you increase batch size, you would fi- find that your latencies become, become higher. Latencies in the sense that for every request that a user is sending, now tokens are coming slower and slower.

554. 1:24:55

     And then you have to make a trade-off at some point. Like, is it still good enough? Is it still more than, say, thirty or fifty tokens per second that users won't perceive?

555. 1:25:03

     And at some point it will become unusable. Uh, and as you increase batch size, the pressure on your GPU memory also increases because all these extra batches, they require KV cache to be kept in GPU memory.

556. 1:25:17

     So you might hit that bottleneck. So depending on all these scenarios, you want to experiment with different batch sizes and find the sweet spot. And, uh, that's, uh, that takes a bit of time, but it's not terribly complex.

557. 1:25:30

     Um, yeah. So this script is there, there for you to modify and play around with. It's pretty simple. It's pretty much a single, uh, Python file. Not much in there.

558. 1:25:40

     You can modify it. You can run it.

559. 1:25:44

     Yes. Yes. Uh, the question was, uh, did I have a max batch size in mind when I was running?

560. 1:26:02

     Yes, because I deployed the model with the config in this, uh, workshop. Let me show you that. I built the model with a max batch size, and you can increase that batch size.

561. 1:26:13

     So let me show you that. Uh, so in this, uh, TinyLlama model that I deployed, I specified a max batch size of sixty-four. So, uh, if I go beyond sixty-four, it's not gonna help me because all those requests will just wait.

562. 1:26:32

     And yeah, actually, there's one m-- one interesting thing I want to show you. Uh, so this is a good question. You can look at the logs here, and in the logs, we put these, uh, metrics for what's going on.

563. 1:26:46

     I think maybe I'm not-

564. 1:26:48

     Yeah, and, and if you wanted to increase your batch size past sixty-four, you just change the YAML and say like, "Oh, match batch size. That should be like one twenty-eight," and build a new engine by deploying it.

565. 1:26:59

     Yeah. So if you look at these logs here, it shows how many requests are running in parallel, active request count. Uh, you can actually observe how many requests are being executed in parallel right on the GPU.

566. 1:27:12

     'Cause there are chances that you haven't, uh, configured something right and that, uh, for whatever reason, the, uh, requests are not all getting executed in parallel. For example, a common mistake one could make is that when you deploy on, on Baseten, and this is Baseten specific, but just to take an example, there are scaling settings in Baseten.

567. 1:27:33

     You can specify the scale, and you can specify what is the max concurrency the model will receive. In this case, I've set it to a very, very high value, so it won't become a bottleneck.

568. 1:27:42

     But there are chances that, that, you know, you forget, you make a mistake there. You, you can check these logs, and they will actually tell you what's happening on the GPU.

569. 1:27:52

     And I think I lost [laughs] that again. Uh, let me go here.

570. 1:27:58

     So yeah, these are actual metrics from the TensorRT-LLM batch manager, which tells you what's going on. It also tells you about the KV cache blocks that are being used, and that helps tune you...

571. 1:28:09

     uh, helps you tune the KV cache size. For example, in this case, it says it's using eight hundred and thirty-two KV cache blocks and forty-five hundred are empty, which means there is the-- there is way more KV cache than is needed for this use case.

572. 1:28:23

     So just to mention that as a, as an aside.

573. 1:28:28

     Yeah, I think that's it for that presentation.

574. 1:28:33

     I'm gonna, I'm gonna, I'm gonna talk about that next slide. [laughs] Thank you. You're reading my mind. Yes.

575. 1:28:51

     It does, yes. TensorRT-LLM does come with a benchmarking toolkit. It's, it's very, very good. Uh, the only downside is that you have to build it from source. It's not bundled with the TensorRT-LLM Python library, which is my gripe.

576. 1:29:06

     I'm gonna ask NVIDIA [laughs] to fix that. Uh, they, they have benchmarking tools. There are two benchmarking tools. One that, uh, that just sends a single batch and, um, and measures the raw throughput you can get without serving it through in-flight batching.

577. 1:29:22

     And there is a separate second, second tool, uh, called the GPD manager benchmark, which actually starts up a server and does in-flight batching on top. So there are two tools, and they're very, very good quality, but they're not available easily.

578. 1:29:34

     Building TensorRT-LLM is like with ninety-six CPUs, it takes us one and a half hours to build it. It's not for the weak of heart. [laughs]

579. 1:29:42

     Or for the short of workshop. [laughs] So, um, we just, we just have a, a... We have a, a few minutes left. Um, so I wanna run through a few slides and then leave time for last-minute questions.

580. 1:29:54

     So, um, I was asked, um, you know, how do we, how do we actually run this in production? What does the auto-scaling look like? How does that all work?

581. 1:30:02

     So, um, how do you run a TensorRT engine? So you use something called Triton, the Triton Inference Server. Um, and that's what helps you, you know, take the engine and actually serve requests to it.

582. 1:30:14

     We're actually working on our own server, um, that uses the same spec, but supports, uh, C++ tokenization, detokenization custom features for, for even more performance. Um, but the... You know, as we've talked about, the engine is specific to versions, GPUs, batch sizes, sequence lengths, all that sort of stuff.

583. 1:30:35

     So, um, that causes some challenges when you're running it in production. We've talked this whole time about vertical scale, right? Like how do I get more scale off a single GPU?

584. 1:30:45

     There's also horizontal scale. How do I just like get more GPUs? How do I, you know, auto-automatically scale my, my platform up to meet my traffic demands? So, um, you know, some challenges in scaling out in general.

585. 1:30:58

     You know, you have to automatically respond to traffic. You have to manage your cold start times. You have to manage the availability and reliability of your GPU nodes. You have to route requests, do batching, all that kind of stuff.

586. 1:31:10

     And then TensorRT-LLM adds a few more challenges. Um, you've got these large image sizes, um, so that's gonna make your cold starts even slower. You've got these specific batching requirements, so you can't just like send whatever traffic however you want.

587. 1:31:24

     And you have these specific GPU requirements. So when you spin up a new node, it's gotta be exactly the same as your old node or your model's not gonna work.

588. 1:31:32

     Um, and, uh, you know, fort-unfortunately, our workshop is almost over. Otherwise, I would love to give you an in-depth answer of how to solve all these problems. Uh, but the quick answer to how to solve all these problems is you run your code on Baseten, uh, because we solved it all for you.

589. 1:31:47

     Um, so Baseten is a model inference platform. Um, it's the company that we both work at. Um, with Baseten, you can deploy models on GPUs. Uh, you can use TensorRT-LLM, but you don't have to.

590. 1:31:59

     You can use any other service, vLLM, TGI. Just like a vanilla model deployment. Um, you get access to auto-scaling, fast cold starts, um, scale to zero, tons of other great infrastructure features.

591. 1:32:12

     You get access to all of our model optimizations. We have a bunch of prepackaged and pre-optimized models for you to work with. Um, so yeah. Uh, the last thing before we go, um, is we are co-hosting a, um, happy hour tomorrow on [REDACTED:location].

592. 1:32:31

     Um, I was not really involved in organizing it, but the sales guys who organized it told me that it's a super sweet spot and that we're gonna have a great time.

593. 1:32:39

     Um, so yeah. So I'm gonna be there, um, and a bunch of other great, uh, AI engineers are gonna be there. So please feel free to sign up, come on through.

594. 1:32:46

     We'd love to have you. Um, it's gonna be super sweet, cool party for the cool kids.

595. 1:32:54

     Um-

596. 1:32:55

     And you've been, you, you've been really great. Thank you for listening to us, and we open it up for questions. Uh, I feel one question we didn't answer about auto-scaling, right?

597. 1:33:03

     Maybe I should take that one now.

598. 1:33:04

     Yeah. Yeah. Yeah, go ahead.

599. 1:33:05

     So how does auto-scaling work? Yeah, yeah, please, uh, yeah, please feel free to... I'll just, uh, finish that question because you asked that, I wanted to answer it. And, uh, how auto-scaling works is that we use a system called Knative, but we forked it to make it work for machine learning use cases.

600. 1:33:20

     Knative, if, if I understand correctly, was built for microservices where your requests are very, very quick, like, you know, one second or two seconds. It doesn't exactly apply to machine learning model use cases where your request is long-lasting and you're streaming and you need to, uh, still scale, but, uh, some of the considerations are different.

601. 1:33:38

     So we had to actually fork it to be able to apply settings dynamically. For example, a lot of settings that you see on Baseten, they apply immediately, like within a second.

602. 1:33:47

     Whereas in Knative, you would need to deploy a new revision for those settings to apply. And it's pra-- it's not even practical because the way you deploy a new model, it creates so much hassle and requires extra capacity that it's not good.

603. 1:34:00

     So we made changes to Knative to cater to the machine learning use case. Uh, but fundamentally, the idea is, is very simple. As a request come in, you specify the capacity of your pod that it can take in, and if it's, uh, if it reaches near there, we spin up a new pod, and then the traffic spreads.

604. 1:34:18

     If your traffic goes down, it goes, uh, your GPUs are, uh, your pods are reduced, your GPUs are freed up, up all the way up to zero. And when the traffic arrives, it kept, it's kept in a queue, and then the model is spin up, and the requests are, are sent there.

605. 1:34:34

     A lot of the machinery at Baseten is around improving cold starts. How do we start up these, you know, giant models, fifty gig models in under a minute, right?

606. 1:34:42

     Minute sounds like a long time, but when you're talking about fifty gigs, fifty gigs is also a lot. And for, for ten gig models, we aim for less than ten seconds.

607. 1:34:50

     For fifty gig models, we aim for less than a minute because that's really important. Unless you can scale up in a minute, your request is going to time out, so you really can't scale to zero.

608. 1:34:59

     So it's, uh, it seems like a detail, but it's very critical. You can't have a scale to zero without very, very fast cold starts. Uh, so we have the whole machinery built out for that.

609. 1:35:09

     Even before LLMs became popular, we've had this machinery in place.

610. 1:35:16

     Yes.

611. 1:35:16

     Yeah, please.

612. 1:35:19

     Um, so what would I expect in terms of performance if I were, like, if I take, took a Llama 8 billion model and I, I used TensorRT versus, um, deploying it on Baseten, would I expect any difference in performance or price points?

613. 1:35:37

     I think if you optimize it... Yeah, can I? Yeah.

614. 1:35:39

     What, what... Are you saying, like, deploy it with TensorRT versus deploy it on Baseten?

615. 1:35:44

     Yeah.

616. 1:35:44

     Uh, so, so you, you would deploy it on Baseten using TensorRT under the hood to run it. Uh, so, so Baseten's just gonna, like, facilitate that TensorRT deployment for you.

617. 1:35:55

     Right. But I, I could deploy it, I guess, on hardware-

618. 1:35:58

     Yes.

619. 1:35:58

     Mm-hmm.

620. 1:35:58

     -as well, like, on hardware, for example, right?

621. 1:36:01

     Yeah, no, no difference. Baseten runs TensorRT-LLM. We just make it very easy to, uh, run TensorRT, TensorRT-LLM. So you... It's, uh, easier and faster for you to get at the optimum point.

622. 1:36:13

     Uh, but if you could do it yourself, yeah, it's, it's the same thing under the hood.

623. 1:36:16

     And then, you know, I'm compelled by the fact that I'm in the marketing department to say things like, we also provide a lot of infrastructure value on top of that so that you're not managing your own, you know-

624. 1:36:27

     Yeah, yeah

625. 1:36:27

     ... scaling mechanisms.

626. 1:36:28

     Actually, that is, that is true, yeah. Because we, we have a large fleet, we, we get good cost. So you actually won't pay higher on Baseten.

627. 1:36:35

     Yeah.

628. 1:36:35

     Like, it's not that you're gonna, uh, it's gonna cost you more on Baseten.

629. 1:36:39

     Yes, over there.

630. 1:36:41

     Yeah, I, I was wondering if, if you had any feedbacks off the bat. Um, is Tru-- like, the open source Truss package that you guys have, I see that that's got a, the, uh, Python inference client and seems to have just, like, some features for being able to push models to Baseten.

631. 1:36:57

     Have you open sourced any of the serving infrastructure? Like, if I just wanted to, you know, go off our, our, uh, one pod example and do, like, a local host serve and send that request first?

632. 1:37:11

     Yeah. So only the packaging part is open source. The serving parts, that, I mean, that, that's kinda, that's kind of the platform, so...

633. 1:37:19

     I do, I do want to mention is that, uh, from Truss, you can create a Docker image, and that Docker image can be run anywhere. So you get pretty close.

634. 1:37:26

     You don't get auto-scaling and, uh, all of those, that nice dev loop, but you do get a Docker image, and you can do a lot with a Docker image.

635. 1:37:34

     Oh, so it builds the Docker image locally?

636. 1:37:36

     It builds, yeah, locally, yeah. You can build... There is a Truss, uh, image build command. You point it to a Truss, it will build a Docker image locally.

637. 1:37:44

     Oh, okay. All right. So I guess then the serving code is, it's gotta be open source then.

638. 1:37:48

     Serving on a single pod or container, but there is also spreading across multiple containers, the auto-scaling and all of the dev loop, uh, that is not, yeah. But the serving, yeah.

639. 1:37:59

     I mean, single model serving is in Truss with that image.

640. 1:38:02

     Could I ask, like, a, um... I'm just kind of curious. I don't know how long it would take you to figure out. Like, uh, what is the stack that you have for serving models?

641. 1:38:12

     Yes. It's, um, it's, uh, FastAPI at the Truss server level. Then internally, we have our own server layer that we wrote to interact with TensorRT-LLM. Uh, that part is not open source.

642. 1:38:25

     We-- It's, it's very new, so we're still figuring out when to or where to open source it. But there is also a version that uses Triton. So there is the FastAPI, then there's Triton, and then there is the TensorRT-LLM library, and then it runs the engine.

643. 1:38:39

     Yeah, yeah, yeah. Exactly, exactly. We have, uh, we have, uh, we work on multiple cloud providers. We are spread across, uh, I don't wanna say globe, like mostly US, but also Australia and a few other places.

644. 1:38:50

     So we have access to many different kinds of hardware. We find the right hardware to build the engines, and then we deploy it.

645. 1:38:55

     Do you use, like, um, uh, like, on-premise clusters or, or you use public cloud ones?

646. 1:39:02

     Yes, you can... Yeah, you can use self-hosted clusters. Our stack is built that way. That's one of our selling points. We're not giving you an API. You can run the entire stack in a self-hosted way.

647. 1:39:13

     Awesome. Well, look, the conference organizers were very clear with us. All sessions and workshops are to end on time. So I'm going to wrap it up here, but, um, we're gonna be right outside if you have any questions.

648. 1:39:25

     If you have any easy questions, come see me. If you have any hard questions, please go talk to Pankaj instead. Thank you all so much for being here. It was so much fun doing this workshop with all of you.

649. 1:39:35

     Again, I'm Philip, this is Pankaj. We're from Baseten, and thank you so much for being here. Have a great conference, everyone. [applause]

650. 1:39:42

     Thank you. [outro music]