How fast are LLM inference engines anyway?

Talk transcript

1. 0:00

   [upbeat music] Thanks everybody for coming.

2. 0:16

   Um, yeah, wanted to talk about some work I've done recently on trying to figure out, uh, just how fast these inference engines are when you run open models on them.

3. 0:29

   Uh, so the kinda ... Been talking at AI Engineer since it was AI Engineer Summit two years ago. Um, and the, for a long time, it's basically been the, like, OpenAI wrapper conference, right?

4. 0:46

   It's like, 'cause just 'cause, yeah, what am I gonna do? Am I gonna run an agent with BERT? Probably not. Um, and that was, like, it was exciting to talk about all these cool new technologies, see people building stuff like Cursor on top of them, or Devin.

5. 0:58

   Um, but for me as somebody coming from, like, having trained my own models a lot, it was like, oh man, I want, I wanna, I wanna touch the weights.

6. 1:05

   I wanna play with them. I want to, like, hack them. And the quality, but the quality just, like, wasn't there yet to do, like, some of the interesting stuff.

7. 1:14

   Um, and that's changed. You've got the Llama series, we've got the Qwen series, we've got DeepSeek. And so now we're, like, catching up to, like, maybe even literally catching up with the frontier labs, which would be pretty crazy.

8. 1:26

   Um, but we're at the very least, like, at the point where a lot of things people have been talking about at AI Engineer for years are possible with open-weights models where they weren't before.

9. 1:35

   Um, and at the same time, there's also the development of the software stack on top of that. Um, also as somebody coming from, like, writing my own, writing and running my own PyTorch models, I was like, how hard could it be to run a language model?

10. 1:45

    Like, you just, you torch.nn.Module, wra-wrap a class around it. And like, yeah, I mean, having done stuff with Transformers before, it's like, oh yeah, there's def- It's a little more complicated with Transformers.

11. 1:55

    Training looks weird, inference is different. Um, but you know, pretty quickly the state of play has advanced a lot on how to run a transformer, right? KV caching is like, you know, just the first thing, and then it's this, you know, um, page detention.

12. 2:08

    Now multi-token prediction, speculative decoding, all this stuff. That's, like, pretty hard to write yourself. And so, you know, you want software to do that for you probably. Um, and so these open source engines are available now.

13. 2:20

    You got vLLM and SGLang and TensorRT-LLM. So the combination of those two things has, like, kind of flipped the, uh, playing field around to where there's like, you'd need a really good reason to run your own models, like you're the US government or something.

14. 2:34

    Um, or you wanna run on an air gap system. Um, or you, like, or you had to, like, believe it in your heart. You know, you had to want open source models like the, um, like, Noose or Prime Intellect.

15. 2:47

    You had to be like a decentralized crypto bro or whatever to want to run your own op-open models. Um, but yeah. Now the, the like, uh, the situation has changed, which is really exciting.

16. 2:57

    It sort of finally makes sense to self host.

17. 3:00

    Um, so, uh, the ... Just, like, wanted to do this really quickly. I don't ... Was anybody here at this AI Engineer Summit 2023? The, like, first one? Anybody?

18. 3:11

    Okay. Did anybody come to the AI Engineering 201 workshop the day before, um, uh, in the [REDACTED:location_address]? Maybe not. I talked to one or two people who were there.

19. 3:22

    Um, so I gave a talk on, on like, you know, how to, how to do your own AI stuff back then. Just wanted to pull out like a couple of slides.

20. 3:31

    Um, so like, the, one of the main ... This is like 2023. We're like, who's gonna win, open models or closed models? And the key statement, uh, in the talk was, like, if capabilities requirements saturate, open models will catch up to proprietary models and then dominate for those cases.

21. 3:48

    Inspired by what you see with operating systems, databases, programming languages. Like, as soon as there's, like, a, a sort of like, you know, um, capability level that everybody, like ...

22. 3:59

    That you don't need the absolute best thing. You just need something that's good enough. Open, uh, like, collaborative projects tend to catch up and, and then have better properties, and that's happened with open models.

23. 4:11

    So, check. Um, uh, yeah. So that, um, that was mostly around, like, capabilities. Um, and you know, at the time there was only Llama, but now we got a lot.

24. 4:22

    Um, the other one was, yeah, talked a little bit about LLM inference libraries at the time. And most of them are gone. TGI, RIP, um, uh, for example. But vLLM was good then, it stuck around now.

25. 4:34

    Um, yeah. So, uh, I don't know what the slides in 2027's AI Engineering, uh, Conference are gonna look like, but, um, the very least, like, three of my slides from two years ago were right.

26. 4:46

    Um, so yeah. Um, hopefully it's not those three slides again in two years. Um, but yeah. All right. So what does the LLM engine landscape look like two years later?

27. 4:56

    Um, let's, uh, take a look. So what, uh, what ... You know, we were advising a bunch of people on how to run ... Like, people were coming to us, like, "I wanna run my own code completion editor, uh, in editors.

28. 5:08

    I wanna run like big backfill jobs to f- to, like, um, enrich data and databases with language models." Uh, it's too expensive or, uh, to run it on OpenAI or, "I train my own model," so it's too expensive to run it on a, like, a provider like Fireworks.

29. 5:24

    They wanna run it on an more generic infrastructure like what we have at Modal. Um, and so they would ... People would come and they'd be like, "All right.

30. 5:31

    Well, how fast can you run an eight billion parameter Llama model with SGLang, uh, at like, with 128 tokens in, 1,024 tokens out on a Tuesday when Mercury's in retrograde?"

31. 5:43

    Um, and that would, like, take ... At first it took a couple days to like re- you know, figure out how to make sure all the packages are working, that we've got the fastest versions installed and, um, like, uh, that we can give like a, you know, trustworthy number.

32. 5:57

    Got that down eventually to like, you know, like an hour or two, um, and then built some benchmarking software so we could get it done in about, like, 15, 20 minutes.

33. 6:06

    But then, like, you know, the people ended up asking a lot of similar questions. So we decided just, uh, you know, fifth law of, uh, fifth mantra of performance is, uh, do it when they're not looking.

34. 6:18

    Um, so like compute the thing ahead of time and store it. Uh, so we ran a giant benchmark over, um, like 10 or so different models on vLLM, SGLang, and TensorRT-LLM on about 10 different context lengths, um, and put that all up on the internet.

35. 6:35

    So let's take a look at that

36. 6:37

    Let's see. I'll drop into this one. So this is a live version of it. So you find this at modal.com/llmalmanac. Um, the idea is that this is just one page in the like, you know, your almanac, your little book that has the useful things you need to know to be an LLM engineer.

37. 6:55

    So to start, we've got our benchmarking results, ah, benchmarking methodology in detail, the open source code for it, and a little executive summary. Um, hope to put more stuff up there as we accumulate the things people need.

38. 7:07

    Um, things about like speculative decoding and multi-token prediction and quantization. Um, but yeah, so to start off, we got this little interface here. So, uh, yeah, anybody, what's a model people wanna see results for?

39. 7:21

    Any-- if, uh, hopefully that's legible to people. Anybody got a favorite? Qwen three. Hmm?

40. 7:27

    Qwen three. Qwen three? Ministrel. Ministrel? Okay. Uh, excuse me, uh, that... Oh, man, that's, that's my boss. I'm not gonna do that one. Um, okay, we'll do any engine here.

41. 7:37

    Oh yeah, we didn't do, uh, so this was fun. SGLang's Qwen three support is a little buggy for the f- um, for the eight bit quant that we ran.

42. 7:46

    So I think we only have results for vLLM. We'll stick with any engine there. Oh yeah, by the way, if you, if you try this thing out, um, like you'll see, like if there's a giant tensor of configurations, right?

43. 7:57

    And we'd love to have that full tensor, but it doesn't alway-- it's like either not always possible or, um, like it's not clear how to do it. So there's a place where you can contribute configurations so we can build up a nice big database of like how to run these models.

44. 8:11

    We also haven't like carefully optimized any of these things. We started with out of the box performance for all the engines, which is like optimizing a hundred configurations is gonna take some time.

45. 8:19

    So we'd love, uh, like contributions of optimized implementations, especially TensorRT-LLM, which has like a ton of knobs. Um, and they have names like user buffer. Like what is that?

46. 8:31

    Um, yeah. Um, okay, yeah. So first token under one second. Let's say this is a, this is a pretty common SLO. Like you want one second's a nice round number.

47. 8:41

    People feel like that's like a good amount of time to wait. I'd say three hundred milliseconds is a tighter one. That's more like interactive. That's your, uh, Doherty threshold if you're a fan of Halt and Catch Fire.

48. 8:51

    Um, made up number, but like if somebody repeats a made up number enough, it's a real number. Um, so yeah, three hundred milliseconds. Okay, so we can get a throughput of about one request per second on Qwen three in the mixture of experts model on vLLM for a hundred and twenty-eight tokens in, thousand twenty-four tokens out.

49. 9:10

    Um, so that's we got. Over here, we got like a little, uh, code snippet here. So it should be the case that you can UVX Modal run this, and if you have a token, that should just work immediately.

50. 9:19

    Uh, and by immediately, I mean after five minutes of loading the model weights and, and spinning up the model server, but that's as immediate as it gets. Um, cool.

51. 9:27

    All right, so that's one result. Uh, who's asking for Qwen? Are you satisfied? Yeah. Okay, great. Um...

52. 9:36

    Gemma three twenty-seven B. Gemma three twenty-seven B. All right, on this one, let's do any engine here. Oh, right, this was... Oh, right, sorry, we got a tight filter on this.

53. 9:44

    I'm gonna put the first token filter up. Oh yeah, this one we're doing the BF16 quant is the only one that we could get working at first. So I think we eventually got the eight bit quants working.

54. 9:56

    I don't think that's a hard blocker, but it was the easiest one to get going was the BF16. So these are definitely slower. Um, so you'll see twenty-seven billion parameter models, so like ten X smaller in, uh, model weights.

55. 10:09

    Roughly the same number of active parameters as Qwen three, but we're getting like about the same, um, like throughput and request per second on the same load, one twenty-eight in, ten twenty-four out.

56. 10:20

    Um, so yeah. So it's interesting, you can see sort of which ones have had a more optimization work on them. I think the Qwen three models and the Llama model series, you see a lot more optimization.

57. 10:31

    Um, this is also one of the ones where you saw the biggest gap between, um, SGLang and vLLM, the Gemma one. So it looks like the vLLM team spent a little bit more time, or Google's contributed a little bit more to vLLM on, uh, getting good results.

58. 10:45

    Oh yeah, let's go. Yeah. So the other thing you'll see is you generally... Like, so I, so I just switched. Sorry, I should say what I'm doing. So this is a hundred and twenty-eight tokens in, ten twenty-four tokens out.

59. 10:55

    Uh, and you can see we're getting about one request per second on this guy. Um, let's just end it, and the first token comes back in four hundred milliseconds.

60. 11:03

    Let's flip it to ten twenty-four in and a hundred and twenty-eight out, right? So this is going from like a reasoning workload to like a RAG workload. Big scare quotes on that, but it's just the difference between whether you're dominated by decode time, like more of your tokens are decode or more of your tokens, uh, prefill.

61. 11:22

    Uh, and what you'll see very consistently in these, in these results is that you get much higher throughput if you have more like tokens in the context as opposed to tokens being generated.

62. 11:32

    Very straightforward. If, I mean, if you know your transformer architecture, it's like auto aggressive versus parallel. Like yeah, one of the first things you would learn if you looked at the, you know, kind of the implementation of the architecture, but it's nice to see it like nice and, you know, very cleanly.

63. 11:45

    More of an empiricist than a rationalist myself, so I like to see data, um, and not like, uh, chalkboard stuff. Um, so yeah. So, uh, what I'm getting at here is that the, um, the request per second that we're seeing here is about four requests per second for vLLM on the same workload, but n- with like context

64. 12:05

    instead of, uh, um, generation as wh- where the majority of the tokens are. Um, so this is, uh, yeah. I gave a talk on GP, like GPUs a little bit earlier today, and like one of the big takeaways there is find things that like are throughput oriented and involve a lot of arithmetic and not like moving me-

65. 12:25

    memory around or communication, and that's exactly the difference here. You have like big matrix, matrix multiplications, load the weights one time, use them a bunch, um, and that's exactly the difference here, and it's a four X improvement.

66. 12:38

    And that's using BF16, which Does not have Tensor Core support? No, no, BF16 has Tensor Core support, but it's the slow Tensor Core support compared to FP8 or FP4 on Hopper and Blackwell.

67. 12:50

    And so you're, like, you're ... The real win there is the shorter, like, shorter numbers, faster multiplication. It's actually quadratic in the bit width, so you get a big win as you go down.

68. 13:00

    So this sh- like, if we were to run some results with FP4 on Blackwells, you would see an even bigger gap than just this, like, 4X improvement. 4X is, like, barely enough to wake up for, you know?

69. 13:11

    Um, but yeah. So that's a little ... Like, not every application can you, like, change that. Um, like your users might be bringing queries to you, so you don't have control.

70. 13:21

    Um, but the, um, uh, it can ... It's more the sort of thing where you, like, a product person is like, "We should improve the quality." You're like, "Oh, how can I improve the quality without tur- like, killing our latency?"

71. 13:36

    Don't have it ... Like, don't immediately reach for reasoning, reach for context instead, 'cause it's gonna be cheaper and you're gonna get better perform ... Like, you're, you're gonna m- find it easier to hit your latency SLAs.

72. 13:46

    I forgot to point this out, but the latency is, like, almost identical in time, in time to first token, even though we're doing 10 times as many tokens. Basically a free lunch.

73. 13:55

    Um, yeah, okay, so that's, uh, that's sort of, like, how I envision people using this interface and the data. Um, there is a, there's a l- URL somewhere where you can just download the raw data.

74. 14:06

    Um, if you're interested in that, hit me up. The code is also open source if you wanna run some of these benchmarks yourself. Um, I think I've ... I'll close there.

75. 14:14

    Lots of other stuff to talk about, like our benchmarking methodology, which is written up here, the, uh, executive summary, which you can share with your, um, with your leadership, um, uh, on, like, running open models.

76. 14:27

    Um, but I'll take a question or two before we close out. Yeah.

77. 14:31

    I mean, how f- how far can you drive the, uh, max throughputs inside of ... You had it set to four.

78. 14:36

    Yeah.

79. 14:36

    Like, how far can you go out before it starts to inflect up?

80. 14:39

    Yeah, so that ... So one thing I'll say is, like, this is throughput per replica, right? So this is one GP- This is one GPU? Yeah, one H100. So, like, what you ...

81. 14:49

    The way you solve your, like, total throughput is by scaling out rather than scaling up, right? Um, but, um, so if you want, if you want 400 QPS or whatever, like, eventually you're just gonna have to scale out.

82. 15:02

    But to your question of like, yeah, how do you know, like, you know, where ... Like, why are we saying this is the highest throughput you can get?

83. 15:10

    Exactly.

84. 15:10

    Yeah. So the answer is, like, goes to our benchmarking methodology. What we do is first we dump, like, a, you know, a thousand requests and wait for them all to come back, calculate the, like, thou- uh, seconds divided, uh, requests divided by seconds, thousand divided by how long it took.

85. 15:26

    That's like a maximum throughput, right? 'Cause we gave it maxim- We exposed the maximum parallelism to the engine, so presumably they knew how to ... They were smart enough to handle that.

86. 15:35

    Gives you a maximum RPS. Any more than that, you should expect from queuing theory that the latency will blow up, right? Then the other side is, um, like, you send one request at a time, wait for it to come back, send another.

87. 15:47

    And that gives us our, like, that's the fastest you could possibly run the server, and we sweep between to get the numbers that are here. But yeah, cool. All right, I'll, uh, gotta move on to the next talk.

88. 15:56

    Uh, I'll be outside if you, uh, have any questions. Thank you very much. [audience applauds] [upbeat music]