Introduction to LLM serving with SGLang

Talk transcript

1. 0:00

   [upbeat music] Hey, everyone.

2. 0:15

   Um, so we're gonna, we're gonna go ahead and get started here. Um, we've got a, uh, nice close group here today. Um, and that's, I think, to everyone's benefit.

3. 0:25

   Um, this workshop is really for you. You know, I love the sound of my own voice. I love talking. That's why I'm a developer advocate. Um, but the, you know, the, the purpose of this workshop is to help you get comfortable with SGLang.

4. 0:37

   So if you have questions, if you have ideas, if you have bugs, uh, ask Yineng, um, or, or me, uh, and we're, we're definitely going to be able to tailor this workshop to you and your interests and what you're working on.

5. 0:52

   Um, so the title of this workshop is An Introduction to LLM Serving with SGLang. Um, we're going to be, uh, you know, talking about SGLang. And little quick introduction.

6. 1:03

   Um, so my co-speaker here, Yineng, um, is a core maintainer of SGLang. Um, has been involved with LMSYS Org for quite a while now. Um, ha- is the sort of, like, influence lead on the project.

7. 1:17

   Um, previously worked at, um, Baidu and some other places, and also is, uh, you know, an author of a few papers, um, including FlashInfer. Um, and I'm Philip, and I got a B+ in linear algebra. [laughs]

8. 1:30

   So, um, whether, you know, whether you're coming in here and you're super cracked or you're brand new at SGLang, we're gonna have something for you. Whatever your skill level, uh, this is, this is the place to be.

9. 1:41

   Um, so what are we gonna do today? We're gonna, you know, introduce SGLang, get set up a little bit. Um, we're going to talk about the history of SGLang, um, talk about deploying your first model, bunch of things you can do to optimize performance after that.

10. 1:57

    And then we're also going to talk a little bit about the SGLang community and how you can get involved, and even do a little bit of a tour of the code base in case you want to start making open source contributions.

11. 2:07

    Um, so by way of introduction, uh, let's see. What is SGLang? So SGLang is an open source fast serving framework for large language models and large vision models. Generally, you use SGLang in a sentence along with either vLLM or TensorRT-LLM.

12. 2:24

    Um, it's one of the multiple options for serving models in production. So the question is, um, why SGLang? Like, why should we, uh, you know, invest in, in learning and, and building this library?

13. 2:38

    Um, and, you know, first off, it's, it's very performant. SGLang offers excellent performance, um, on a wide variety of GPUs. It's production-ready out of the box. Um, it's got day zero support for new model releases from, uh, labs like Qwen and DeepSeek, and it's got a great community, strong open source ethos, um, which means that if something

14. 3:00

    is broken in SGLang, if you don't like something, you can fix it, uh, which is, which is pretty huge advantage.

15. 3:07

    Um, so who uses SGLang? Well, uh, we do at Baseten. Uh, we use it as part of our inference stack for a variety of different models that we run.

16. 3:17

    Um, we also, um, see SGLang being used very heavily by xAI for their Grok models, as well as a wide variety of inference providers and cloud providers and research labs, universities, and even product companies like Cosa.

17. 3:34

    So quick history of SGLang. Um, it's honestly really impressive to me how quickly this project has come up and gotten big. Um, if you look at, you know, the archive paper was released in December 2023.

18. 3:46

    That's 18 months ago. So in just 18 months, this project has gone from a paper to 15,000 GitHub stars almost. You should all go star it so that we can get a little closer.

19. 3:57

    Um, and it's, uh, you know, supporting all of those logos, all those companies we saw on the last slide. Um, it's got a growing and vibrant community. Um, it's got international adoption.

20. 4:09

    So yeah, incredibly impressive what the team has done in that time. Um, and I'm going to turn over to Yineng now to talk a little bit more about that history and also, like, how you got involved in the project.

21. 4:20

    Okay. Uh, hello, I'm Yineng. Uh, I'm the core developer of the SGLang project, and I'm also the software engineer at Baseten. And, uh, before I joined Baseten, I work as ma- I worked at, at Meituan and at that time, I work for the internal, uh, click-through rate ranking model optimization and the LLM inference optimization.

22. 4:42

    And at that time, the creator of SGLang, named Lian Min, just reached out and then we, we have, uh, yeah, Google Meet. So at that time,

23. 4:53

    I... When I left Meituan, I joined the SGLang project. So I worked closely with Lian Min and Yin on SGLang. Also, you know, SGLang use FlashInfer heavily because Fla- w- we use FlashInfer as the attention kernel library and the sampling kernel library.

24. 5:09

    So I also worked with Zihao, uh, on the FlashInfer project. And, uh, yeah, cur- currently, I'm the core maintainer of the SGLang project, and I'm also the team member at LMSYS Org.

25. 5:22

    And that's the, uh, little point of trivia. That's the same LMSYS Org that just got $100 million to build Chat Bottaweena, uh, from a16z. Um, I, I learned that while I was putting together the slides for this talk.

26. 5:35

    So, um, if you were here early, you were able to, um, scan this QR code and get everything set up for the workshop. Um, if not, uh, definitely grab that right now.

27. 5:46

    Um, you've got the QR code, you've got the URL that takes you to the same place. Does anyone still need the QR code? Um, okay, I've got a couple people still.

28. 5:57

    All right. Anyone still need the QR code? Going once, going twice. Yep.

29. 6:18

    To, uh, folks watching at home, you've got this great button on YouTube, uh, it's called the fast-forward button, so you can just skip this part. [laughs]

30. 6:35

    All right, we, uh... Looking good. Um, if you, uh, if you need this again, uh, just, just let me know. I'll, uh, throw it back up there. So we're going to talk about, um, how to deploy your first model, um, on SGLang.

31. 6:52

    Um, so if you, um, go over to the GitHub...

32. 7:00

    Yes. So, um, in this step, we're just gonna get familiar with the, uh, basic mechanics of SGLang. Um, SGLang is basically just like a SoVo command, um, that you're going to run in your Docker container.

33. 7:23

    There's a little bit of sort of difference, uh, with using it the way we're gonna use it in the workshop right now versus how you might use it if you're working directly on a GPU.

34. 7:32

    The difference is you're using something called Truss to package it. Basically, you're putting in your SGLang, uh, dependencies and your command into this YAML file, you're bundling it, and you're shipping it up to a GPU.

35. 7:47

    Uh, the reason we are using Truss is because that is the way that you can get on Baseten, and the reason you are using Baseten is because that is the only company on Earth that will give me free GPUs because I work there.

36. 7:58

    Um, so we're, uh, we're gonna be working, um, on all these examples on L4 GPUs, uh, because they are cheap and abundant, um, and they also support FP8. Uh, but the same, uh, the same product works on H100, um, H200, and Blackwell's coming soon.

37. 8:16

    Yeah, yeah.

38. 8:17

    Yeah, coming soon. Um, so yeah, it's gonna be basically, like, the same principles. Um, if you go through here, um, the, uh, the configuration, um, you can actually... In your Truss config, you can change the, um, hardware type to H100 if you want, um, and, uh, in the, yeah, in the, uh, accelerator line right there.

39. 8:42

    Um, but yeah. So what is, like, the actual SGLang launch SoVo command, um, that we're, that we're running here? So it's basically just, like, a bunch of flags. That's the thing to understand about using SGLang.

40. 8:56

    It's all about knowing what flags are available, knowing what configuration options are available, knowing the support matrix that exists for them, and knowing how they interact with each other.

41. 9:07

    Um, if you, you know, turn on a major speculation algorithm and then you also jack your batch size way up, well, that's probably not going to go so well for you.

42. 9:16

    Um, but if you want to do, say, like, your, you know, quantization along with some of these other optimizations, those play nice. Um, so yeah, um, what we're gonna do, um, this is the fun part of leading a workshop, um, is the part where we just, like, stand around watching you type.

43. 9:33

    Um, what we're gonna do is give everyone, uh, about five minutes to work through this first example. Um, we're gonna circulate the room if you have any questions, um, and then we're gonna come back together after, uh, running the first example.

44. 9:48

    Sound good? All right, let's do it. Uh, can you cut the mics for five minutes?

45. 9:57

    Pause, skip. It's, it's great. They... These buttons, they're magical.

46. 10:06

    Has issues... Is anyone having issues where you're, like,

47. 10:11

    stuck trying to get into Baseten? Um, you're in, like, a waiting room and it won't let you out. Um, if you are a dev- if you are, uh, flag me.

48. 10:19

    Um, if anyone is having issues where you're getting, like, an error in your code, please don't show me, show him.

49. 10:29

    And to check on progress, has anyone managed to get the first model deployed and running?

50. 10:36

    Deploying.

51. 10:36

    It's deploying. Awesome. Let's hope it's deploying really fast.

52. 10:42

    Let me, let me take a look here.

53. 10:45

    All right. Sounds good. Can you, uh, take a look at the logs for me real quick?

54. 10:52

    Wow. Our, our Wi-Fi's just amazing here. I promise Baseten is usually faster than this.

55. 11:05

    Oh, okay. Well, it looks like it, it came up. Um-

56. 11:08

    Oh, okay

57. 11:08

    ... so you can, um-

58. 11:11

    Okay

59. 11:11

    ... you can use the, um, sample code, um, in call.py, oh, call.ipynb, um, or, like, you can just use an ordinary OpenAI client. Um,

60. 11:27

    what you need to call it, if you go back to your Baseten workspace with the model, um,

61. 11:33

    what you need is, uh, scroll back up a little bit for me. You need that model ID. That's what's going to, um, unlock your calling code.

62. 11:42

    Got it.

63. 11:44

    Um, that, yeah, paste it in right there. Um, you'll, you'll need to run an act- run an actual Jupyter notebook to, to run that.

64. 11:55

    All right. We've had our first successful deploy.

65. 12:00

    If you wanna call it using the OpenAI SDK, using the call.ipynb, uh, notebook, um, this thing up here, it's gonna be different for everyone. Um, this within the UI is your model ID that you use to put, uh, set up the URL

66. 12:17

    Um, hey, everyone. We're gonna come back together here. Um, it's about nine forty-five, um, so we're going to move on to the next stage of the workshop, where Yineng is gonna do some really awesome demos.

67. 12:28

    Um, if you are still getting everything set up, uh, no worries. All this stuff is gonna be live on, um, GitHub. You've... Oh, sorry. Yeah, on GitHub, um, the, the repository with the workshop information is gonna stay up.

68. 12:40

    Um, so you can keep following along. Um, this is also all gonna be published, um, so it's gonna be easy to go back if you have any issues. Um, anyway, so the next thing that we're gonna look at, um, now that we have a sort of basic idea of, okay, SGLang is just, like, running a model server,

69. 12:59

    uh, how are we going to actually make it fast? Um, so Yineng's going to show, um, one demo, which is the, um, CUDA, um... What is it? What's the-

70. 13:10

    CUDA graph.

71. 13:11

    Yes, CUDA graph match max BS flag, um, and how to set that to improve performance. Um, and then we're also going to take a look at EAGLE-3, which is a new speculative decoding algorithm, which, uh, also can improve performance.

72. 13:26

    Okay. Okay.

73. 13:26

    So take it away, Yineng.

74. 13:28

    Yeah. Uh, can, can you see my screen? Maybe I can-

75. 13:33

    Yes. Yeah. Good, good call. Zoom it in a little bit.

76. 13:39

    Um, we're using OnePod because, uh, on Baseten you don't get SSH access into your GPUs because, uh, security or something, I guess. I don't know.

77. 13:49

    Okay. So here, I will use the L4 GPU. Yeah, this is the L4 GPU, and I have already installed the SGLang. Yeah, we, we can just use the pip install or install from source.

78. 14:05

    And, uh, here is the, the, this command line. Uh, sorry. We launch the server, and we use the Llama three point one eight B instruct model.

79. 14:21

    And, uh, the attention backend use FA3. This is the default. And when we... [clears throat]

80. 14:45

    Okay. It, it has started to lo-loading the weights.

81. 14:49

    So, uh, just to, just to give everyone a little bit of context, um, the top window you're seeing here is the, um, L4 that's actually running the SGLang server.

82. 15:01

    The bottom window here, um, LM Eval is a sort of industry standard benchmarking tool, um, that we're just gonna use to throw a bunch of traffic at the running server.

83. 15:10

    Yeah, yeah. For sure. And, uh, yeah, we can see the, the log from, from the server. Uh, it shows that we capture CUDA graph batch size. I think a CUDA graph is turned on by default.

84. 15:22

    But the CUDA graph max batch size for L4, for this model, is eight. So it only capture one, two, four, eight. And, uh, okay, the server is ready to roll, and we can use the LM Evo to send a request.

85. 15:42

    Yeah, we can see that from, from the log. Here is the prefill batch, and here is the decoder batch. And, uh, we can see, uh,

86. 15:54

    under the decoder batch, when the running request is ten, it means that there are 10 running request. And the CUDA graph is false because the running request time is larger than the max CUDA graph batch size eight.

87. 16:09

    That's why this one, this flag is false. And when this is false, uh, we get, uh, one hundred fifty-five generation token per second. And we can use this one, divide it t- divide it ten.

88. 16:24

    So I think produce nearly fifteen, uh, token per second.

89. 16:30

    Okay. We can kill the cl-client, and we can also kill the server.

90. 16:42

    So yeah, we, we can use this command li- as a base and, uh, set the CUDA graph max BS.

91. 16:54

    CUDA gra... For example, we can just set, uh, thirty-two.

92. 17:02

    You, you, you've got a, you've got a typo in there.

93. 17:05

    Um, oh, sorry.

94. 17:07

    Yep.

95. 17:13

    The network is not good.

96. 17:16

    Here everyone is learning a very important lesson in the value of latency.

97. 17:21

    Okay. Yeah. Yeah, it's loading weights. Yeah, and we can see that, uh, after we set the max CUDA graph batch size, the capture CUDA graph BS, uh, I

98. 17:46

    think that the max is, uh, thirty-two is larger than the eight. And the server is ready to roll. We also use the LM Evo to send a request.

99. 18:09

    Okay. So first is the prefill batch, and then we can... Here is the decoder batch. Oh, okay. And, uh, yeah, here is the decoder batch. We can s-

100. 18:26

     Oh, wait for a moment. Decode batch.

101. 19:03

     Yeah, for example, here the, the decode batch and, uh, there are 13 running requests, and the CUDA graph is true, and here is the generation throughput.

102. 19:17

     And I think per user should be 12, and we can compare with before.

103. 19:31

     Oh, it's, it's not easy to compare. Uh, yeah, yeah. I, I, I think, uh,

104. 19:38

     we have recording this video, and we can also see here CUDA graph, and we upload this one, cudaGraphMaxBatchSize demo.

105. 19:51

     We, we want the CUDA graph to be true during decode because I think this is very important for the decoding performance, uh, but the default max batch size is eight on L4.

106. 20:03

     And when we used LMe4 to send a request, we found out that, oh, the max batch size is larger than eight. That's why we want to set or adjust the, the parameter.

107. 20:13

     Here, when we set it to 32, uh, we can handle the realistic batch during benchmark.

108. 20:21

     Yeah. Do you have any questions? Where were the commands to run this?

109. 20:28

     Oh, okay. The LL, LMe4? Mm-hmm. Yeah, yeah, yeah. I think LMe4 is a evalua- evaluation tool, and we need to specify the model, and, uh, here is the model name, here is the URL.

110. 20:45

     Because I, I just used Runpod to run this, and it used the same node, so that's why the URL is the local host. And we, we specify the port, this one, 8000.

111. 20:59

     That's why I will use 8000. And we use the OpenAI compatible server, and, uh, here, the number concurrent or the, the batch size is, uh, 128. We set the max generation tokens.

112. 21:13

     We just used GSM8K. I think it's a classical, uh, evaluation data set. And, uh, because we use the chat completion API interface, that's why we need to apply chat complete, and I just used few shot eight.

113. 21:27

     The limit means that because, you know, uh, the GSM8K, it has, uh, 1,319

114. 21:37

     promotes, and when we use, uh, the limit

115. 21:41

     0.15, I think it's nearly, uh, 200 promotes. I can share, I can also share this, this command line in the, in the repo. Yes. Yeah, yeah. Maybe I can add it.

116. 21:58

     Oh, sorry.

117. 22:02

     Yeah. So just to, just to be clear, um, this command is running on the actual GPU itself.

118. 22:09

     Yeah.

119. 22:10

     Um, so this is for when you have SSH access into the GPU you're running. Um, on a, on the service we're all using on the, the Baseten GPUs, you can't SSH in.

120. 22:20

     Um, but if you do have the access to a GPU where you can get SSH access, then you would use this, um, LM, uh, eval tool, um, in order to simulate that traffic.

121. 22:34

     Yeah.

122. 22:34

     Um, if you're using a more, like, standard HTTP connection, um, to a, you know, remote GPU, then you would use a, a different benchmarking tool, um, that's, you know, request-based.

123. 22:47

     Yeah. Okay. And, uh, do you have any other questions for CUDA graph?

124. 22:55

     Why is default eight selected?

125. 22:57

     Yeah, yeah. I, I think the default eight is because the L4 GPU VRAM. And we, we have, uh, some default configuration. Uh, we will, yeah, set the ... When, when, when you didn't set the CUDA graph max batch size, the default value is none.

126. 23:16

     And when the default value is none, we will set internally for, for specific hardware, for specific model. Yeah. For, for example, it- it's TP1, and it's on L4, so the, uh, default is, is just eight.

127. 23:31

     Yeah.

128. 23:32

     So what if someone, by mistake, right, he adds a higher one for a large model or something default?

129. 23:38

     Yeah, you can just try that, because when you launch the server, you can see the, the, uh, startup parameters, and then, well, you, you have a workload, right? And you use the LMe4 to benchmark, for example, and you can analyze the server log.

130. 23:55

     And you find out that, oh, during the decoding, the CUDA graph is disabled. And we, actually, we want to enable CUDA graph. That's, that's why we increased the max CUDA graph batch size.

131. 24:06

     Yeah. Okay.

132. 24:08

     Awesome. Um, so let's, um, let's see. Do you wanna show the Eagle stuff, or do you wanna show the code-based stuff?

133. 24:15

     Yeah, yeah, yeah, yeah. Mm-hmm. Okay. Uh, I think the, the next very important is about the, the Eagle stuff.

134. 24:23

     Yeah.

135. 24:23

     Yeah.

136. 24:23

     So Eagle3 is a speculative decoding framework. It came out very recently, right?

137. 24:28

     Yeah, yeah.

138. 24:29

     The paper was released a few months ago. Um, and so SGLang supports Eagle3, um, and, uh, with it you can configure a wide different, a wide variety of, um, different parameters around how many tokens you're speculating, how deep you're speculating, that kind of stuff.

139. 24:48

     Um, and Eagle3, um, can have much higher acceptance, token acceptance rate. Um, so obviously when you're speculating, the higher your token acceptance rate, the better performance you're going to get.

140. 25:00

     So we can take a quick look at some of those parameters that you showed, um, and then maybe, uh, the benchmark script you were showing me the other day.

141. 25:08

     Yeah, yeah, yeah. I, I think for the Eagle3, you ... Yeah, we, we also provided the, the example. We can just, uh, yeah, change directory to, to this directory and then use trust push.

142. 25:20

     I- it's very easy. I just want to explain, uh, some details. For example, we need to specify the speculative decoding algorithm. Here is the Eagle, like this one.

143. 25:32

     Uh, yeah, we need to specify, uh, speculative decoding algorithm Eagle, and we also need to specify the draft model path because, uh,

144. 25:44

     this one, the model path, this is the target model, and here is the draft model.

145. 25:51

     I ... Sorry. Here is the draft model for the Eagle3,

146. 25:56

     yeah, Llama, Llama 3, 8B.

147. 25:59

     So one thing that's different about Eagle, um, all the different Eagle algorithms is instead of like a standard draft target where you're, say, maybe using Llama 1B and Llama 8B together, um, Eagle works by qu- pulling in multiple layers, um, of the target model, using that to build a draft model.

148. 26:19

     Um, so the draft model is kind of derived directly from the target model versus being just a smaller model that you're also running.

149. 26:26

     Yeah, yeah. And you also need to specify, uh, this parameter, the number of steps, the Eagle top K and draft, uh, verified tokens. For example, uh, the depth of the drafting if it's three and the top K is one, I think the, the most number draft tokens should not more than four.

150. 26:49

     That's why we said four here. And, yeah, you can see more de- details about this configuration at the SGLang official documentation. And I also will share, show something about how to tune in these parameters.

151. 27:04

     You know, we have these parameters. I think the model path and, yeah, is fixed and the, how about this one, the number of steps, top K, and the number of draft tokens, we can tune in these parameters, and I will show you how to tune in that.

152. 27:19

     So in the SGLang main repo, we have a script and, uh,

153. 27:26

     we have a playground. Yeah, we have a bench speculative decoding.

154. 27:33

     Okay. So we can just use, uh, this script to tune in these three parameters. For example, on a single GPO, or when we want to u- This, this is the target model, Llama 2 7B, and this is the, uh, draft model.

155. 27:48

     Here is some default parameters. The batch size is from one, two, four, eight, 16, and the steps i- is here. Zero, one, three, seven, five, seven, and the top K is here.

156. 28:03

     This is the number of draft tokens. What does that mean? I think it's, it's very easy to understand. For example, we have different combinations of these different parameters, and this script will run all of these combinations, and you will get as a result.

157. 28:19

     And from the result you will get a, you, you will get to know that, oh, this, for example, this combination is, is best. For example, the, uh, at the batch size eight, uh, the three steps maybe, and the top K is one, and the number of draft tokens is four.

158. 28:36

     You will get some result h- about the, the, the speed and about the accept rate. Then you can use this parameter for your online serving, for your production serving.

159. 28:47

     Yeah.

160. 28:48

     Yep. And when you're running this benchmark, uh, do be sure to set the prompts to things that are representative of your actual workload.

161. 28:55

     Yeah, yeah.

162. 28:56

     Because speculation, um, in any format, including Eagle, is all about guessing future tokens, uh, if you are benchmarking on data that is not representative of your actual inputs and outputs, um, that you're seeing live in production, then you're probably gonna end up with the wrong parameters.

163. 29:13

     Um, speculation is a very topic and content-dependent, uh, optimization.

164. 29:18

     Yeah, yeah, I think so. So you, you can also update these promotes. Uh, here in this bench spec- bench speculative decoding, uh, uh, py- Python script, we have some promotes, and I think you can update these, yeah, just according your needs.

165. 29:35

     Yep.

166. 29:35

     Okay.

167. 29:36

     So let's, uh, let's take a look at some of the, um, stuff around, you know, the community and getting involved.

168. 29:42

     Yeah, yeah, yeah.

169. 29:42

     Yeah.

170. 29:43

     Also, I, I think, yeah, SGLang currently i- i- is become very popular, and if you want to participate in this community and contribute some code, I think, uh-

171. 29:56

     Yeah, I'll sh- I'll show the, the slides real quick.

172. 29:58

     Okay.

173. 30:02

     Yeah, so, um, you know, S- SGLang does have a really great community. Um, and, uh, you know, some, some quick ways to get involved, um, you can start on GitHub, file issues and bug reports as you build.

174. 30:14

     Um, they have a great tagging system of post issues to get involved with, which Yineng's gonna show in a second. Um, but the number one thing you can do is follow [REDACTED:username] on Twitter, um, and then join the Slack, um, to keep an eye out for online and in-person meetups.

175. 30:31

     Um, so this is a link to the community Slack Um, you can, uh, scan that real quick if you, uh, if you wanna get involved with SGLang. Um, these slides are also all in the, um...

176. 30:43

     these slides are all in the repo, um, that you got from the workshop, so you can access this, uh, this link and stuff later. It's also just slack.sglang.ai. Pretty simple link.

177. 30:54

     Um, so if you are going to get involved and you do want to, um, you know, start contributing to the code base, um, we can kind of show you, um, some of the stuff.

178. 31:06

     So at a high level, um, the code base has the SGLang runtime. Um, it's got a domain-specific front-end language, and it has a set of optimized kernels. Um, you can go actually on this DeepWiki page, um, and get a really good co-tour of the code base, um, as well as a tour from, um, this, uh, other repository

179. 31:30

     that we have linked, um, which is also by one of the SGLang people, um, with some, some diagrams about, like, exactly how this stuff works. Um, and then, yeah, Yineng's just gonna show a quick overview of the code base on GitHub, um, in case you're interested in getting involved and contributing.

180. 31:47

     Yeah. I, I think that the best way to getting involved in this project, uh, first we need to use that, and then you will find out some issue or, or you will find out some feature are missing in this repo.

181. 31:59

     And then the first thing that is, uh, you can raise a new issue here.

182. 32:05

     Oh, it's loading. [laughs] Yeah, you can just, uh, create a new issue, feature request, or something like this. And also, I, I think, yeah, we have labeled something like,

183. 32:17

     "Good first issue" or "Help wanted." Yeah, you can see that there are nearly, uh,

184. 32:30

     twen-twenty-six. So I think, yeah, if you are interested in, in this issue, for example, if you are interested in, uh, support or serving VLM, VLA model, uh, you, you can just start with this.

185. 32:44

     I think, uh, good first issue and help wanted issue, yeah, we are welcome the contributions. And here is the development roadmap.

186. 32:56

     So yeah, i-if some feature is missing or if, if some feature you care about, you, you, you can find it in the roadmap. I think you can, uh, join us for, for this feature development.

187. 33:10

     Or you can also, yeah, raise a new issue about this. And the last one is about, uh, overall walkthrough. Okay. So yeah.

188. 33:27

     In the SGLang repo, uh, we, we have some component. This one is the SGL kernel. It's a SGLang kernel library. Uh, we implement attention, normalization, activation, gem of them in this kernel library.

189. 33:42

     And if you are familiar with CUDA kernels and if you're interested, yeah, with kernel programming, you can just contribute at this part. Now, here is the SGL router. Last year we published, uh, SGLang, the three version, and we supported the cache-aware routing.

190. 34:01

     If you, yeah, you are interested in this part, you can work on the SGL router.

191. 34:09

     Currently, we, we use SGLang as a LLM inference runtime. So I think the Python part, the SRT, is the core part. We support desegre- PDD desegregation. We support the concentrated decoding.

192. 34:24

     We support function calling. Yeah. We support OpenAI compatible server, and we also support a lot of models. If, yeah, I think if you want to support the custom model, you can just, uh, yeah, take this as a reference.

193. 34:40

     For example, you can take Llama as a reference. I, I think, uh, the popular open source model, the architecture i-is very, very similar. So if, if the model you are interested has not been implemented in the SGLang, you can just take this as reference and do some modification, and then we welcome contributions.

194. 35:01

     Yeah. That's all.

195. 35:03

     Awesome. So, um, if we get the slides back up here. Um,

196. 35:09

     yeah, so to, uh, you know, wrap it up, um, first off, thank you so much for coming out. Thank you for bearing with us. Thank you for waiting for webpages to load on this, uh, wonderful, uh, internet connection that we all have.

197. 35:23

     Um, to kind of wrap things up, um, I do wanna issue a couple invitations to everyone in this room today. Uh, number one, we're having a really cool, uh, happy hour with the folks from Oxen AI.

198. 35:35

     Um, Oxen AI is a fine-tuning company. Um, their CEO just had a really cool demo that he published a couple weeks ago, where he took GPT 4.1, um, and made it, you know, do a SQL generation benchmark.

199. 35:49

     Took the score, said, "Okay, I think I can do better than this." Took Qwen 0.6B, yes, you heard me right, less than a billion parameters, fine-tuned it on some SQL generation data, and actually beat GPT 4.1 with a model that you can run on, like, three years ago iPhone.

200. 36:05

     Um, so yeah, we're gonna be, uh, you know, at this happy hour. We're gonna be talking about fine-tuning and stuff. It's gonna be a great time. Um, second invitation I want to extend to everyone in this room is if you think this stuff is cool, if you are, you know, seeing all the stuff that Yineng was talking

201. 36:19

     about around contributing to the code base, and you're like, "Yeah, I love CUDA programming," um, just come work at Baseten. Uh, if you're bored in your job, you won't be bored here.

202. 36:28

     Uh, we've got a lot of open roles for both infrastructure and for model performance. Uh, if you're at all interested, just come talk to me. I'm gonna be here all, uh, all three days.

203. 36:38

     Um, so yeah, that's pretty much our, um, workshop today. Thank you so much for coming through, um, and happy to take any questions in the, uh, remaining time we have.

204. 36:48

     Yes. Yeah, that's a great question. Um, you know, I think that, uh, what we... At Baseten, like we use all sorts of different runtimes, um, model to model. Um, sometimes you just wanna use whatever one is best for your use case.

205. 37:07

     Um, but in general, uh, I think that the reason that we've been really attracted to it is because of how configurable and extensible it is. Out of the box with basic parameters, you're gonna get more or less the same performance from anyone.

206. 37:22

     Um, but if you're able to, number one, have like a really deeply and well-documented code base like SGLang, where you're able to really deeply understand all the different options that you have, um, that can get you a long way.

207. 37:35

     And then as we were just talking about, it's super easy to contribute, um, so we're constantly like making fixes and, and contributing them back. Um, and that means that, you know, if you're using a, a different library, you might be blocked waiting for the core developers to implement support for a model or something.

208. 37:52

     SGLang you can unblock yourself. Yes.

209. 37:57

     When there are multiple vendors and different kind of applications around the endpoint or within the subnet you are defining, how would you define your, um, cybersecurity or security protocols?

210. 38:11

     How would you enhance your protocols?

211. 38:14

     Yeah. I mean, that's a great question. I don't really think that your, your choice of runtime engine like affects that too much, um, because you- you're just packaging it up in a, in a container.

212. 38:26

     Um, you know, within, within Baseten, we've thought a lot about this in a sort of runtime agnostic way, um, where we're thinking about of course like least privilege, um, we're thinking about, you know, making sure that there's a good deal of isolation built into the system.

213. 38:42

     Um, but from a, from a runtime perspective, I don't think there's anything special we have to do for security with SGLang, right?

214. 38:48

     Yeah.

215. 38:48

     Compared to like vLLM or anything else.

216. 38:52

     Thank you. So, um, I, I'm from, uh, Department of Defense and-

217. 38:58

     Oh, awesome

218. 38:59

     ... so extensive experience in-

219. 39:01

     Yeah

220. 39:01

     ... financial applications. So to, uh, do some,

221. 39:07

     uh, uh, product developments in house, um, do you think I have... I can do the entire product development in-house within a subnet, don't have to go back and forth open?

222. 39:21

     Right now, for example, I'm just throwing an example.

223. 39:23

     Yeah.

224. 39:24

     One of those, uh, uh, CMMC cybersecurity certifications, I have to go through the endpoint controls and define the endpoint control and then go connect to the ChatGPT, OpenAI, ChatGPT-

225. 39:42

     Gotcha

226. 39:42

     ... kind of thing.

227. 39:42

     Yeah. Yeah. So in that case, this would actually help you out a lot. Um, instead of relying on that remote server, um, you can just spin up a cluster like within the same, uh, VPC or like within the same physical data center, um, as the workload that's relying on the AI model.

228. 40:01

     Um, you can clone SGLang, you can cut, you know, take a release, um, and fully inspect the code because it's open source, and then fix, um, on that release so that there's nothing changing under the hood.

229. 40:15

     Um, and then, yeah, with that you'd be able to, you know, run the models just directly on the GPU, as you saw in Yineng's demo when he was doing the, um, CUDA graph stuff.

230. 40:25

     Um, you're able to, you know, call it on even a local host basis and run inference. Um, so yeah, it gives you all the tools you need if you're trying to build even like a sorta AO gapped type of system, um, with, you know, all of these, uh, open source runtimes.

231. 40:42

     You can pull that code in, inspect it, lock it, um, and then, uh, build off of it.

232. 40:48

     Very impressive. And also, um, currently I'm working on, uh... I'm also a PhD student.

233. 40:55

     Yeah.

234. 40:56

     So I'm working on blockchain-based, uh, quantum computing and some kind of, uh, AI deliverables. So how do you

235. 41:07

     circumvent within your product? Can you... So blockchain is completely another community based-

236. 41:12

     Yeah

237. 41:13

     ... code development. So how do you inte- can we integrate different community based or, uh, a combination of, uh, both hybrid community based protocol? Or so what is, uh...

238. 41:26

     Because blockchain is kind of, uh, um, decentralized network, whereas this one is kind of within a container.

239. 41:36

     Yeah. Um, to be perfectly honest, like I haven't really experienced anything with that. Um, pretty much all of, uh, the use cases that I've run with SGLang are just traditional client server applications.

240. 41:50

     Any other questions? Yeah.

241. 41:54

     vLLM has a inference engine anywhere in Baseten, and if so, for what kind of workloads and how is like SGLang different than vLLM? Like what parts of SGLang shines and I...

242. 42:04

     I mean, you shared something on the 100 million AP aspect, but-

243. 42:08

     Yeah

244. 42:09

     ...

245. 42:09

     Great. Um, so yeah. So in Baseten, like what we do is we, we call it like the Baseten inference stack, where we're taking all of these different, um, all of these different providers, the, the vLLM, the SGLang, and the TensorRT-LLM, which we actually probably use the most heavily of the three, um, and taking them in, customizing them,

246. 42:29

     doing all that stuff I'm supposed to say for marketing purposes. Um, but we are customizing it quite a bit. Um, anyway, where we generally pick vLLM, um, I'm sorry, I'm talking about, uh, I'm talking about them doing a SGLang talk, um, but where we use vLLM is oftentimes for compatibility.

247. 42:46

     Um, for example, like I know our Gemma, uh, models that we have up in the library are using vLLM, um, because like it's what was supported, uh, when, when it dropped.

248. 42:57

     Um, so yeah, that's, that's, in my mind, like the best use case for vLLM is like super broad compatibility.

249. 43:06

     Any other questions? Awesome. Well, uh, like I said, we're gonna be around all day. Um, and um, I'm gonna be, uh, at the Baseten booth, uh, for the next three days.

250. 43:19

     So if you have any questions about SGLang, model serving, model influence in general, um, or if you want one of them jobs I was talking about, we are hiring very aggressively.

251. 43:29

     Uh, so definitely stop by the booth, hang out, uh, grab one of these shirts. Um, and yeah, thank you so much for coming. [upbeat music]