Under 5 minutes to a deployed LLM endpoint — Audry Hsu, RunPod

Talk transcript

1. 0:00

   [upbeat music] Audry.

2. 0:15

   Um, I am from RunPod. Um, this is an intro to RunPod. Can I just get a quick hands to see how many people have already heard of RunPod or maybe even used RunPod before?

3. 0:27

   Okay, newbies for everybody. Great. Um, so RunPod, we are a cloud AI infrastructure company. So we have the hardware, we have the GPUs, and we make it easy for developers to deploy, um, models.

4. 0:43

   And that can be your own private model, it can be an open source model from Hugging Face. Doesn't matter to us. You bring your code, and we'll bring the rest.

5. 0:54

   Um, just really quickly, what problems does RunPod solve? Like, why are we even here today? Um, infrastructure can be hard managing it. Um, [sighs]

6. 1:05

   I think about back in the day before we had AWS, um, Google Cloud, when everybody would have to have on-prem servers and manage those, maintain those. That is something that we don't want to have to do as developers.

7. 1:18

   Those are things that we happily, um, have moved away from and given off to DevOps, and now it's even, it's even more abstracted for us. GPU access is slow and opaque.

8. 1:30

   So, um, I don't know if you-- if anybody has tried to buy a GPU recently. We're in a global supply crunch. Um, it's a bit like in COVID when everybody went to the store and bought all the toilet paper because we didn't know how, how long they would need to be at home for.

9. 1:48

   We're a little bit in that right now. Um, but we expect the market will recover, um, as customers, companies, people figure out a little bit better-- uh, get a little bit better at, um, estimating what kind of compute they need.

10. 2:01

    Um, and then last, builder primary focus should be building. So again, um, we wanna build app... We as so-software developers, uh, we bring, bring the value through the applications that we build, um, not for managing the infrastructure.

11. 2:19

    And I think RunPod has a pretty unique story. These are, these are our founders, Zen and Pardeep. Um, so they had a couple of GPU rigs in their basement in 2022.

12. 2:30

    Um, failed crypto mining, and then so they were like, "What are we gonna do with our GPUs now?" Um, so they prototyped what is now the foundations of RunPod.

13. 2:40

    They posted on Reddit and said, "Hey, anyone wanna use these GPUs for free? Just give us feedback on it." And that is literally how our company has started, and we have been, um, revenue generating ever since.

14. 2:56

    Um, and the reason why I wanna tell this story is, um, not because it's, it's very, like, bootstrappy, but because, um, the origin, origin story of RunPod has always started with, uh, builders and getting feedback from the community, and that is still true today.

15. 3:13

    So I won't promise that we'll be perfect, but, um, we are definitely very engaged with, um, our users on, on Reddit. Um, on Reddit, on Discord. So, um, we're always trying to stay engaged with y'all.

16. 3:26

    Um, just at a glance, to give you an idea of RunPod, we have over five hundred thousand developers on our platforms, thirty-plus data centers across the world, including, um, Europe and the EU.

17. 3:40

    Um, and we've just passed a significant revenue milestone for us, a hundred and twenty million in annual recurring revenue.

18. 3:50

    Uh, these are just a few of our customers. Um, you might be surprised to see some of the AI cloud-native companies on here too, but, um, they come to us for the same reasons that most of our customers come to us.

19. 4:03

    It's, um, because they need flexible and reliable GPU infrastructure.

20. 4:12

    This is a really high over-- high-level overview of, um, different ways you can build on RunPod. So I would say our... at our core, um, Pods, it's our sandbox virtual environment.

21. 4:26

    We spin up a container for you, um, allocate GPUs to it, and we manage the rest. You just bring your, um, Docker files. You bring your code. Serverless, um, it's our auto-scaling product.

22. 4:42

    So when you're thinking more about, like, bursty workloads or batch workloads, um, Serverless is really great because, um, instead of being always on like a container is, Serverless, um, your workers spin down, and when they're idle, you don't pay for anything.

23. 4:59

    Clusters, um, if you're doing some heavy duty training, there's a place for you as well on RunPod. Um, multi-node clusters with high-speed networking. And then the Hub, which I'll, I'll, I'll switch to in a second.

24. 5:12

    Um, it's kind of like our central repository for AI repos. Um, these are already pre-configured, pre-vetted. Um, we have a couple of examples of listings by RunPod for popular models, but also our community con- um, contributes to them as well.

25. 5:29

    So they're just repos that you can fork, you can watch, and then, um, you can star and deploy on RunPod.

26. 5:39

    Um, so today we're gonna be talking mostly about Serverless. Um, so Serverless is best for real-time inference. I talked about the auto-scaling that comes with it. Um, why teams use it is mostly because they don't need to, um, preempt and figure out how much compute they need ahead of time.

27. 6:01

    Um- You can set, you can configure the number of max workers that you wanna scale up to. You can set limits for caps, for spending caps, and you can also configure workers that are always on, so they're, um, already have your models downloaded, and they can respond to requests, um, immediately.

28. 6:19

    For a lot of teams, serverless is the fastest way, um, if you want to start deploying a production-ready API.

29. 6:30

    And now I'm going to switch over and just show y'all really quick how easy it is to get started and deploy something.

30. 6:45

    Okay. Where are we? [clears throat] Okay, so right now I'm, um, gonna do everything via the console so that it's nice and pretty for you guys to see.

31. 7:03

    But we also have, um, CLI support. We have skills, um, to help work with RunPod, everything that's ready for your agent. So you don't have to read our documents, but since we're all humans here today, I'm gonna show you via the console.

32. 7:19

    Um, we'll start in the Hub, which is... If you're just trying to explore and see what's out there, what is something that you can get up and running right now, the Hub is a great place to start.

33. 7:32

    So like I mentioned, these are already vetted open source listings for, um, AI repos, and I am going to pick vllm. Um, and I'll just open the underlying

34. 7:48

    repository as well, so you c- guys can see what... It is literally just a GitHub, um, repo.

35. 7:56

    It tells you how to get set up for it. Um,

36. 8:01

    we can see there's already the Docker file here. It's already pre-configured for you. It's got some defaults for you. Um, depending on the listing, you can, uh, pass in different environmental variables,

37. 8:15

    um, to configure it how you wish. But I'm just gonna go ahead and click deploy, and

38. 8:28

    I have a model that I wanted. I've...

39. 8:37

    Let me see. I was gonna just pick Gwen.

40. 8:44

    Works well. This is gonna download it from Hugging Face, and just expand the advanced options, and look for the max model length. And I'm gonna bump this up for the context window and leave everything else as the defaults.

41. 9:02

    But there's settings for Max LoRAs. Um, all of these configuration options get passed as, um,

42. 9:11

    as flags to the, uh, vllm serve. And

43. 9:19

    I'm gonna spin it up as an endpoint here.

44. 9:25

    So this might take a minute since... or two, since this is the very... I just created it. We've gotta initialize my workers. Let's check out.

45. 9:35

    So the default configuration here is it's gonna deploy on some H100s, and A100s are the backup here. I have my pricing. This is l- a fraction of a cent per second.

46. 9:48

    Um, like I mentioned before, that this is only going to be charged for while the worker is actually running and handling a request. Max workers is where I can bump this up if I want to have

47. 10:02

    my workload scale up, up to 15 workers at a time. And I can set, [smacks lips] um, some active workers, ones that I want always to be on, that I don't want, um, the container to ever spin down.

48. 10:16

    And I can save that. Okay, so how do I in- how does one interact with the serverless endpoint? Um,

49. 10:27

    this is just an API, uh, HTTP endpoint right here. Um, we provisioned this endpoint for you. You can send requests to this. Your customers can send requests to this.

50. 10:37

    If I just hit run, and I'm gonna add a few. Let's...

51. 10:45

    What should, what should we ask the LLM today? Does anyone...

52. 10:52

    Okay. How did... I'm, I'm [REDACTED:origin], so how did Big Ben get its name? I don't know.

53. 11:02

    Um, okay, while these requests are queued, let me check on our workers.

54. 11:08

    Okay. We have a handful that are in- initializing. Um, this is the containers being created. That's the model being downloaded, um, getting ready. And the ones that are running, they've already finished.

55. 11:21

    These are probably gonna be the ones who are gonna pick up those requests that we just added. I've got telemetry about, um...

56. 11:30

    It's blank right now, but the number of requests, execution time, delay time. So you have observability into how your endpoints are operating.

57. 11:41

    And let's see. Okay, it's already done. I've got a request back, and it sat in the queue for about 41 seconds. Um, that's going to be a l- little bit longer than all of the subsequent requests because of some of the cold start time that I talked about, like downloading the model, um, initializing the first container.

58. 12:02

    But, um, execution time, only about one and a half seconds. So yeah, that was probably less than five minutes to get s- started and get something deployed, um, on serverless from a Hub listing.

59. 12:22

    Does anyone have any questions? This is, this is a very short and sweet intro. Um, we have another session later today at 4:00, um, and that one is gonna be focused on our Python Flash, um, SDK.

60. 12:36

    And that one is going to be completely via the terminal. Um, and I'm gonna sh- walk you through how I can spin up, uh, and deploy my code on...

61. 12:47

    my code as a remote, remote function onto a GPU, um, and, uh, deploy in the end, and make it, like, a production-ready endpoint here as well.

62. 13:04

    Okay. But that's all I got for today, so yeah. Thanks. Thanks for coming. [laughs] [audience applauds] [upbeat music]