GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Talk transcript

1. 0:00

   [on hold music] Hey, everyone.

2. 0:16

   I'm Audry. Um, I work at RunPod. Have-- Were any of you in my earlier session?

3. 0:24

   Okay, good, 'cause then I'm gonna s- s- This intro is the same, but what I'm gonna show is l- a little bit different. Um, d- has anyone heard of RunPod or used RunPod before?

4. 0:35

   You have? Do, do you mind if I ask you, um, how you've used us or heard about us before? Yeah, as part of our university, we have some RunPod credits.

5. 0:42

   So I, I use this as, uh, yeah, for LLM training and- LLM training? Okay, at your university. And where do you go to uni? Uh, u- Oxford. Oxford. Yeah.

6. 0:49

   I did a study abroad there one summer. It's awesome there. Yeah. [laughs] Okay. Love it. Thank you. Okay. And your name is? Yunus. Yunus. Okay. So Yunus might know a little bit about this already, but I'll, I'll, I'll talk you guys through a little intro.

7. 1:02

   Um, what do we do? We're a AI cloud infrastructure company. Um, and our mission is to build the foundational platform for developers to scale their AI workloads. And basically, what that means is, um, we bring the hardware, we bring the GPUs, and the compute.

8. 1:23

   Um, we make it easy for you guys to bring your code, bring your models, and deploy as quickly as possible. We don't want you spending time, um, configuring infrastructure and thinking about things like, um, scaling.

9. 1:39

   W- why RunPod exists. A lot of teams that we've talked to, they are all wrestling with the same thing. That infrastructure-- They're spending more time with the infrastructure than, than they are with the models.

10. 1:49

    So things like CUDA version, um, alignment, like what versions of PyTorch run well together, um, which, which, uh, new of the GPU, uh, SKUs, like, have been tested, and, like, keep figuring out the bugs there.

11. 2:05

    So a lot of that is-- are things that we try to take that, um, configuration problem away from you guys, so you guys can just build, um, focus on training your model or building your apps.

12. 2:17

    And a little bit of a backstory about our company. So this is Zen and Pardeep, our two founders. Um, they started RunPod in 2022. They had a failed crypto mining venture, so they had a bunch of spare GPUs in their basement.

13. 2:34

    Um, they built a prototype of what is the foundation of RunPod today, and they just posted on Reddit and said, "Does, does anyone want some free GPUs, um, in exchange for feedback?"

14. 2:47

    And that is literally how our company started, and ever since then, we've been building with, in public with the community. Um, so we've been revenue generating from the very beginning, um, which is very, very rare.

15. 3:00

    Um, and even today, we have around five hundred developers on our platform.

16. 3:06

    We're in thirty-plus data centers across ten countries. Um, in Europe, that includes France, Romania, Iceland, if that's part of Europe, uh, [laughs] Asia Pacific. Um, and we recently hit, um, a pretty big milestone of one hundred and twenty million in annual recurring revenue.

17. 3:27

    This is just a quick glance of some of the customers that we have. You might be surprised seeing that some of these are AI native companies, um, and some large enterprises as well.

18. 3:41

    And kind of the bottom line of what they have in common is that they need flexible and reliable GPU infrastructure. So, um, I definitely would say we're punching above our weight class.

19. 3:55

    Really quick, high level, um, there's different ways to build on RunPod, depending on what you're trying to do. Um, so if you need a more persistent, um, VM environment, then Pods is a great use case.

20. 4:07

    Um, if you, you can rent a pod on demand, um, pay by the second, and once you're done, you can tear it all down, um, and start again. Pods are if you need, um, reserved GPU, so as long as your pod is running, then the GPU is yours and no one can take it away from you.

21. 4:26

    Um, serverless, if you have-- if you're ready to deploy something and you, and, and you care more about scaling, so your workloads are more variable in terms of, um, frequency and load, um, we help you auto scale, um, your workers for you and scale them back down when you don't have any requests happening, so you don't pay

22. 4:45

    for, um, any idle time. Um, clusters, great use case for training, multi-node. And then Hub is also, um, a place where you can deploy already, uh, open source AI repos that have already been pre-vetted by us for popular models, um, like ComfyUI, um, Stable Diffusion, um, vLLM.

23. 5:11

    Um, and that's one way if you're just exploring, uh, to just click around and get started really quickly.

24. 5:18

    Uh, I'm gonna talk about serverless, um, today, and the product that we just...

25. 5:26

    And I'm gonna switch my displays again here really, really quick so we can mirror my screen.

26. 5:43

    So one of the things that is a huge pain for developers is if they're still in the iteration or the development phase. So normally, um,

27. 5:54

    when you are working on, let's say, some, some code around your, um, inference model and you're still testing things out, you have to make a commit- Push it to GitHub, um, build your Docker image, um, pull it down from the i- um, container registry, [smacks lips]

28. 6:13

    um, and then load it onto a server, and then allocate a GPU on it, and then you get to test it and see if it's working as you expected.

29. 6:21

    And then you do that all over again [chuckles] until, until you're ready. So the problem that Flash is trying to solve here, and Flash is our Python SDK, is that we want to eliminate all of that iteration cycle so that you can, um, deploy your function on a GPU right from your local development environment.

30. 6:45

    And I'll zoom in here really quick. So this is all you need to know about Flash in one little paragraph, is it's... You have a regular async Python function.

31. 6:58

    You add our Flash endpoint decorator, and it's gonna deploy and package everything inside your function onto a GPU cloud. Everything ar- around it, your, uh, main function, any helper functions that you have, those all run on your local development environment.

32. 7:15

    But if you need GPU compute, that, that can run, um, on the cloud, and you can... We have hot mod, uh, h- file reload, so if you change anything in your application anywhere, then it gets repackaged and pushed up immediately, and you can test and iterate super quickly.

33. 7:33

    And I'm just gonna show an example of this.

34. 7:42

    Okay. So I have a function here, generate image. Simply, I'm loading PyTorch. I'm loading a pre-trained Stable Diffusion model, Stable Diffusion XL Turbo.

35. 7:59

    Really great for fast, uh, generation of images. And [smacks lips]

36. 8:05

    I'm going to save the image down and that... gonna return it Base64 encoded. [smacks lips]

37. 8:12

    So I can run this right now here. I've already, um, installed all my dependencies. I already have a Flash project going. I'm going to flash run,

38. 8:27

    uh, image_generation.py. And what I'm going to actually do is I have a little [keyboard clacking]

39. 8:34

    flash, flash run. So Flash Run spins up a local development server here. Uh, it's just a FastAPI server, and I can send my request here

40. 8:52

    to this endpoint, and I'm gonna do that really quickly. Just get to, to my project. And [keyboard clacking]

41. 9:06

    this is just a little helper script that's gonna send a post request to it and then, um, decode that image so that you guys c- actually get to see what it looks like once it's generated.

42. 9:20

    And it was image_generation_async. Here we go. [keyboard clacking] And let's pass a prompt to it.

43. 9:38

    Can I get a help from the audience? [chuckles] What do we wanna generate today? [chuckles]

44. 9:43

    Literally anything, anything random. Cats flying in the sky. Okay. Cats flying in the sky. W- what does the sky look like? What time of day is it? Cloudy. Um- London.

45. 9:56

    London ... in London. Yeah. Flying in, flying on a cloudy day in the sky

46. 10:05

    s- somewhere in London, and I passed it correctly.

47. 10:17

    He's, he's looking at it so closely. He's helping me debug live. I love it. [laughs] Thank you. [laughs] It's super hard to see in dark mode, but, uh-

48. 10:26

    I passed URL, and that's true. Based after a localhost? Um.. [keyboard clacking]

49. 10:39

    I must have... Oh, you've got your... Oh, no, never mind Is it my HTTP?

50. 10:53

    Okay, there we go. Okay. Going back to the local dev server.

51. 11:05

    It sees the request. It's started the job. It's queued it.

52. 11:14

    And we're just gonna wait for a second to see if it finishes. And so while that's happening, let me bring your attention back to... [sighs]

53. 11:28

    Make it bigger for you guys. [smacks lips] Um, the endpoint decorator. So this is where all the magic happens.

54. 11:38

    I have passed a name for my endpoint. I specify a GPU family, um, so the Ada 80 Pros, these are different variations of NVIDIA H100 cards. Um, I can specify my max number of workers to be five, so I can have at max five of them running at once.

55. 11:57

    Um, I just put one active worker, so this is one that's always gonna be running and always on, [smacks lips] and that's definitely a dragon. [chuckles]

56. 12:06

    And it didn't take my prompt probably because I...

57. 12:11

    Did I not pass it as a... I didn't pass it as a flag.

58. 12:21

    Prompt, there we go. Okay, now it's definitely generating cats flying.

59. 12:30

    Okay, back to the endpoint decorator. Um, and then there's other different configurations for timeout, which is how long, um, a worker is idle. Here we go.

60. 12:41

    Okay, this looks terrible, guys. [laughs] They are cats. They're abstract cats. Um, and I'm not from London, but maybe someone can tell me if this looks like a London Jimny.

61. 12:56

    Maybe? Oh, yeah. [laughs] Okay. So I don't, I don't, I don't like, I don't like what just happened, so what we're going to do instead is we're going to switch out our model.

62. 13:08

    So I'm just gonna comment out this code here,

63. 13:12

    and then down here let's swap in DreamShaper, which is a fine-tuned, um,

64. 13:24

    model based off of Stable Diffusion 1.5. Um, so this one is while... where Stable Diffusion XL Pro is, um, more optimized for just quick generation, I think this one is gonna generate a more, uh, a better quality image for us, and it's specifically better for kind of like more art and illustrative styles.

65. 13:46

    So we've changed some of the parameters in it. It's gonna have a few more, um, inference steps to it. Just gonna set that to 25. Height and width, 10 by 24.

66. 13:57

    That's fine. And let's just send the same request again,

67. 14:03

    and let's see what happens, what's different. So again, what made this really fast is instead of making a code change, committing it, rebuilding my Docker, uploading it somewhere, and then allocating GPU infrastructure, all of this is happening right here from my IDE, and I never have to leave.

68. 14:28

    Oh, it's good. This is good, right, guys? Mm-hmm. We like this one?

69. 14:34

    Okay. So one more... one last thing that I'm just gonna show you guys to round things out is I think where, um, using a developer tool like Flash makes a big difference is when you're trying to, um, not just, like, make one single call to one model.

70. 14:52

    It's, it's about all the orchestration code around it, right? So I have here a pipeline that I've pre-prepared, um, and what it's going to do is instead of me generating and writing out every prompt, um, it's gonna send a request to Qwen that's already hosted on a public endpoint.

71. 15:13

    Um, and Qwen is g- Qwen 3 is gonna generate all the prompts for me. Um, and then after that, it's gonna send that to our, um, DreamShaper running on our endpoint, and then, um, after that, there's one more pipeline that it goes through.

72. 15:29

    It's gonna send the request to NanoBanana 2, which is a premium Google model that's really good at composing photos together. Um, and I'm hoping that I can compose some cool pictures of our founders, and I can send them to them after this demo is done.

73. 15:47

    Okay. Um, now let's run the whole pipeline here.

74. 15:56

    And check... Okay. Prompt, two men walking in London on a... It is cloudy today.

75. 16:10

    Cloudy tay. Close up of their faces. Um, any other requests for these two men?

76. 16:21

    AI engineers? AI engineers? I don't know if it's- How do they look? Like, are they doing something?

77. 16:28

    Glasses? I don't know. Glasses? Yeah. Okay. Two men

78. 16:34

    with glasses. Close up of their faces. Okay, and let's generate,

79. 16:43

    let's generate three of those, and let's compose it together.

80. 16:55

    Okay. And so, so how does it work in terms of pricing? Pricing? Sure. Um, so every request that we send to it, you're only charged for how long that request is running.

81. 17:07

    So let's see. I'm gonna... I said this whole session was gonna be only in the terminal, but I'm gonna go back into the console just to show you what's, what's running.

82. 17:21

    Let's see. This is the endpoint that we created from the terminal. Um, here are the workers.

83. 17:32

    I think we said, like, five workers, so we have about five or six here that are provisioned. Um, three of them are running 'cause I asked for three photos.

84. 17:40

    Um, and so this is uptime. This is what you're being charged for, and let me see the cost of an H100 right now is .00116 cents per second.

85. 17:57

    Is it the same as for a pod, or is it, uh... or is this pricing a bit different? Pricing is a little bit different for serverless versus pods because pods, um, you don't get any of the scaling with it.

86. 18:08

    So there's a little bit of a premium for serverless. So what we usually recommend is if you're still, um, experimenting- Then either start with a very low worker count or start with pods, right?

87. 18:19

    'Cause when you're experimenting, you might only need limited number of GPUs, one GPU at a time, two GPUs at a time. Um, serverless for when you need hundreds of workers running on hundreds of GPUs, and you want them distributed for a better availability across different data centers. [laughs]

88. 18:40

    Okay, guys. Here, [laughs] here's th- here's our final presentation. So this was our original prompt, two men with glasses walking in London on a cloudy day, close-up of their faces.

89. 18:49

    So on the left, this is what, um, DreamShaper generated based off of the, um, prompt engineering that Qwen 3 did for us. So it's a lot better of a prompt than what I sent in.

90. 19:02

    It has a lot better cues about notes on, like... I can read it out to you since I know it's hard to see. Um, "Thoughtful expressions and weathered faces.

91. 19:12

    Soft focus on background clouds. Muted urban palette with grays and deep blues. Overcast lighting."

92. 19:19

    Um, and then on the right is the final composed photo, um, of... This, this is a hand- this is a very handsome picture of Pardeep, and this is somehow a [laughs] very old photo of Zed.

93. 19:32

    And I'm just gonna scroll down to show you that this is the reference photo that I sent it. [laughs] [laughs]

94. 19:41

    Um, but overall, um, overall, yeah, I just wanted to show you guys, like, this is how you can get started really quickly. Um, you can start in your local development environment.

95. 19:52

    You can use open sourced models. You can use bring your own model, um, private model. Um, and this was really fun to do. So thank you guys for hanging out with me. [clapping] [outro jingle]