Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Talk transcript

1. 0:00

   [upbeat music] So it's 9:00 AM, June 11th, 2024, and we send out the announcement and hold our breath, waiting, uh, to see the user

2. 0:25

   sign-ups pour in. We're expecting significant traffic for the launch of Dream Machine, Luma's first video model. And we were woefully unprepared for what came next. Uh, we'd allocated about 500 H100 GPUs.

3. 0:39

   We thought that was a lot at the time. It wasn't. Um, [chuckles]

4. 0:44

   and, uh, over the next hour, we saw request after request pour in, and a giant queue of requests start to pile up. Luckily, we had a contingency plan for this, um, which was to take every GPU we had access to from every provider and start manually running SSH commands against them to spin up workers, uh, pulling in

5. 1:08

   work from a global queue. Um, and we were able to get to about 5,000 H100s over the next six hours. And our queues of almost 100,000, uh, finally started to drain around, like, 2:00 PM that day.

6. 1:27

   And that is when Amit, our CEO and self-appointed chaos monkey, uh, decided to tweet that we had scaled up by 10X. Uh, come on, come on in. [chuckles] It'll be faster now.

7. 1:42

   And we'll see how it goes. Um, so we were-- our, our queues were down to about, you know, 300 at this point. And you can kinda see [chuckles] me here freaking out a little bit going, "Okay."

8. 1:53

   About 10 minutes after the tweet goes out, uh, they start to go up again, even though we've scaled up by 10 times. So now we're at 350, we're at 400, we're at 1,400.

9. 2:03

   And as this is going on, um, we're basically taking, uh, the entire training cluster. This is the only, uh, the only GPUs we hadn't used yet. Uh, taking the training cluster over, uh, it's another, like, 4,000, uh, H100 GPUs.

10. 2:18

    And, uh, it barely made a dent. The queues were still going up. [chuckles] Uh, so you'll see this, uh, it's, it's the KEKW, um, emoji, which has become a cultural staple of Luma over, over time, and it was very much the mood at the time.

11. 2:32

    Like, what am I supposed to do with this? [chuckles]

12. 2:36

    So, uh, I'm here to talk to you guys today a little bit about Luma, um, who we are, what we do, and how we've managed to kinda scale our infrastructure up, um, to initially a million users in four days, which was pretty nuts.

13. 2:50

    Uh, for some context, ChatGPT hit a million users in five days. Um, and we processed about half a million videos over the course of this 12 hours. Um, so yeah, we're gonna talk a little bit about what we learned scaling up and, you know, how we did it.

14. 3:04

    Um, but first, very quick, a little intro to Luma if you're not familiar with us. So we're not just a video model company, we're a foundation model lab, and we're aiming to build general multimodal intelligence that can, you know, generate, understand, and operate in the physical world just like a human can.

15. 3:21

    And, uh, to give you some sense of kinda what our models can do and where we're at today, um, this is a feature we dropped yesterday, a demo video for it. [dramatic music]

16. 3:31

    This is Modify Video. All these videos, um, are initially taken on iPhones and basically uploaded to our platform with a text prompt, and you can turn it... Just turn that raw footage into anything you want.

17. 3:46

    And for the AI engineers out in the audience, um, shout out to Kiran. He, uh, manages our public API, and you can integrate this functionality into your applications using it very easily.

18. 3:58

    Um, you don't need to do any kind of crazy prompt engineering. We've taken care of that for you. Just send us raw user, uh, prompts, generative media, and we'll send you back images and videos that, uh, meet your users' needs.

19. 4:14

    So if you're interested in that, uh, definitely hit me or Kiran up. Um, he's done a wonderful job building out, you know, our SDK and our whole kinda DevRel, uh, side of things.

20. 4:23

    So yeah, um, that is, uh, my little plug for our API in case anyone's interested. So back to infrastructure. This is what our serving stack basically looked like when we launched.

21. 4:36

    So we had a bunch of just tightly, tightly coupled, uh, containers working together. Um, one of the benefits of this is that we could kinda just launch these on raw machines with zero other dependencies.

22. 4:48

    Um, so that worked out well for us at launch. Um, but there were some challenges, uh, scaling this up. So, you know, like any good engineer, I didn't want to reinvent the wheel initially, so we, we reached for Triton Inference Server, um, which is kind of a classic, um, general purpose model serving, uh, server.

23. 5:06

    But there, there were some issues with it. Um, this setup was brittle. Um, if Triton went down, the, the CPU processes didn't necessarily know that it went down, and so you'd be kind of pulling jobs and they'd fail.

24. 5:18

    It was annoying. Um, worse though, with these video models, you're needing to, uh, run these on multiple GPUs and actually multiple nodes in a lot of cases to get to the latency you need, and Triton's just not built for that.

25. 5:31

    Um, also we run our inference now on multiple different chipsets. So, uh, NVIDIA is the, the company that builds Triton. Uh, they don't have great support for things like AMD or Groq or any of those.

26. 5:45

    Um, and finally, the, the biggest kinda hurdle was that this was really difficult to develop against for the researchers. Um, it had a whole bunch of different idioms, a whole bunch of, um, kind of incantations you needed to make to make it work well, and the overall setup was just, uh, it just felt very janky.

27. 6:01

    So what we ended up doing, um, was, you know, re-architecting to address some of these things and, uh, building our own serving stack on top of, you know, b- vanilla PyTorch for all the GPU work.

28. 6:13

    Um, that worked out really well because most of the vendors that are building, uh, these different chipsets, they make sure that PyTorch is fully supported. It's kind of this great substrate to build on top of.

29. 6:25

    If you support, you know, very vanilla PyTorch things, um, you can typically make your model run anywhere. Um, you may need to optimize certain optimi- uh, operations, uh, depending on your model to make things fast, but it's relatively easy to, uh, to get started.

30. 6:40

    Um, and in terms of, like, the decoupled architecture you h- you see here, uh, the CPU workers being decoupled is quite, quite useful and important because you can use, um, use them to queue up work and, you know, pull in when you're dealing with, you know, videos, images, multimedia inputs on top of just text.

31. 6:58

    You want those to be in the cluster ready for the GPUs to pull, so you're not blocking the GPUs at all. Um, and also with this architecture, you can actually run the GPUs anywhere.

32. 7:09

    As long as you can connect to Redis and our distributed storage, you can kind of see there, SeaweedFS, um, you can have a, a GPU in, you know, any kind of random provider, any random VM, and use Tailscale, connect, uh, connect to the rest of this architecture, and then scale up without having to do all that other

33. 7:29

    kind of crazy parallel SSH stuff. Um, so, you know, you wanna run, uh, compute on your training cluster, now you don't need to, um, you know, provision special machines or anything.

34. 7:39

    You just run a command. There were a few more challenges that we hit, um, after we got through kind of the, the initial hurdles of this, uh, this infrastructure.

35. 7:49

    So one of, uh, one of the big ones was backpressure. So because this is decoupled, um, you can get into a sit- and because there's multiple clusters that are, you know, pulling in work from the same global queue, you can get into this situation where you have too many CPU workers pulling in work to one cluster, and

36. 8:07

    they're, they're kind of waiting there to be processed by the GPUs on that cluster, uh, when they could have been processed somewhere else. So we came up with this, uh, dispatch, uh, limitation system.

37. 8:17

    So we came up with, like, a state where if a GPU has been pulled into a, uh, sorry, if a job has been pulled into a cluster and is waiting to be picked up by a GPU, uh, you can put a limitation on, like, how many jobs are in that state so that you can avoid this, this

38. 8:32

    issue. Um, I'll talk more in depth about, uh, priorities and our fair scheduling, uh, woes. That was another one. Um, you know, we have multiple tiers of users, and deciding whose jobs get processed first is, uh, a constant optimization challenge.

39. 8:48

    Um, also handling different models. So, like, these video models are big. They are typically made out of, you know, 10, 20 different, um, sub-models. So you're pulling in a lot of weights.

40. 8:59

    You're spending a lot of time compiling these things. So traditional auto-scaling is super wasteful. You're wasting, you know, 10, 20 minutes of GPU time just warming things up. Um, and finally, you know, handling bursts.

41. 9:15

    So, um, we basically built this system to handle bursts where you could scale up automatically on our training cluster, which our researchers hate me for. Uh, it makes them very upset.

42. 9:29

    But it allows us to keep up with the demand from our users. Um, and so that decoupled architecture plus a very simple scheduler that runs on top of Slurm and can just run these PyTorch workers, um, handles, you know, increasing the total pool of GPUs when we need it and scaling down when we don't.

43. 9:50

    So this system is a pull-based system, and there is, you know, queues that submit to queues that pull from queues, and it is, um, [laughs] a- as, you know, Sourish and Vasu can both attest to, uh, they're here from Luma, it is the least, um, enjoyable part of working at Luma is managing these queues.

44. 10:09

    It is, it's everybody's, uh, least favorite task. So, um, initially when we, uh, launched Ray 2, which is a much bigger, more resource-intensive model, um, we began to actually have to deal with the, the fact that we couldn't just keep scaling up more and more compute.

45. 10:28

    It just wasn't economical or feasible. Um, so we had to deal with limited resources. And, um, when you deal... when you have, like, a pull-based scheduler with a bunch of queues like this, there's this concept of work starvation that, that comes into play, where essentially, you know, we've got API, uh, tier jobs, which we try to p-

46. 10:49

    process very quickly. We've got enterprise. We've got unlimited, plus light, free. So we've got all these different tiers that have different priority, and who gets to go first? So if you just naively process them in priority order, there may be enough enterprise or API jobs that the light jobs never get processed.

47. 11:08

    So we were having people waiting for, like, seven, eight, nine hours, and they were very unhappy. Um, so also shout out to Sourish. He, [laughs] actually on a weekend after we designed this system, in a couple hours implemented, um, this SLO-based system, which I'll go into, that allows us to more fairly schedule, uh, work across the, the limited

48. 11:30

    resources that we have. So how the system works is when you think about, um,

49. 11:38

    the, the concept of Work starvation. One of the typical approaches you can, you can use to manage this is aging. So the idea is the longer something waits in the queue, the higher its priority goes.

50. 11:52

    But what's the actual function to control that aging mechanism? Um, so we had the insight that this is a product problem, and it really comes down to how long, worst case, are you okay with different tiers waiting?

51. 12:07

    Um, so we have service-level objectives that the product team defines, and it kinda controls this aging behavior. And how that works is, you know, an API job, we may not want to wait in a queue more than a couple minutes.

52. 12:19

    But a light job, maybe we're okay with them waiting for 10 minutes. And you can configure these and then set a threshold. Once, uh, once the threshold gets hit, so say 50% of that, you know, worst-case timing, then that job gets pulled to the front of the queue.

53. 12:36

    And initially, uh, that worked all right, but then you can actually hit another case of work starvation, um, if, if you have a bunch of, uh, jobs that are potentially breaching the SLA, where, um, you know, a, a job with a long SLO, since it's been waiting in the queue for a longer time, will actually starve out

54. 12:55

    the resources of, say, an API job that you don't want to wait as long. Um, so how we handled that was with a, uh... We, we ranked the jobs by the percentage of their SLO, um, that they, that they were at.

55. 13:08

    So, like, if an API job's waiting for a minute, that gets treated the same as a light job that's been waiting for 10 minutes. And that actually works out really nicely in practice and results in kind of intuitive fair scheduling behaviors.

56. 13:23

    The last piece I wanna talk about is kinda how we actually manage all these models. Um, so this was something where I was glad that we kinda reached for Triton initially.

57. 13:32

    They had this nice concept of a model repo, um, that, that we've really leaned into. So every model, um, has a folder in object storage somewhere with a bunch of subfolders that have different versions.

58. 13:48

    Um, and so as you, you know, develop the code and fine-tune these models, deploy new checkpoints, you, uh, create a bunch of these immutable versions. And then you have a simple YAML file in the root of the model folder that lets you define which one's active.

59. 14:06

    So that's really nice because you can kinda re- reproducibly, um, you know, i- in these, sorry, in these versions, you can, um... You store the full Python environment, so all the dependencies needed to run the model and the checkpoints.

60. 14:20

    And this system's pretty nice because, you know, if you ever need to roll back or, you know, you want to run things in all these different disparate environments, you can make sure that, you know, you're running the exact same Python environment, the exact same checkpoint that you were before.

61. 14:35

    And very rarely are there issues that, um, you can't really solve by just rolling back. So that's been quite useful for us and, um, we've actually built, like, a automated kinda rollout system on top of this too, where when we update those YAML files, the workers will just kinda switch versions on the fly without restarting.

62. 14:55

    And, um, that, that helps us kind of roll out, uh, new, new model changes to the whole fleet of these thousands of, uh, H100 and, uh, AMD GPUs all at once, um, which makes managing this much more sane than the early days of parallel SSH.

63. 15:12

    And, um, yeah, if any of this seemed interesting to you, up your alley, you know, we're hiring. We're actively looking for cracked engineers, researchers, um, AI enthusiasts in general.

64. 15:22

    Please, uh, please hit us up and, uh, yeah. I just wanted to thank everybody for taking the time today and, uh, thank the team at Luma 'cause, uh, it's, it's incredible the work that everyone there does. [audience applauding]

65. 15:36

    We've, we've got about three minutes for questions. So let's take one or two, and let's see where we get. Any questions?

66. 15:47

    You spoke about the, uh, or, or the ability to deploy different chipsets, um, and PyTorch being your lowest common denominator. Could you dive into that a little bit more?

67. 15:58

    Because each of these chipsets have their own providers. How does, how does all that work out?

68. 16:03

    Yeah. So the, the kinda cheat code here is that the chip, the chip providers who we typically partner with really closely, um, they're always making sure that PyTorch, at least of, at a certain version, works for their chipset.

69. 16:16

    So they're doing a lot of that work for us, and then typically what'll happen is we'll try to run the models on whatever the new chipset is. Usually, it'll work, and then it'll work a bit s- more slowly.

70. 16:29

    So we've got actually a team of, like, 10 guys we call our Excel team that are optimizing the low-level operations within PyTorch using things like Triton and working with the chip providers to make sure that things are actually fast.

71. 16:42

    So you'll typically with PyTorch be able to run things, um, anywhere, but a lot of the times they won't be fast, and then we just work closely with the chipset, uh, providers to actually, you know, optimize the model over time.

72. 16:55

    Are you able to share who, uh, which, which chipsets you work with?

73. 16:59

    Yeah, yeah. We work really closely with NVIDIA and AMD and, uh, we are exploring, you know, some other providers, um, but nothing really deep yet. Like, you know, Amazon's got their own chips.

74. 17:09

    Uh, Groq has some chips. We just announced a big partnership with, uh, Humane, who works really closely with, with Groq. And so yeah, we're exploring some of these other chipsets, um, through, through some of these, uh, partnerships.

75. 17:23

    Cool. Any other questions? Yeah.

76. 17:25

    I'm just curious if you're comfortable saying if you run on, like, bare metal Kubernetes.

77. 17:33

    We do not. We... Well, I don't think so. Um, but yeah, we, we, we work with, uh, cloud providers, uh, that basically kinda give us a, um, a working Kubernetes cluster at the very least, which, which is nice.

78. 17:48

    Um, we're not provisioning the nodes ourself or anything. But actually would have to check with the individual providers if there's, like, VMs or if they're actually metal machines. I know, uh, when we work with Amazon, they're not, but some of the other ones might be.

79. 18:00

    I work with Amazon AWS, so instead of bunch of keywords- [laughs] ... I'm trying to see if you get [laughs] [laughs]

80. 18:09

    Cool. Yeah.

81. 18:11

    Yeah. Um, your video model is super impressive, but I saw on your website, uh, you're exploring, like, other use cases. Have you shipped any visual QA models yet?

82. 18:21

    Uh, not yet. Um, the way the, like, general application works, um, does involve, uh, video and, like, image QA models. So, um, essentially the a- the way the actual application works is we have, like, an agent that you're interacting with.

83. 18:38

    So when you upload a video or an image, that is actually being, you know, captioned by v- you know, VLMs to enhance the, the, the total pro- the overall prompt.

84. 18:50

    But, um, no, like, true VQA stuff quite yet. But there, there may or may not be some coming. [upbeat music]