AI Engineer World's Fair 2025
Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI
Read the talk
Dream Machine: Scaling Beyond the Emergency GPU Fleet
Luma’s launch surge turned GPU provisioning into a scheduling problem: keeping accelerators fed, sharing scarce capacity fairly, and deploying models across a changing fleet.
From a talk by Keegan McCallum
Before you start: Familiarity with worker queues, GPU inference, and basic Python will help with the architecture and scheduling examples.
When ten times the capacity is still not enough
At 9 AM on the launch morning—June 11, 2024, in Keegan McCallum’s account—Luma announced Dream Machine, its first video model, and waited for sign-ups. The team had allocated about 500 H100 GPUs. Within an hour, requests were accumulating faster than the fleet could process them. The contingency plan was direct: take every available GPU from every provider, SSH into the machines, and start workers pulling from a global queue.
McCallum reports expanding to about 5,000 H100s over six hours, with a backlog approaching 100,000 requests beginning to drain around 2 PM. The launch-day chat shown in the recording captures two queue counters at 14,247 and 70,600.
Then Amit, Luma’s CEO and self-appointed chaos monkey, tweeted that capacity had increased tenfold and invited people back. The queue had fallen to roughly 300; about ten minutes after the tweet, it climbed through 350, 400, and 1,400. Luma commandeered the remaining training cluster—another roughly 4,000 H100s—but McCallum says it barely dented the growing queues. The KEKW laughing emote became a recurring expression of the team’s predicament: even a dramatic capacity increase could immediately attract more demand.
McCallum reports one million users in four days and about half a million videos processed in twelve hours. He compares the adoption milestone with ChatGPT reaching one million users in five days. That comparison concerns user growth; the video count describes Luma’s processing volume, not a comparable inference benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The workload includes more than a text prompt
Luma describes itself as a foundation model lab pursuing general multimodal intelligence: systems that generate, understand, and eventually operate in the physical world. Video generation is one expression of that ambition. In the Modify Video demonstration, introduced as a feature released the previous day, footage recorded on iPhones becomes the input to a transformation directed by a text prompt. The selected frame shows a basketball player leaping toward a hoop in a blue-lit arena.
The accompanying API pitch is that developers can send raw user prompts and media while Luma handles the prompt engineering needed to return images and videos. McCallum credits Kiran with managing the public API, SDK, and developer relations. The product demonstration and API availability have separate release histories: Luma’s changelog lists Modify Video on June 4, 2025, and its API release on June 24. The API invitation should therefore not be read as establishing that this particular feature was already publicly available through the API on the day of the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The launch stack was easy to start and difficult to operate
The original serving stack consisted of tightly coupled containers. That packaging had a valuable property during the emergency: the team could launch it on raw machines without additional infrastructure dependencies. Luma initially chose NVIDIA Triton Inference Server rather than build a serving system from scratch.
As the system grew, four limitations became consequential:
- Failure coordination: If Triton went down, CPU processes did not necessarily notice. They could continue pulling jobs that would fail.
- Distributed execution: Luma’s video models needed multiple GPUs, and sometimes multiple nodes, to reach acceptable latency. McCallum found the stack poorly suited to that workload. This is a claim about Luma’s video-serving experience; NVIDIA also documents a multi-node TensorRT-LLM deployment path.
- Hardware choice: McCallum cites weak support for non-NVIDIA hardware, naming AMD and Groq as examples.
- Research iteration: Researchers had to learn unfamiliar serving idioms and configuration rituals. The environment was brittle and awkward to develop against.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate media preparation from GPU execution
Luma rebuilt GPU serving around vanilla PyTorch. Chip vendors already invest in supporting the framework, so ordinary PyTorch operations provide a useful starting point for moving models between accelerators. That makes execution portable, not automatically fast: particular operations can still need hardware-specific optimization.
The replacement also decoupled CPU workers from GPU workers. CPU workers prepare jobs and bring their video, image, and other media inputs into the cluster before a GPU needs them. The accelerator should pull ready work, rather than wait for input preparation. Separating these responsibilities lets the CPU side prepare upcoming work while the GPU side runs inference.
A GPU worker can join from an arbitrary provider or VM as long as it can reach Redis and the shared SeaweedFS storage. Tailscale connects those machines to the rest of the architecture. This reduces the work required to add capacity: even on the training cluster, inference workers can start with a command, without specially provisioned machines or another round of parallel SSH orchestration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep preparation from trapping work in the wrong cluster
Decoupling creates a new backpressure problem. Multiple clusters pull from the same global queue. If one cluster’s CPU workers pull too aggressively, they can accumulate jobs waiting for that cluster’s GPUs while GPUs elsewhere could have processed them. Preparation has effectively committed the work to a congested destination.
Luma introduced a dispatch limit on a specific intermediate state: a job has entered a cluster but has not yet been picked up by a GPU. Capping the number of jobs in that state prevents one cluster from taking too much work out of global circulation. The important quantity is not simply how many CPU workers are available; it is how much prepared work the local GPU pool can absorb. Which user’s job should enter that pipeline is a separate fairness problem.
Adding GPUs also has a startup cost. McCallum describes video models containing roughly 10–20 sub-models and traditional autoscaling warmups consuming 10–20 minutes of GPU time loading weights and compiling. Frequent cold starts can therefore spend scarce accelerator time preparing to serve rather than serving.
To handle bursts, Luma built a simple scheduler on Slurm that starts PyTorch workers on the training cluster. It expands the inference GPU pool when demand rises and shrinks it when demand falls. Researchers dislike losing training capacity, but the shared architecture makes that capacity available to users without rebuilding the deployment environment each time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When capacity is finite, priority needs a clock
The resulting system is pull based, with queues submitting to other queues and workers pulling through several stages. McCallum calls queue management the team’s least enjoyable task, invoking colleagues Sourish and Vasu as witnesses. When Ray2, a much larger and more resource-intensive model, arrived, Luma could no longer solve every backlog by adding compute. Further expansion was neither economical nor feasible.
The service had API, enterprise, Unlimited, Plus, Lite, and Free tiers. Processing them strictly in priority order creates work starvation: enough API or enterprise traffic can prevent a Lite job from ever reaching a GPU. McCallum reports users waiting seven to nine hours under this priority starvation. After the team designed an SLO-based alternative, Sourish implemented it in a couple of hours over a weekend.
The usual remedy is aging: increase a job’s priority as it waits. But an aging function still needs a policy. Luma treated that policy as a product decision: how long is it acceptable, in the worst case, for each tier to wait? The product team defines a service-level objective, or SLO, for each tier. McCallum illustrates an API target of a couple of minutes and a Lite target of ten minutes. A configurable threshold—50% of the allowed wait in his example—promotes a job to the front of the queue.
Promotion alone introduced another starvation case. Among jobs approaching their objectives, sorting by absolute waiting time favored jobs with long SLOs. A Lite job could have waited longer in minutes while a newer API job was much closer to missing its own target. Luma’s final ranking compares the fraction of each job’s SLO already consumed:
A larger fraction means greater urgency. The policy compares waiting time against the promise made to that tier, rather than treating a minute as equally consequential for every user.
For a concrete calculation, use a two-minute API objective and a ten-minute Lite objective. The following Python expresses the normalized ranking among jobs that have crossed a 50% promotion threshold; it does not specify the rest of the queue policy.
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Job:
id: str
wait_seconds: int
slo_seconds: int
@property
def urgency(self) -> float:
if self.slo_seconds <= 0:
raise ValueError("SLO must be positive")
return self.wait_seconds / self.slo_seconds
jobs = [
Job("api-a", wait_seconds=90, slo_seconds=120),
Job("lite-b", wait_seconds=360, slo_seconds=600),
]
promoted = sorted(
(job for job in jobs if job.urgency >= 0.5),
key=lambda job: job.urgency,
reverse=True,
)
for job in promoted:
print(f"{job.id}: {job.urgency:.0%} of SLO")
| Job | Wait | SLO | SLO consumed |
|---|---|---|---|
api-a | 90 seconds | 120 seconds | 75% |
lite-b | 360 seconds | 600 seconds | 60% |
Both jobs qualify for promotion. The ranking places api-a first even though lite-b has waited longer. These are illustrative job values; ranking does not mean either job has executed.
McCallum gives a separate example in which an API job waiting one minute receives the same treatment as a Lite job waiting ten minutes. Equal normalized waits in that example imply SLOs with a 1:10 ratio, rather than the earlier two-minute and ten-minute configuration. The mechanism is the same: urgency follows the percentage of the allowed wait. McCallum says this produced intuitive, fair scheduling behavior in practice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Version the model and its entire Python environment
Luma retained one useful idea from Triton: the model repository. Each model lives in an object-storage folder containing version subfolders. As researchers change code, fine-tune models, or deploy checkpoints, they create immutable versions. A YAML file at the model folder’s root selects the active version. The repository concept is borrowed; the YAML selector and rollout behavior are Luma’s implementation, not Triton’s standard repository configuration.
Each version includes both its checkpoints and its full Python environment, with the dependencies needed to run the model. That makes rollback more precise than selecting an older weights file: workers recover the same environment and checkpoint across otherwise disparate execution environments. McCallum says rollback resolves most of the issues they encounter.
An automated rollout system watches the YAML selections. Updating them causes workers to switch model versions on the fly without restarting. McCallum describes using this mechanism across a fleet of thousands of H100 and AMD GPUs. The deployment operation has moved from manually reaching individual machines to changing which immutable version the fleet should run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Portable models still need an acceleration team
The first audience question asks what PyTorch portability actually means when every chipset has its own provider and software stack. McCallum’s answer starts with the vendors: they make sure at least a particular PyTorch version works on their hardware. Luma then tries its models on the new chipset. Usually the model runs, but initially it can run more slowly.
Luma has an acceleration team of about ten people, according to McCallum, optimizing low-level PyTorch operations and working directly with chip providers. The tools include the Triton GPU language and compiler, distinct from the Triton Inference Server discussed earlier. PyTorch provides a route to functional execution; efficient execution still requires kernel work and vendor collaboration.
Asked which hardware Luma uses, McCallum identifies close relationships with NVIDIA and AMD. He distinguishes those from exploration of other providers, including Amazon’s chips and Groq. He also points to the announced HUMAIN partnership, describing HUMAIN’s relationship with Groq as another route for exploration. These remarks do not establish production inference on Groq hardware.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Providers supply the Kubernetes clusters
Another question asks whether Luma runs Kubernetes on bare metal. McCallum initially says no, then qualifies the answer: providers supply working Kubernetes clusters, and Luma does not provision the nodes itself. He would need to check individual providers to say whether their underlying machines are VMs or bare metal. He identifies the Amazon setup as not bare metal while leaving the other providers uncertain. The clear operational boundary is who supplies the functioning cluster, not a uniform claim about the physical machines beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Visual understanding already sits inside the application
The final question returns to Luma’s broader multimodal ambitions: has it shipped visual question-answering models? Not yet, McCallum says, although the application already uses image and video understanding internally. Users interact with an agent, and uploaded videos or images are captioned by vision-language models, or VLMs, to enhance the overall prompt.
That internal understanding step is distinct from a publicly shipped VQA offering. At the time of the talk, McCallum says Luma has not released that offering; he hints at possible future work without making a commitment. The generation interface already depends on interpreting the user’s media, even when visual question answering is not exposed as a separate product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The June 2025 announcement explains video transformation, motion preservation, and the Adhere, Flex, and Reimagine controls.
Documents model folders, version directories, configuration files, and supported storage locations.
Distributed storage software with object and file interfaces, setup instructions, and deployment examples.
Explains how Tailscale connects machines across networks using encrypted connections.
Introduces Slurm's cluster resource allocation, job execution, and scheduling components.
Tools and tutorials for writing GPU kernels, distinct from Triton Inference Server.
Further reading
An archived NVIDIA guide to deploying large language models across GPUs and nodes with Kubernetes.
Updates since the talk
Ray2-era API examples for modifying videos with prompts, reference frames, and strength settings; the page also directs readers to newer API documentation.
Read the complete timestamped transcript
- 0:00
[upbeat music] So it's 9:00 AM, June 11th, 2024, and we send out the announcement and hold our breath, waiting, uh, to see the user
- 0:25
sign-ups pour in. We're expecting significant traffic for the launch of Dream Machine, Luma's first video model. And we were woefully unprepared for what came next. Uh, we'd allocated about 500 H100 GPUs.
- 0:39
We thought that was a lot at the time. It wasn't. Um, [chuckles]
- 0:44
and, uh, over the next hour, we saw request after request pour in, and a giant queue of requests start to pile up. Luckily, we had a contingency plan for this, um, which was to take every GPU we had access to from every provider and start manually running SSH commands against them to spin up workers, uh, pulling in
- 1:08
work from a global queue. Um, and we were able to get to about 5,000 H100s over the next six hours. And our queues of almost 100,000, uh, finally started to drain around, like, 2:00 PM that day.
- 1:27
And that is when Amit, our CEO and self-appointed chaos monkey, uh, decided to tweet that we had scaled up by 10X. Uh, come on, come on in. [chuckles] It'll be faster now.
- 1:42
And we'll see how it goes. Um, so we were-- our, our queues were down to about, you know, 300 at this point. And you can kinda see [chuckles] me here freaking out a little bit going, "Okay."
- 1:53
About 10 minutes after the tweet goes out, uh, they start to go up again, even though we've scaled up by 10 times. So now we're at 350, we're at 400, we're at 1,400.
- 2:03
And as this is going on, um, we're basically taking, uh, the entire training cluster. This is the only, uh, the only GPUs we hadn't used yet. Uh, taking the training cluster over, uh, it's another, like, 4,000, uh, H100 GPUs.
- 2:18
And, uh, it barely made a dent. The queues were still going up. [chuckles] Uh, so you'll see this, uh, it's, it's the KEKW, um, emoji, which has become a cultural staple of Luma over, over time, and it was very much the mood at the time.
- 2:32
Like, what am I supposed to do with this? [chuckles]
- 2:36
So, uh, I'm here to talk to you guys today a little bit about Luma, um, who we are, what we do, and how we've managed to kinda scale our infrastructure up, um, to initially a million users in four days, which was pretty nuts.
- 2:50
Uh, for some context, ChatGPT hit a million users in five days. Um, and we processed about half a million videos over the course of this 12 hours. Um, so yeah, we're gonna talk a little bit about what we learned scaling up and, you know, how we did it.
- 3:04
Um, but first, very quick, a little intro to Luma if you're not familiar with us. So we're not just a video model company, we're a foundation model lab, and we're aiming to build general multimodal intelligence that can, you know, generate, understand, and operate in the physical world just like a human can.
- 3:21
And, uh, to give you some sense of kinda what our models can do and where we're at today, um, this is a feature we dropped yesterday, a demo video for it. [dramatic music]
- 3:31
This is Modify Video. All these videos, um, are initially taken on iPhones and basically uploaded to our platform with a text prompt, and you can turn it... Just turn that raw footage into anything you want.
- 3:46
And for the AI engineers out in the audience, um, shout out to Kiran. He, uh, manages our public API, and you can integrate this functionality into your applications using it very easily.
- 3:58
Um, you don't need to do any kind of crazy prompt engineering. We've taken care of that for you. Just send us raw user, uh, prompts, generative media, and we'll send you back images and videos that, uh, meet your users' needs.
- 4:14
So if you're interested in that, uh, definitely hit me or Kiran up. Um, he's done a wonderful job building out, you know, our SDK and our whole kinda DevRel, uh, side of things.
- 4:23
So yeah, um, that is, uh, my little plug for our API in case anyone's interested. So back to infrastructure. This is what our serving stack basically looked like when we launched.
- 4:36
So we had a bunch of just tightly, tightly coupled, uh, containers working together. Um, one of the benefits of this is that we could kinda just launch these on raw machines with zero other dependencies.
- 4:48
Um, so that worked out well for us at launch. Um, but there were some challenges, uh, scaling this up. So, you know, like any good engineer, I didn't want to reinvent the wheel initially, so we, we reached for Triton Inference Server, um, which is kind of a classic, um, general purpose model serving, uh, server.
- 5:06
But there, there were some issues with it. Um, this setup was brittle. Um, if Triton went down, the, the CPU processes didn't necessarily know that it went down, and so you'd be kind of pulling jobs and they'd fail.
- 5:18
It was annoying. Um, worse though, with these video models, you're needing to, uh, run these on multiple GPUs and actually multiple nodes in a lot of cases to get to the latency you need, and Triton's just not built for that.
- 5:31
Um, also we run our inference now on multiple different chipsets. So, uh, NVIDIA is the, the company that builds Triton. Uh, they don't have great support for things like AMD or Groq or any of those.
- 5:45
Um, and finally, the, the biggest kinda hurdle was that this was really difficult to develop against for the researchers. Um, it had a whole bunch of different idioms, a whole bunch of, um, kind of incantations you needed to make to make it work well, and the overall setup was just, uh, it just felt very janky.
- 6:01
So what we ended up doing, um, was, you know, re-architecting to address some of these things and, uh, building our own serving stack on top of, you know, b- vanilla PyTorch for all the GPU work.
- 6:13
Um, that worked out really well because most of the vendors that are building, uh, these different chipsets, they make sure that PyTorch is fully supported. It's kind of this great substrate to build on top of.
- 6:25
If you support, you know, very vanilla PyTorch things, um, you can typically make your model run anywhere. Um, you may need to optimize certain optimi- uh, operations, uh, depending on your model to make things fast, but it's relatively easy to, uh, to get started.
- 6:40
Um, and in terms of, like, the decoupled architecture you h- you see here, uh, the CPU workers being decoupled is quite, quite useful and important because you can use, um, use them to queue up work and, you know, pull in when you're dealing with, you know, videos, images, multimedia inputs on top of just text.
- 6:58
You want those to be in the cluster ready for the GPUs to pull, so you're not blocking the GPUs at all. Um, and also with this architecture, you can actually run the GPUs anywhere.
- 7:09
As long as you can connect to Redis and our distributed storage, you can kind of see there, SeaweedFS, um, you can have a, a GPU in, you know, any kind of random provider, any random VM, and use Tailscale, connect, uh, connect to the rest of this architecture, and then scale up without having to do all that other
- 7:29
kind of crazy parallel SSH stuff. Um, so, you know, you wanna run, uh, compute on your training cluster, now you don't need to, um, you know, provision special machines or anything.
- 7:39
You just run a command. There were a few more challenges that we hit, um, after we got through kind of the, the initial hurdles of this, uh, this infrastructure.
- 7:49
So one of, uh, one of the big ones was backpressure. So because this is decoupled, um, you can get into a sit- and because there's multiple clusters that are, you know, pulling in work from the same global queue, you can get into this situation where you have too many CPU workers pulling in work to one cluster, and
- 8:07
they're, they're kind of waiting there to be processed by the GPUs on that cluster, uh, when they could have been processed somewhere else. So we came up with this, uh, dispatch, uh, limitation system.
- 8:17
So we came up with, like, a state where if a GPU has been pulled into a, uh, sorry, if a job has been pulled into a cluster and is waiting to be picked up by a GPU, uh, you can put a limitation on, like, how many jobs are in that state so that you can avoid this, this
- 8:32
issue. Um, I'll talk more in depth about, uh, priorities and our fair scheduling, uh, woes. That was another one. Um, you know, we have multiple tiers of users, and deciding whose jobs get processed first is, uh, a constant optimization challenge.
- 8:48
Um, also handling different models. So, like, these video models are big. They are typically made out of, you know, 10, 20 different, um, sub-models. So you're pulling in a lot of weights.
- 8:59
You're spending a lot of time compiling these things. So traditional auto-scaling is super wasteful. You're wasting, you know, 10, 20 minutes of GPU time just warming things up. Um, and finally, you know, handling bursts.
- 9:15
So, um, we basically built this system to handle bursts where you could scale up automatically on our training cluster, which our researchers hate me for. Uh, it makes them very upset.
- 9:29
But it allows us to keep up with the demand from our users. Um, and so that decoupled architecture plus a very simple scheduler that runs on top of Slurm and can just run these PyTorch workers, um, handles, you know, increasing the total pool of GPUs when we need it and scaling down when we don't.
- 9:50
So this system is a pull-based system, and there is, you know, queues that submit to queues that pull from queues, and it is, um, [laughs] a- as, you know, Sourish and Vasu can both attest to, uh, they're here from Luma, it is the least, um, enjoyable part of working at Luma is managing these queues.
- 10:09
It is, it's everybody's, uh, least favorite task. So, um, initially when we, uh, launched Ray 2, which is a much bigger, more resource-intensive model, um, we began to actually have to deal with the, the fact that we couldn't just keep scaling up more and more compute.
- 10:28
It just wasn't economical or feasible. Um, so we had to deal with limited resources. And, um, when you deal... when you have, like, a pull-based scheduler with a bunch of queues like this, there's this concept of work starvation that, that comes into play, where essentially, you know, we've got API, uh, tier jobs, which we try to p-
- 10:49
process very quickly. We've got enterprise. We've got unlimited, plus light, free. So we've got all these different tiers that have different priority, and who gets to go first? So if you just naively process them in priority order, there may be enough enterprise or API jobs that the light jobs never get processed.
- 11:08
So we were having people waiting for, like, seven, eight, nine hours, and they were very unhappy. Um, so also shout out to Sourish. He, [laughs] actually on a weekend after we designed this system, in a couple hours implemented, um, this SLO-based system, which I'll go into, that allows us to more fairly schedule, uh, work across the, the limited
- 11:30
resources that we have. So how the system works is when you think about, um,
- 11:38
the, the concept of Work starvation. One of the typical approaches you can, you can use to manage this is aging. So the idea is the longer something waits in the queue, the higher its priority goes.
- 11:52
But what's the actual function to control that aging mechanism? Um, so we had the insight that this is a product problem, and it really comes down to how long, worst case, are you okay with different tiers waiting?
- 12:07
Um, so we have service-level objectives that the product team defines, and it kinda controls this aging behavior. And how that works is, you know, an API job, we may not want to wait in a queue more than a couple minutes.
- 12:19
But a light job, maybe we're okay with them waiting for 10 minutes. And you can configure these and then set a threshold. Once, uh, once the threshold gets hit, so say 50% of that, you know, worst-case timing, then that job gets pulled to the front of the queue.
- 12:36
And initially, uh, that worked all right, but then you can actually hit another case of work starvation, um, if, if you have a bunch of, uh, jobs that are potentially breaching the SLA, where, um, you know, a, a job with a long SLO, since it's been waiting in the queue for a longer time, will actually starve out
- 12:55
the resources of, say, an API job that you don't want to wait as long. Um, so how we handled that was with a, uh... We, we ranked the jobs by the percentage of their SLO, um, that they, that they were at.
- 13:08
So, like, if an API job's waiting for a minute, that gets treated the same as a light job that's been waiting for 10 minutes. And that actually works out really nicely in practice and results in kind of intuitive fair scheduling behaviors.
- 13:23
The last piece I wanna talk about is kinda how we actually manage all these models. Um, so this was something where I was glad that we kinda reached for Triton initially.
- 13:32
They had this nice concept of a model repo, um, that, that we've really leaned into. So every model, um, has a folder in object storage somewhere with a bunch of subfolders that have different versions.
- 13:48
Um, and so as you, you know, develop the code and fine-tune these models, deploy new checkpoints, you, uh, create a bunch of these immutable versions. And then you have a simple YAML file in the root of the model folder that lets you define which one's active.
- 14:06
So that's really nice because you can kinda re- reproducibly, um, you know, i- in these, sorry, in these versions, you can, um... You store the full Python environment, so all the dependencies needed to run the model and the checkpoints.
- 14:20
And this system's pretty nice because, you know, if you ever need to roll back or, you know, you want to run things in all these different disparate environments, you can make sure that, you know, you're running the exact same Python environment, the exact same checkpoint that you were before.
- 14:35
And very rarely are there issues that, um, you can't really solve by just rolling back. So that's been quite useful for us and, um, we've actually built, like, a automated kinda rollout system on top of this too, where when we update those YAML files, the workers will just kinda switch versions on the fly without restarting.
- 14:55
And, um, that, that helps us kind of roll out, uh, new, new model changes to the whole fleet of these thousands of, uh, H100 and, uh, AMD GPUs all at once, um, which makes managing this much more sane than the early days of parallel SSH.
- 15:12
And, um, yeah, if any of this seemed interesting to you, up your alley, you know, we're hiring. We're actively looking for cracked engineers, researchers, um, AI enthusiasts in general.
- 15:22
Please, uh, please hit us up and, uh, yeah. I just wanted to thank everybody for taking the time today and, uh, thank the team at Luma 'cause, uh, it's, it's incredible the work that everyone there does. [audience applauding]
- 15:36
We've, we've got about three minutes for questions. So let's take one or two, and let's see where we get. Any questions?
- 15:47
You spoke about the, uh, or, or the ability to deploy different chipsets, um, and PyTorch being your lowest common denominator. Could you dive into that a little bit more?
- 15:58
Because each of these chipsets have their own providers. How does, how does all that work out?
- 16:03
Yeah. So the, the kinda cheat code here is that the chip, the chip providers who we typically partner with really closely, um, they're always making sure that PyTorch, at least of, at a certain version, works for their chipset.
- 16:16
So they're doing a lot of that work for us, and then typically what'll happen is we'll try to run the models on whatever the new chipset is. Usually, it'll work, and then it'll work a bit s- more slowly.
- 16:29
So we've got actually a team of, like, 10 guys we call our Excel team that are optimizing the low-level operations within PyTorch using things like Triton and working with the chip providers to make sure that things are actually fast.
- 16:42
So you'll typically with PyTorch be able to run things, um, anywhere, but a lot of the times they won't be fast, and then we just work closely with the chipset, uh, providers to actually, you know, optimize the model over time.
- 16:55
Are you able to share who, uh, which, which chipsets you work with?
- 16:59
Yeah, yeah. We work really closely with NVIDIA and AMD and, uh, we are exploring, you know, some other providers, um, but nothing really deep yet. Like, you know, Amazon's got their own chips.
- 17:09
Uh, Groq has some chips. We just announced a big partnership with, uh, Humane, who works really closely with, with Groq. And so yeah, we're exploring some of these other chipsets, um, through, through some of these, uh, partnerships.
- 17:23
Cool. Any other questions? Yeah.
- 17:25
I'm just curious if you're comfortable saying if you run on, like, bare metal Kubernetes.
- 17:33
We do not. We... Well, I don't think so. Um, but yeah, we, we, we work with, uh, cloud providers, uh, that basically kinda give us a, um, a working Kubernetes cluster at the very least, which, which is nice.
- 17:48
Um, we're not provisioning the nodes ourself or anything. But actually would have to check with the individual providers if there's, like, VMs or if they're actually metal machines. I know, uh, when we work with Amazon, they're not, but some of the other ones might be.
- 18:00
I work with Amazon AWS, so instead of bunch of keywords- [laughs] ... I'm trying to see if you get [laughs] [laughs]
- 18:09
Cool. Yeah.
- 18:11
Yeah. Um, your video model is super impressive, but I saw on your website, uh, you're exploring, like, other use cases. Have you shipped any visual QA models yet?
- 18:21
Uh, not yet. Um, the way the, like, general application works, um, does involve, uh, video and, like, image QA models. So, um, essentially the a- the way the actual application works is we have, like, an agent that you're interacting with.
- 18:38
So when you upload a video or an image, that is actually being, you know, captioned by v- you know, VLMs to enhance the, the, the total pro- the overall prompt.
- 18:50
But, um, no, like, true VQA stuff quite yet. But there, there may or may not be some coming. [upbeat music]