GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe

All right. Hello, everyone. Uh, my name's Connor. I'm a developer advocate at Crusoe, and I'm joined by my colleagues Nikhil and Young. And we're gonna be talking today about the infrastructure we've built at Crusoe, and how it handles critical hardware errors. And so our customers, when they're running extremely large scale training workloads across thousands of GPUs, GPU failures are inevitable. And so at scale, manual remediation is completely unsustainable. And so what we'll cover today is how

by using both Slurm and Kubernetes, we can provide a robust platform with automated remediation that maintains ease of use for both machine learning engineers running jobs and the platform teams managing the underlying clusters.

So to provide a little bit more context on what Crusoe Cloud is, for those of you who haven't heard of us, um, we provide infrastructure as a service consisting of compute, storage, networking, and the latest gen GPUs from NVIDIA and AMD. But the focus of this talk will be on the tooling that we've built on top of that to orchestrate these large fleets of GPUs, including our managed Kubernetes service, called CMK, and our managed Slurm service, which is built

on top of CMK and built off of Slinky, which is SchedMD's official open source project for running Slurm on top of Kubernetes. And this allows us to get the high performance job scheduling of Slurm with the infrastructure resilience features and observability of Kubernetes, including one of the systems we've developed called AutoClusters, which is our system for automatically detecting and replacing failed GPU nodes when a GPU failure has been detected. And so

to explain how and why we landed on this architecture, it's helpful to understand where traditional Slurm both excels and falls short, which Young will get into now.

Thanks, Connor. Um, traditionally, I mean, looking at it from the trad- traditional perspective...

Sorry, can, can you hear me now? Okay, cool. I'll, I'll just do it like this. But basically, traditionally, Slurm was built, Slurm was built by the researchers for the researchers, uh, universities, labs, over twenty years ago, uh, built for a high performance computing, uh, which as we've observed with our customers and others that are using Slurm today, translates very well to modern AI, specifically around training workloads. This idea that when you're running a multi-node training

job, you need this idea of tight collective communication across the ranks that are-- that you're using, uh, including, you know, specific network built for training workloads today. Um, concepts like gang scheduling, topology awareness, uh, Prolog and Epilog for cluster validations. These are all the features that are readily available with Slurm today that helps researchers run their jobs, uh, essentially running

their jobs with tools that are already familiar to them, scripts, sbatch, srun commands, the things that are pretty standard in your Slurm cluster today. Now, where we feel like this is a bit of a shortcoming or a short-- falls a little bit short around the modern training is the things that you see on the screen right now. Um, AI training workloads today are very dynamic, and it doesn't just include training. Uh, there's a lot of other aspects that your AI

labs and oth- other companies are running as part of your, your AI workloads, including training, post-training, eval, inferences, some of which may or may not even land on your Slurm cluster today. But because of, of what we've observed as, as a GPU capacity constraints and other things, kinda limits you to how you need to be able to be more dynamic around the resources that you have. Uh, Slurm traditionally is pretty static, and it's partitioning in other, other, uh, functionalities. The

other thing I wanna mention is this idea of health checks and maintaining your nodes. Uh, you know, you can make sure your nodes are healthy or not healthy. GPUs do fail. How do you manage that? Uh, Slurm obviously has features around it to be able to, you know, re-queue the jobs, uh, detect bad nodes, uh, using things like Prolog and other open source tools that are now available to be able to do this. It just adds an operational burden, uh, that we've seen from customers

today. You know, this idea that you have to detect if nodes are bad, you have to drain the nodes when they're bad, you have to kind of work around that. That could either be manual or you need some sort of an automation or scripting to be able to do all of this. And the last thing is observability. Um, you know, jobs can fail. Jobs can also come with degraded performances for any number of reasons. I mean, GPU failure is just one of them. There's other things like network flaps and other things within your, your switches

that may be causing degraded or failed performance. Uh, Slurm knows when a job fails. It doesn't necessarily be able to tell you everything about why it failed, and that's where the observability comes into key, uh, key play. So all of this kind of encompasses the reason why we, we built this product that Nikhil's gonna go over and sort of describe some of the specifics around Managed Slurm.

Um,

all right. Uh, thanks, Young. So yeah, I'm gonna be talking about our, uh, Crusoe Managed Slurm product. And, uh, like, like we mentioned before, um, there's a lot of benefits, uh, that Slurm can bring to the, uh, training teams and, uh, researchers that are building the next bes- best models. But there are some issues with, you know, setting up Slurm, managing it, keeping it reliable. Uh, and on the other hand, you have Kubernetes, which

is the OS of the cloud. It's wide-widespread use and, uh, familiarity. It's a very mature ecosystem with lots of tools, extensions, and plugins for whatever you need to add, whether it be observability, networking, security. It helps, uh, maintain services with their, you know, built-in self-healing, uh, load balancing and auto-scaling that keeps everything running smoothly. And so we often see that, uh, customers will, uh, come with us starting with, say,

inference running on Kubernetes, and then they wanna expand to training, uh, or vice versa. And so maybe a Kubernetes-familiar team struggles with deploying and managing a Slurm cluster, um, or, uh, the incoming Slurm native training team is unfamiliar with the existing Kubernetes frameworks, uh, and adds, you know, additional friction that just slows down your team. And with, you know, how fast everyone's trying to get to, uh, the next best model and

next best product, um, teams get set up with the tools they're familiar with, and you end up with two separate infrastructure stacks, which has its own host of extra operational burdens, uh, and extra costs. So we decided to build out our Crusoe-managed Slurm, uh, on top of Kubernetes. So we have our Crusoe Slurm operator, uh, which coordinates with our Crusoe-managed Kubernetes platform and the broader Crusoe cloud, so including storage, networking, et cetera.

And, uh, CSO manages, uh, Slurm users, Slurm partitions, configuration, and storage, so you don't have to do it piecemeal. You just, you know, one, one command, you create your Slurm cluster, and there it is existing in your Kubernetes cluster. So the benefit is for the Slurm users, um, it's just another Slurm cluster. They just need, uh, an IP address. They can SSH with their, uh, user, and now to them, it's a Slurm cluster with all the bells and whistles that they need. They don't even need to know that Kubernetes is running

underneath. For your platform team and your, uh, infrastructure team, uh, the benefit is Slurm is now just another service that plugs into your existing infrastructure. You don't need to set up an entirely new infrastructure stack, new observability tools, new, you know, on-call teams, new runbooks, nothing like that. It's all built into the same thing. It runs alongside all your other services. And, um, resources in Slurm, so any jobs, make use of the, uh, built-in topology-aware scheduling,

um, and the platform team just sees it as another service. Um, and now since all your GPUs and hardware are in one cluster, they can move around seamlessly. Uh, it enables, uh, more interesting, uh, tooling and dynamic a-availability, like when you need to burst your, uh, inference service when you're reaching, uh, high, uh, users, high number of users. You can take away some of your-- or even reallocate some of your hardware from your training cluster to your inference service, or vice versa.

You know, when your user, uh, load goes down, you have all these GPUs sitting idle. That's the perfect time to up your training job or do a, a larger hero run across more GPUs. Um, this is something that's only possible when you have everything in a single, uh, infrastructure stack like, uh, Crusoe-managed Slurm. Um, an additional benefit is, uh, now Crusoe-managed Slurm can make use of our AutoClusters product, which is our auto-remediation for additional hardware errors.

So Connor is, uh, gonna walk through that and show a demo of, uh, of how that works.

So this is an overview of what exactly happens when AutoClusters detects a critical hardware error, in this case, an XID seventy-nine error, where one of the GPUs is completely unusable. And so the first thing that happens is that the user is notified, although there's no action required from them. Then the Slurm operator sets that, that specific node to down and cancels any running jobs. There's then a, um... The process rec-receives a sigterm signal before

being terminated, and you have up to two minutes to handle that sigterm to either sa-save your checkpoints or flush your logs before that node is eventually replaced. The cancel job is automatically re-queued, and then AutoClusters starts. So first, it verifies that no pods on the node have a label that disables automatic replacement. Assuming that node can be replaced, the Kubernetes node is then cordoned and drained, and then the

unhealthy node is removed from the node pool and replaced with a new healthy node from spare capacity. And then a record of the alert and the remediation is logged for the user's reference, uh, but there's still no action required from them. And then once that healthy node comes back online, then the Slurm job starts, your application code runs to load your model and lo-load your checkpoint, and resume training.

And so here's what that looks like, um,

in a demo. And so here on the left, I'm just using sbatch to launch a PyTorch training script. On the right is the Crusoe cloud console, where I have a GPU node pool of two A100 nodes. And I'm gonna launch this as normal, um, and then what I'm gonna do is run a script in-- to tell our back-end internal monitoring system that there's been an XID seventy-nine error on one of the GPUs, and this will trigger

AutoClusters. And so we can see immediately the GPU utilization drops, and so we know that that n-- that GPU has gone offline, and AutoClusters node replacement starts immediately. And the full process from detecting a critical hardware error to getting a healthy node back into the node pool takes roughly five minutes with AutoClusters. The rest of the time is mainly application-level code. And so in this example, it's a

little under fifteen minutes, um, from end to end. And here I'm just showing some of the alerts that, uh, the user receives when there's a critical hardware error like this. And so a-as you'll see, um, there's no action required from the user. The Slurm job will restart. It'll load the checkpoint and continue training where it left off. And we can see the GPU utilization returns back to normal, and the total downtime is less than fifteen minutes. And this is, um, much better

than having an engineer have to log on in the, in the middle of the night and spend hours debugging something. All of this is handled automatically on our platform.

Now, historically, creating this type of environment has been challenging to merge both Slurm and Kubernetes. And so we're really excited to announce what we call One-Click Slurm, where everything that we've talked about in this presentation, this entire environment can be provisioned with, with just a single command. And so this one command includes provisioning the underlying Kubernetes cluster, the Slurm controller, login nodes, and storage. And then adding a GPU node

pool is only one command beyond that. And so this means that engineers can simply SSH in and start running jobs immediately, and AutoClusters is, is enabled by default.

So wrapping up what we covered, um, starting with the limitations of traditional Slurm, it gives you the high-performance job scheduling, but not the automated, the automated healing that modern AI infrastructure requires. And so by building Slurm on top of Kubernetes, we can enable these advance, advanced features without having either team change, uh, their workflows. Then we covered the full node remediation flow from an error being detected to the Slurm job resuming and the

checkpoint-- and the training resuming. And so the key takeaways here, first, is that traditionally, Slurm and the infrastructure that it runs on are two separate systems. But together with Kubernetes, both systems stay in sync. And so a failed GPU node can get cordoned automatically, and that signal will propagate str- directly to the Slurm operator. Also, like I mentioned earlier, neither

the platform teams nor the machine learning teams have to change their workloads. Researchers can still use the same commands to launch Slurm, and although Kubernetes is running underneath, researchers never have to SSH into a pod or worry about containers. But the Kubernetes teams still get access to the same level of telemetry and observability that they're used to. And then lastly, like Nikhil mentioned, the GPU nodes are Kubernetes

nodes first, and that means that Slurm is just a workload running on top. And when a Slurm job finishes, those GPU nodes don't have to sit idle waiting for another job. You can schedule inference pods on those immediately through Kubernetes because from the platform's perspective, they're still just nodes in a pool.

And so the, the core idea here is that failures are inevitable, um, and so in today's AI landscape, the architecture of infrastructure should be designed in a way so that when there is a critical error, all the right actions are handled autonomously, and engineers can focus on building their application and not worrying about the infrastructure.

So hopefully, this was an informative talk. Um, if you have any questions, feel free to, um, connect with us at Booth UG-12 or on social at crusodev. And, uh, thank you for listening, and feel free to ask any questions. Thanks.