Covalent Launch: The GPU Cheatcode: Fine-tune 20 Llama Models in 5 Minutes

Talk transcript

1. 0:00

   [on hold music]

2. 0:13

   So the talk is actually going to be about, um, uh, how you run things extremely easy directly from Python. And the example that I'm going to show you here is obviously I just have five minutes in on my end, but I'm going to try my best to showcase how you can fine-tune pretty much, twenty is an arbitrary

3. 0:29

   number here, but hundreds of models that you can do right from Python without needing anything like Kubernetes, Docker, or anything on your side in. Uh, so before that, you can find the talk and the actual code for what I'm gonna do in this QR code, and you'll find a lot more interesting examples over there to try out

4. 0:45

   and run as well. Um, okay. So what do we do? Um, so Covalent is an open source/open core, uh, product on its end. And what we do is we help people write Python locally and ship the code to any kind of compute backend that you need to send it to.

5. 1:03

   So what that means is, hey, you have a Python function that you want to run on a GPU, um, in your local laptop, open up a Jupyter Notebook, add a single decorator on top to say, "Hey, I want to run this on H100 with thirty-six gigs of memory for two days maximum time limit," and press Shift+Enter in

6. 1:19

   your Jupyter Notebook, and that's it. The code gets shipped to a backend in a GPU, and you get back the result on your side in. In the open source case, it sends it to your own compute.

7. 1:28

   You can attach your own compute cluster, and it runs over there. In the cloud case, it runs in our GPU cluster, and you just pay for the GPU time that it runs in.

8. 1:36

   So it runs for five minutes, you pay for five minutes of H100. It runs for ten seconds, you pay for ten seconds of H100s on your side in. You can also bring your own compute and attach to us, and we'll help you orchestrate the entire compute that you're handling it on your side, be it your own cloud

9. 1:49

   or on-prem systems or whatever it is on your end in.

10. 1:53

    Okay. So, uh, Covalent basically has a bunch of parameters that you define in. You can submit in jobs, which are called single functions. So essentially all you need to do is, as I said, add a single decorator on top and say, what is the compute that you need to ship it to?

11. 2:07

    It goes there, it runs, and you get back the Python object back, and you just pay for the function that you are running in.

12. 2:13

    We also let you run inferences, and again, it's completely Pythonic. You don't Dockerize, you don't run Kubernetes cluster, you don't do anything. You just say, "Hey, I have an initializer function, and I have a-- I need an endpoint called /generate."

13. 2:27

    And you define your Python functions. You click a single cc.deploy command, uh, in your Jupyter Notebook. The entire service gets shipped to us, and we scale. You get back an API endpoint that scales to zero or scales in its requests as and when your new request comes in.

14. 2:41

    You can define your custom auto-scaling mechanism. Like, "Hey, I want to auto-scale it to ten GPUs exactly at nine o'clock every day," or, "I wanna auto-scale whenever my GPU utilization hits in eighty percent," or, "I wanna auto-scale whenever the number of requests I get in is thousand."

15. 2:56

    Uh, so you can define whatever auto-scaling you want. You can define authentication and everything, and everything happens in the background for you. You don't even touch a single code of Kubernetes or Docker or anything on your side in.

16. 3:07

    And the talk I'm gonna give in is a very tiny example, um, of what we do from our side. But if you go to this link in, there's a whole host of examples, uh, that you can run in right from real-time time series analysis to, uh, you know, using inverter transformers for time series, which is like a

17. 3:25

    state-of-the-art, uh, time series transformer on its end, uh, running in large systems, um, large language models on your serving, uh, systems, and even building an entire AI model foundry out of our just pure Pythonic code, uh, on your side in.

18. 3:40

    So without further ado, I'll quickly run through the code example of how you do essentially fine-tune a bunch of huge set of models, uh, directly just from Python on your end.

19. 3:51

    And I'll also show you how it looks like, uh, from the front-end side as well. So, um, it's rather simple. All you do is I have written a bunch of, uh, normal Pythonic training functions in my local package called fine-tune and evaluate on our end.

20. 4:06

    And what we are going to do is, hey, I'm going to define a Python task, which essentially calls in my fine-tune function, which is going to accept a model and data and return back a fine-tune model.

21. 4:15

    So this is a simple Python function, and I'm gonna say, hey, I want to run it on a twenty-four core CPU with one GPU in it of type H100 with forty-eight gigs of memory, and going to give a max limit of eighteen hours on it.

22. 4:27

    And then I'm gonna say, hey, I'm gonna-- Once the model is done, I'm gonna accept the model and then evaluate its accuracy on its end. And finally, I'm going to just sort the model among all the best models and then pick the best model in it.

23. 4:39

    And I want this to run on a CPU-based machine. I don't wanna waste GPU for my sorting or whatever I'm gonna do in on my end. And finally, I'm gonna deploy the best model that I figured, um, in its-- that has performed well on my end.

24. 4:51

    And this is like in a simple decorator to add to say, "Hey, this is my initialization service, and I'm gonna create an endpoint called /generate. Um, and I'm going to generate the text and give back the prediction."

25. 5:04

    And finally, this is where the magic happens. To tie together all of these things, what I do is I'm gonna create a workflow where I'm pretty much just going to simply loop over a bunch of models to fine-tune, call the fine-tune function, evaluate the task and get the accuracy, make a list of all the models and accuracy,

26. 5:20

    sort the best models, and then deploy the model from my end. And this is completely Pythonic. And once you dispatch this to our server, which is essentially calling a single line over here, what you will go back and see is a new job that creates in our application, and all of the functions that you called will run

27. 5:39

    in the respective devices that you just defined. So for instance, here is, uh, one of the evaluation step that ran in, and it has its own machine that we ran in.

28. 5:49

    It ran in L14. It ran for six minutes, and you get back just eighty-seven cents to evaluate your model in. Another model ran in V100 on its end, and it ran for six minutes again.

29. 6:02

    It costed eleven cents to do it in. And in total, you finally have deployed, tri-- fine-tuned, untrained completely in Python without needing anything like Docker or Kubernetes on your end.

30. 6:13

    And we have a booth over there. Do visit us, and we can have more chat over there. Thank you, guys. [outro music]