Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Talk transcript

1. 0:00

   Hi, my name is Erik [REDACTED:username]. It's great to be here virtually. Uh, who am I? I am the CEO of a company called Modal. We are based here in New York.

2. 0:10

   Most of my background is in data, AI, machine learning, and in particular, I was at Spotify for many years and built a music recommendation system there. I did leave about 10 years ago and did all kinds of other stuff in between.

3. 0:21

   But started Modal about four or five years ago during the pandemic, and the mission I had at that point was to build an infrastructure platform for data, AI, machine learning in a way that takes all the-- that, that makes it fun again to write these applications.

4. 0:40

   To, to basically to, to deploy models, scale them out, run large-scale batch jobs, making it possible to focus on writing code and not have to deal with the infrastructure.

5. 0:49

   As it turned out, GenAI was a perfect use case for this. We just didn't know it at that time. Uh, Modal is very much focused on high code use cases.

6. 0:57

   What that means is we focus on people who wanna write their own code, in particular writing their own models, but also in many cases, using existing models in a way where you have-- wanna have control over the workflow or, or other thing.

7. 1:08

   And so you can think of us more as like Kubernetes or AWS Lambda in the sense that we can run arbitrary containers or arbitrary code. We do focus on Python right now, might add other languages in the future.

8. 1:19

   Unlike system like Kubernetes, we're fully managed, so we, we run all the infrastructure. We have a big pool of thousands of GPUs and CPUs, uh, but we let you run all kinds of applications in our cloud and, uh, this could really be anything.

9. 1:35

   In that sense, we're not an AI API. We don't have one model or ten models that we put behind an API doing next token prediction. You can really run anything, which puts a little bit more onus on the developer to, to build this thing, but it also makes this a lot more powerful.

10. 1:50

    In particular, when I think about platforms and how they make you productive and makes it fun to write code, a lot of my experience is that it comes down to fast feedback loops.

11. 2:02

    So in order to make engineers fast and make them more productive, you wanna have this like super fast feedback loop that let you iterate on code very quickly. I think cloud has been a phenomenal invention and lets us build things with, you know, far more powerful things, but it's arguably a step backwards in terms of developer experience.

12. 2:19

    And, and thinking a lot about this problem, what, what I realized was in order to solve this, we had to build our own system to start containers in the cloud very fast.

13. 2:28

    'Cause if you can start containers in the cloud very fast, you can take code that r- that the user is building locally and execute in the cloud, maybe inside a custom image, running on a GPU, whatever, uh, and, and have that sort of fast feedback loop that you like when you run things c- locally.

14. 2:43

    Uh, as it turns out, solving container cold start in a distributed system is a very, very deep rabbit hole. We had to build our own scheduler, we had to build it all, all-- build our own file system, and, and many other things.

15. 2:55

    So we, we, we set out on a multi-year journey that we still haven't completed, building a lot of this very, very core, very foundational infrastructure. Modal today, you can think of it as two facets.

16. 3:04

    One is a big resource pool. We run thousands of GPUs, different types, H100s, A100s, L4s, T4s, you name it. And the only way to access those is through a Python SDK.

17. 3:16

    We might add other languages in the future, like I mentioned, but right now it's Python. And, and the reason we started with Python is obviously that Python is such a dominant language in AI, machine learning, and, and data applications.

18. 3:28

    One way to think about Modal is that it's a serverless framework that basically lets you take any Python function and turn that into serverless function. And so you do that by applying this decorator, as you can see in this code sample.

19. 3:40

    I'll show you a little bit more examples in a second. Uh, people use Modal for very large scale applications, but also small scale applications. The, the biggest use case is most likely GenAI inference.

20. 3:52

    Uh, we in particular have seen a lot of traction within Diffusion models. So for instance, AI-generated music, video, images, but also a lot of batch jobs, a lot of, for instance, processing very large scale medical images or doing computer vision on, on frames of videos.

21. 4:12

    Um, seeing a lot of traction in computational bio, things like protein folding, both things running on GPUs or, or but also CPUs. Of course, LLMs. You can't talk about GenAI without mentioning LLMs.

22. 4:23

    We have a lot of fine-tuning applications, batch embeddings, uh, of course inference as well. Uh, some of our customers, one customer I'm, um, I always think is incredibly cool is, is Suno.

23. 4:35

    They, uh, do AI-generated music, uh, and, and run a lot of their inference on Modal. Uh, but we have many other use cases for, for Modal. Uh, some running at very large scale, uh, doing all kinds of different, different applications.

24. 4:50

    Modal, it's a little bit abstract to talk about Modal without going into code, so I'm gonna do some live coding. Uh, so let's jump into a terminal, and I'll show you exactly-- uh, try to give you an idea of like what it looks like in code.

25. 5:03

    So let's look at a very, very basic Modal application. Uh, Modal, basically, one way to think about it is we take Python functions and turn them into things that run in the cloud.

26. 5:14

    Uh, there's a very simple function in it called square, which run- returns a square of a number and also prints some stuff to standard error. And this decorator that we apply, app.function,

27. 5:25

    takes that and turns that into a serverless function running in the cloud. And there's a few different ways to invoke this thing, but we have a little thing here that basically makes sure to trigger it from our laptop when we run it from the command line.

28. 5:36

    So we're gonna do that. So Modal has a little command line interface where basically it lets you run things interactively. And what happens when we run this thing is we take the code, we stick it in a container, we execute it in the cloud.

29. 5:49

    As it's executing, it streams the output back. And the whole point of this is like we wanna make it fast and feel like we're almost developing things locally. It's almost as fast as running things locally.

30. 5:58

    And this extends to things like, let's say you wanna edit this thing,

31. 6:02

    uh, and just, you know, print something else.

32. 6:05

    And in- instead of having to rebuild a container, push up the container to the cloud, download logs, et cetera, with this slow feedback loop, we just-- it just picks up the latest code, right, and rebuilds the container automatically and all these things, right?

33. 6:17

    And so while you're, like, building applications and rewriting code, you can always just run things in the cloud very, very fast. So far, this is, uh, just showcases like the sort of iteration speed, but also, let's, let's look at the power of Modal.

34. 6:30

    Like, what can you do with Modal? Like, what kinds of stuff can you... Can we get to scale? Can we run things on, on, on other types of hardware?

35. 6:36

    So let's, let's actually run this on an H100, and, and the way in Modal you do that is by saying just on the function decorator, you say GPU equals one G-- H100.

36. 6:47

    Uh, we have a bunch of other types, as I mentioned. We have A-A100s and T4s and all kinds of other ones. But let's run this on an A-H100, which is NVIDIA's flagship.

37. 6:56

    Um, and we can get access to an H100 in a couple of seconds. This is obviously not using the H100, uh, but we're running it in a container that has access to an H100.

38. 7:07

    So let's say we wanna actually access it. Now we need to probably install some software, right? So we might wanna install Torch in this case. Uh, and there's a few different ways you can do that in Modal.

39. 7:17

    You can give us a Dockerfile. You can also point to Docker image. But the easiest thing to do that is to basically define the entire compute environment in code.

40. 7:26

    So we're gonna define the con-con- the container image using Modal's Python SDK. So we're gonna say image equals modal.Image.debian_slim() as the base image, and we're gonna pip install torch.

41. 7:39

    And then we're gonna use this image on this function, and we're gonna import torch. And just to show that it works, we're gonna print

42. 7:47

    torch.cuda.getdevice_name(). And hopefully this works when I run this. We'll delete this line. And when I run this thing, hopefully it will print something like we're running on an H100.

43. 8:00

    And, um, as you can see, it's still very fast, but slightly slower this time, uh, because loading Torch takes a little bit of extra overhead. And we'll talk about in a second what we've done to, to reduce that overhead, but, but it takes maybe about a second to initialize Torch.

44. 8:17

    Uh, okay, cool. So now we can run stuff on H-H100s. Let's try to run things on a lot of H100s. Uh, and so let's try to scale things out a little bit.

45. 8:27

    Uh, in Modal, any function can... You can map over any function in Modal just in code. So instead of calling just a single function invocation, we're gonna fan out and do a thousand, or maybe let's do ten thousand function invocation.

46. 8:40

    And you can do this in code by just saying we're gonna map over five thousand. I said ten thousand, actually, so let's do that. And we're gonna unpack the, the iterator, and let's, um, print x just to show some progress.

47. 8:57

    And what Modal does when you fan out is that it's gonna spin up as many containers as possible. And so you can see we're already running five containers, six containers, eight containers.

48. 9:08

    Uh, it makes it very easy to, to fan out and start, you know, even hundreds of containers or even thousands of containers running on GPUs. If we keep this running for several minutes, we can easily scale up to very large, um, number.

49. 9:21

    Um, so this gives you basically the ability to take something like, you know, that needs a lot of compute and, you know, something like a, a batch job and fan out, spin up thousands of containers, parallelize over it, and, and get results much faster.

50. 9:37

    Uh, we're gonna s- take a look at the

51. 9:40

    UI for a second. Uh, Modal also has a UI, uh, that you can access if you go to the, uh, website. Uh, the, the URL is printed in the console.

52. 9:52

    So let's take a look at that. Uh, so we can see the app details in our UI. There's all kinds of interesting things here. Modal has a pretty rich UI that lets you see container metrics, logs, uh, lets you set up users, and many other things.

53. 10:08

    Uh, so if you zoom in, for instance, on the number of containers, we can see here we spun up eighteen containers at peak. As I mentioned, if we had kept going, we would reach a much larger number.

54. 10:17

    Uh, we got up to eighteen containers at this point. Can look at the CPU utilization, GPU, et cetera. Um, could look at GPU temperature, thirty-three Celsius. Uh, even the, the watt consumption.

55. 10:30

    So there's a lot, a lot of other things here. We can look at app logs and many other things. Um,

56. 10:36

    okay, let's switch back to the terminal for a second and see some other stuff. There's a lot of stuff, so I'm not gonna go into every single possibility of how to use Modal.

57. 10:48

    But one thing I didn't show that I think is interesting and very valuable is you can also deploy these things. So, so far we only showed how to run things interactively, which means we have sort of, you know, we run things from our laptop.

58. 11:00

    But if I take this code and deploy this using modal deploy,

59. 11:06

    uh, we get this persistent endpoint. And what's nice about that is now we have this thing we can call from any other context in Python. And I'm just gonna show this using my REPL.

60. 11:16

    If we import modal, and if we do lookup like this,

61. 11:22

    uh, we get this handle to this remote function. So let's call this, and the first time we're gonna call it, we're gonna have incur a cold start. So it's gonna take a couple seconds 'cause the container has to start up.

62. 11:32

    And remember, we're, we're importing Torch, and we're running this on an H100, so it takes a little bit of extra time. The container keeps running for a few, for, for, for sixty seconds by default, and then it shuts down.

63. 11:41

    So the... Now it's actually idle. So if we call this again, typically it'll be a little bit faster. And, uh, we're obviously, you know, wasting an enormous amount of, uh, FLOPS using a GPU to calculate the square root of a number.

64. 11:53

    Uh, but, but this showcases, you know, how you can easily take things and deploy it, uh, even on very powerful hardware and, uh, and build these serverless endpoints. For instance, doing inference and, and, you know...

65. 12:04

    And Modal handles all the scaling. So when you invoke this function multiple times, we'll just scale up using more and more containers and shut down. Um- And many other things you can do with Modal.

66. 12:14

    You can, uh, set up distributed file systems that you can mount to each container, so you can, like, exchange information using the file system. You can s-set up web endpoints, you can set up cron jobs, and many other things.

67. 12:27

    Uh, so this hopefully gives you a little bit more of an idea of, like, what Modal looks like from an engineering perspective. Like, what does it look like when you're interacting through code with Modal?

68. 12:36

    Let's talk a bit about how Modal works under the hood. And as I mentioned, Modal, in order to deliver on this developer experience that I always wanted to have, we had to go down this very deep rabbit hole and build a lot of custom infrastructure ourself, and that's the only way we felt that we can make it

69. 12:52

    fast enough. We couldn't use Kubernetes, we couldn't use Docker, so we had to build a lot of this stuff ourselves. And it should be pointed out we're standing on shoulders of giants here.

70. 13:01

    Uh, we're using a fantastic container runtime called gVisor, uh, that gives us isolation, but we had to build a lot of stuff around it. Uh, we had to build our own scheduler and many other things, but we're obviously using a lot of the existing things in Linux and, and other systems, and we're using fantastic cloud tools as

71. 13:18

    well. Uh, in order to deliver the developer experience that we wanted to, as I mentioned, and the feedback loop that we wanted too, we had to figure out container cold start.

72. 13:27

    And container cold start, starting containers fast in a distributed system is a hard problem. So let's talk about what containers are to start with. Containers are, and this is my super crude, unfair generalization of what a container is or a container image, it's basically two things.

73. 13:44

    It's a root file system, so that's like the slash that you have in, in Linux that contains all the data on your drive. And then it's a bunch of stuff to iso-isolate processes so they can't tamper with each other.

74. 13:56

    Uh, there are many inefficiencies with how container images are stored and how container images are transferred. In particular, one of the issues is that there's a lot of junk.

75. 14:08

    There's a lot of stuff we're never gonna read. Like, many container images has Perl installed by default, man pages, locale information, time zone information for Uzbekistan. You're never gonna read this stuff, so we're sending all this data back and forth and, and, and the, the core thing here is we wanna start containers on a remote file, on

76. 14:25

    a remote worker very fast. We wanna minimize the amount of data that has to be transferred. We wanna do as little as possible. The other inefficiency is that there's a lot of redundancy in this.

77. 14:35

    Uh, a lot of the files that are being transferred back and forth are actually the same files. So if you grab just, like, three very different, uh, container images like I did in this case, and you look at the files, it actually turns out to be mostly the same files to a very large extent.

78. 14:49

    So with those two tricks, with those two observations, there's a number of tricks we can do and, and we... So we built what's called a content address storage. And this is not a new invention.

79. 14:59

    This is not something we came up with, but it's rarely used in production systems. Notably, AWS Lambda actually uses the same technique. And the idea is that instead of storing the images directly, we store the images, the container images, as just a bunch of metadata that points to blobs.

80. 15:18

    And for each blob, we compute a checksum or a hash value, and then we use that to deduplicate all the blobs, 'cause there's an enormous amount of redundancy in these blobs.

81. 15:29

    And this means the container images themselves are actually just little pieces of metadata, and in many cases, we can cache a very large percentage of the container images, and we can also avoid pulling data that we, we're not gonna need by lazy loading a lot of the data on access.

82. 15:49

    Uh, this is tricky because container cold start, in particular with Python, is very latency sensitive because we end up doing a lot of very sequential file accesses. So in many cases, when a container starts up in Modal or, or in any, in Python in any case, uh, it requires reading every single module, every single Python module, which

83. 16:12

    is many, in many cases it ends up being several thousand Python modules. Each one of them requires accessing the file system, and so what we can't allow is that to take several milliseconds, 'cause if you're doing something that takes several milliseconds and you're doing it a thousand times, it ends up taking several seconds, and we wanna avoid

84. 16:31

    that. So there's a lot of tricks that we have to do in order to basically get this down below a second. We do a lot of prefetching. We do a lot of task tracing.

85. 16:40

    We look at, you know, historical runs and see what types of files was accessed last time it ran. And then building these containers is, is obviously also another whole challenge.

86. 16:51

    We, we basically built our own container image builders. Uh, another technique that we're also more recently started leveraging is we can snapshot the CPU memory. So we talked about how we snapshot the container images, and we, we, we, we cache a lot of the data, which means, like, when you're loading it, you don't have to fetch a

87. 17:09

    lot of data. But what if we can avoid loading the data in the first place? What if we can just, like, revert to the, the memory state, the CPU memory, the RAM of, of, uh, a container?

88. 17:20

    And as it turns out, gVisor actually supports this, and that's another way that arguably supersedes a lot of the previous stuff. Uh, in practice, they end up kind of both reinforcing this, this container cold start.

89. 17:31

    But this lets us cut down even more dramatically, uh, th-things like Stable Diffusion we can now start in, in a couple of seconds, even though it involves loading very, very large, uh, model weights like, you know, five or 10 gigabytes.

90. 17:44

    Uh, we're also looking at GPU snapshotting, which will make things even more, even faster, which is very exciting. Um, and so doing all these things, you know, owning the entire stack, owning the file system, you know, building a storage system.

91. 18:00

    I didn't talk about the storage system. We, we basically use R2, and we run in many different regions, and so we use both the CDN and the, the, the R2.

92. 18:08

    And, and, and so all these optimizations together means we- Kind of solved the problem of container cold start. And let's remember, what it-- why did we originally set out to start this thing is because we wanted to deliver good developer experience.

93. 18:24

    As it turns out, it's good for other things too. So container cold start is also good because it enables serverless. So what does serverless mean? Uh, it means a lot of different things.

94. 18:35

    I think part of l- why sometimes I avoid the term serverless is that it has so many different definitions. But the, the promise of serverless was always don't provision more than you actually need.

95. 18:46

    Just, just, you know, only ch- pay for capacity you're actually using. And so especially with GPUs, which are very expensive, as it turns out, you can pack-- take a lot of different users, pull them together, and give people dynamically the resources they need and get dramatically better utilization.

96. 19:09

    And so that in turn means we can get lower cost. It means there's no capacity planning. It, it also, because we can pull a lot of these users, the, the variance, the, the total variance goes down, the relative variance, uh, which means we can run a much more predictable set of, of resource pools, uh, the, the total

97. 19:28

    capacity that we run. Uh, which is another problem, by the way. So we need to run thousands of GPUs. We use a lot of different cloud vendors. We use a lot of different regions.

98. 19:37

    We scale up and down continuously. Uh, in fact, we actually end up solving a mixed integer programming problem to, to do this, uh, minimizing the total cost spend. Uh, and, and this is some of the stuff we have to do for, for our customers so they don't have to think about it.

99. 19:51

    So through Modal, you can come in, you can request 100 GPUs. Under the hood, there's enormous amount of work that we had to put in in order to get the capacity somewhere in the world, uh, you know, spinning up GPUs if needed.

100. 20:04

     Uh, but in many cases, it happens instantaneously because we can maintain a buffer that makes it very fast to get access to these compute resources for any customer. Um, this was very technical, but just to kind of go back and look at a high level again, uh, why do people like Modal?

101. 20:21

     People pick Modal in-- because they can run their own code. We're not an AI API, so to speak. You can run almost anything with Modal. Uh, we make it possible to iterate very quickly.

102. 20:32

     We're fully usage based. So when you run things in Modal, you only pay for the time the containers are actually active. You have to never think about capacity. You don't have to, you know, go out and buy, you know, hundreds of GPUs or thousands of GPUs.

103. 20:45

     We can get you that within, you know, seconds or at least minutes. Uh, so there's a lot of things, the sort of burden of infrastructure, building your own internal platform, setting up Kubernetes, setting up, you know, Docker and all these things.

104. 20:57

     You don't have to think about this with Modal.

105. 21:00

     Um, how do you try Modal? It's actually very simple. Uh, you go to your terminal and you do pip install modal. The, the Python client automatically, you know, configures itself to connect to, uh, Modal, and you can immediately start running stuff because we give everyone $30 a month, uh, per, per month of free credits.

106. 21:17

     If you are a startup, uh, we can give you up to $50,000 in credits in order for you to get started.

107. 21:25

     Thank you, and I really hope you enjoyed this. And if you have any questions, feel free to reach out, [REDACTED:email_address]. Uh, you can also follow me on Twitter, [REDACTED:username], or check out my blog, erikbern.com.