AI Engineer Summit 2025
Keynote: The AI developer experience doesn't have to suck – why and how we built Modal
Read the talk
Making cloud AI development feel local
Modal’s edit–run loop connects Python functions to remote GPUs, but making that loop fast requires rethinking container storage, initialization, and shared capacity.
Before you start: Familiarity with Python functions, decorators, and basic container concepts will help you follow the examples.
Write the application, not its infrastructure
How do you deploy models, scale them out, and run large batch jobs without spending your development time managing infrastructure? Erik Bernhardsson, who previously built Spotify’s music recommendation system, describes that as the founding problem for Modal. He started the company during the pandemic, four or five years before this talk, to make building data and machine-learning applications enjoyable again.
The intended user wants to write code: perhaps a custom model, perhaps a workflow around an existing one. Modal is closer to Kubernetes or AWS Lambda than to a fixed-model inference API: it runs arbitrary code and containers, while managing the underlying pool of thousands of GPUs and CPUs. The tradeoff is straightforward. Developers retain control over the application, but must implement more of it themselves. Python is the supported language in the platform described here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast feedback starts below the SDK
A fast edit–run loop depends on fast container startup. Cloud infrastructure provides more powerful machines, but putting those machines behind image builds, uploads, and remote execution can make development feel slower than working locally. Modal’s goal is to take locally authored code, execute it remotely in the right image and on the right hardware, and return feedback quickly enough to preserve the rhythm of local development.
That requirement led to a custom scheduler and file system, part of a multiyear infrastructure effort Bernhardsson describes as still unfinished. The resulting platform has two faces: a resource pool containing H100, A100, L4, and T4 GPUs, and a Python SDK that exposes that pool. Python was the starting point because of its dominance in AI, machine learning, and data work. At the application boundary, a decorator turns a Python function into a serverless function.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One execution platform, many workloads
Bernhardsson identifies generative inference as probably Modal’s largest use case, but the workload range extends well beyond serving language models:
- Diffusion and media: generating music, video, and images.
- Batch processing: processing medical images and applying computer vision to video frames.
- Computational biology: protein folding and other workloads on GPUs or CPUs.
- Language models: fine-tuning, batch embeddings, and inference.
Suno is the concrete customer example: Bernhardsson says it runs much of its AI music inference on Modal. The common requirement is flexible compute under application code, rather than a single prescribed model API.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A square function crosses the cloud boundary
The live demonstration starts with a deliberately small application: square returns a number’s square and writes a message to standard error. The app.function decorator makes it remotely executable; a local entrypoint triggers it from the laptop. The essential structure is:
python
import sys
import modal
app = modal.App("square-demo")
@app.function()
def square(x: int) -> int:
print(f"Squaring {x}", file=sys.stderr)
return x * x
@app.local_entrypoint()
def main():
print(square.remote(12))
The arithmetic stays ordinary Python. The decorator and .remote() call establish where it runs.
With that file saved as square_demo.py, the interactive command is:
bash
modal run square_demo.py
Modal packages the code into a container, executes it in the cloud, and streams output back to the terminal. Bernhardsson then changes a print statement and runs it again. The new code is picked up and the container rebuilt automatically, removing the manual cycle of rebuilding an image, pushing it, and retrieving logs. He describes the experience as nearly local in speed; the demonstration emphasizes iteration rather than presenting a timed comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Request hardware and define its environment
The next change requests an H100 through the function decorator. A100s and T4s are other available choices. Bernhardsson reports getting access to an H100 in a couple of seconds in this demonstration. Merely attaching a GPU does not move Python arithmetic onto it: x * x still does not use the accelerator.
To inspect the device, the function needs PyTorch installed. Modal accepts a Dockerfile or an existing Docker image, but the demonstration defines the environment directly in Python: start with modal.Image.debian_slim(), install torch, and attach the image to the function. Extending the same example gives:
python
import sys
import modal
app = modal.App("square-demo")
image = modal.Image.debian_slim().pip_install("torch")
@app.function(image=image, gpu="H100")
def square(x: int) -> int:
import torch
print(torch.cuda.get_device_name(), file=sys.stderr)
print(f"Squaring {x}", file=sys.stderr)
return x * x
@app.local_entrypoint()
def main():
print(square.remote(12))
torch.cuda.get_device_name() returns the CUDA device’s name. It establishes hardware access without pretending that squaring an integer is useful GPU work. Bernhardsson estimates roughly one second of Torch initialization overhead in this run. That remaining initialization cost becomes important later.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Map the function, then inspect the actual scale
Once a function can run remotely, batch parallelism becomes a change in how it is called. After considering smaller batches, Bernhardsson settles on 10,000 invocations. Replace the local entrypoint with a mapping call and consume its result iterator:
python
@app.local_entrypoint()
def main():
for result in square.map(range(10_000)):
print(result)
Modal fans the work out across containers. During the ramp, Bernhardsson points out five, then six, then eight running containers. He describes hundreds or thousands of GPU containers as possible with a longer run; that larger scale is not reached in this demonstration. The mechanism is the useful part: independent batch inputs become parallel function invocations without application code explicitly provisioning each worker.
The console also prints a link to the application dashboard. There, the run can be inspected through app details, container metrics, and logs; the UI also includes user management. The demonstrated run peaks at 18 containers. Bernhardsson then examines CPU and GPU utilization, a GPU temperature reading of 33°C, and power consumption. These views connect the convenient mapping call to the resources it actually created.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the function available, not the container
Interactive runs are initiated from the laptop. Deployment makes the function persistently addressable from another Python context:
bash
modal deploy square_demo.py
In the recording, Bernhardsson opens a REPL, imports modal, and uses a lookup operation to obtain a remote function handle. The important distinction is between a persistent callable endpoint and a permanently running container. The deployed function remains available even when its execution container has shut down.
The first deployed call incurs a cold start, including Torch import in an H100-backed container. Bernhardsson describes a default 60-second idle retention period before shutdown. A subsequent call can reuse the warm container and is typically faster. The tiny arithmetic workload still wastes the GPU’s compute capacity; it demonstrates deployment and lifecycle behavior, not acceleration. More invocations can create more containers, followed by scale-down.
The same platform also supports distributed file systems mounted into containers for exchanging information, web endpoints, and cron jobs. Remote function calls are therefore one interface to a broader application environment, rather than the only way to organize work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start containers by moving less data
The infrastructure explanation begins with the speed requirement. Bernhardsson says Kubernetes and Docker could not deliver the desired startup behavior for Modal’s internal execution stack, so the team built its own scheduler and surrounding machinery. That does not conflict with accepting Docker images as inputs. The execution system uses gVisor for isolation, alongside existing Linux facilities and cloud tools.
For this explanation, a container can be reduced to two pieces: a root file system, and process isolation that prevents workloads from interfering with one another. The file-system side contains two opportunities to avoid work. First, an image may contain data the application never reads—Perl, manual pages, locale information, or Uzbekistan time-zone data. Second, different images contain many identical files. Bernhardsson’s comparison of three otherwise different images finds extensive overlap. Moving every byte of every image independently wastes both bandwidth and startup time.
Modal uses content-addressed storage to exploit that overlap. An image becomes metadata pointing to blobs; each blob is identified by a checksum or hash, allowing identical content to be deduplicated. Bernhardsson notes that AWS Lambda uses the same broad technique.
| Observation | Storage response | Startup benefit |
|---|---|---|
| Images share identical content | Address blobs by content hash | Reuse cached blobs across images |
| Many files are never read | Load data on access | Avoid unnecessary transfers |
| Images can reference shared blobs | Store image metadata separately | Keep the image description small |
Caching and lazy loading solve different parts of the problem: caching avoids fetching content already nearby, while lazy loading avoids fetching content that is not needed at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predict reads, then bypass initialization
Lazy loading introduces another problem: Python startup can require reading several thousand modules, with many file accesses occurring sequentially. A small delay on each read accumulates. Bernhardsson explains that millisecond-scale accesses repeated a thousand times can add seconds to startup. Saving transfer volume is not enough if each required file introduces another remote round trip.
Modal therefore uses prefetching and task tracing. Historical runs reveal which files were accessed, allowing the system to anticipate those reads on a later start. Bernhardsson describes these techniques as part of the effort to get startup below a second. They preserve demand loading’s avoidance of unnecessary data while reducing the latency of predictable reads.
The next step goes beyond fetching files efficiently. Modal builds its own container images, but also snapshots CPU memory. Instead of loading data and repeating initialization, it can restore the container’s RAM state through gVisor’s checkpoint/restore support. Image caching and memory restoration reinforce one another: one makes the underlying data available cheaply; the other avoids recreating already initialized process state.
Bernhardsson reports starting Stable Diffusion in a couple of seconds despite roughly 5–10 GB of model weights. The talk does not specify the model version, hardware, cache conditions, or latency percentile for that claim. GPU snapshotting is still under investigation at this point in the recording. CPU memory restoration should therefore not be read as a claim that the demonstrated system restores GPU state.
These optimizations sit on a distributed storage system that Bernhardsson says uses R2 and a CDN across multiple regions. Together, the scheduler, file system, caching, prefetching, and snapshots bring the discussion back to the original goal: making remote execution fast enough to support an enjoyable development loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast startup changes the economics of shared GPUs
Fast cold starts also make serverless resource allocation more practical. Bernhardsson frames the promise as provisioning only what is needed and paying for used capacity. With expensive GPUs, pooling many users and assigning resources dynamically can improve utilization and reduce cost. Pooling also reduces relative demand variance, making aggregate demand more predictable than individual customers’ needs.
The capacity-planning problem moves to the provider. Modal operates thousands of GPUs across cloud vendors and regions, scaling its fleet up and down. Bernhardsson says the system solves a mixed integer programming problem to minimize total capacity spend. A request for 100 GPUs may require acquiring capacity somewhere in the world and starting machines; a maintained buffer can instead satisfy the request immediately. The simple function interface rests on continuous procurement and allocation work beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an infrastructure platform to an install command
The resulting contract is to run your own code, iterate quickly, and let the provider manage capacity. Bernhardsson describes billing as covering the time containers are active, and says hundreds or thousands of GPUs can become available within seconds or minutes. Active-container billing is not the same as billing only while useful computation runs: retained idle resources can still incur charges. The developer avoids operating an internal platform built around Kubernetes, Docker, and GPU capacity planning.
The talk closes with pip install modal and describes the client as configuring its connection automatically. For a present-day setup, the installation guide also requires an account and authentication:
bash
pip install modal
modal setup
The recording’s Python-only access and REPL lookup belong to its SDK era; current documentation also supports JavaScript/TypeScript and Go clients for calling Functions and managing resources, while Python remains primary for implementing Functions. At the talk, Bernhardsson offers $30 per month in free credits and up to $50,000 in startup credits. Those are the recorded offers, rather than a statement of current startup eligibility. The long infrastructure journey ends at a deliberately small entry point: install the client, write a function, and let the platform supply its execution environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official documentation for saving and restoring container execution state in gVisor.
Further reading
Engineering explanation of content-addressed images, FUSE, shared caches, and the latency costs of fetching container data.
Explains CPU snapshot restoration, its interaction with lazy loading, and measured startup improvements for example workloads.
A July 2023 systems paper describing Lambda's container storage, caching, deduplication, and demand-loading architecture.
Modal's customer case study describes how Suno moved from batch preprocessing to deployed inference and autoscaling.
Updates since the talk
Current setup instructions and a complete Python example defining an image, requesting a GPU, and running code remotely.
Current snapshot configuration, lifecycle hooks, and limitations, including alpha GPU snapshot support.
Explains queueing, initialization, warm-container settings, and the cost of retaining idle resources.
Read the complete timestamped transcript
- 0:00
Hi, my name is Erik [REDACTED:username]. It's great to be here virtually. Uh, who am I? I am the CEO of a company called Modal. We are based here in New York.
- 0:10
Most of my background is in data, AI, machine learning, and in particular, I was at Spotify for many years and built a music recommendation system there. I did leave about 10 years ago and did all kinds of other stuff in between.
- 0:21
But started Modal about four or five years ago during the pandemic, and the mission I had at that point was to build an infrastructure platform for data, AI, machine learning in a way that takes all the-- that, that makes it fun again to write these applications.
- 0:40
To, to basically to, to deploy models, scale them out, run large-scale batch jobs, making it possible to focus on writing code and not have to deal with the infrastructure.
- 0:49
As it turned out, GenAI was a perfect use case for this. We just didn't know it at that time. Uh, Modal is very much focused on high code use cases.
- 0:57
What that means is we focus on people who wanna write their own code, in particular writing their own models, but also in many cases, using existing models in a way where you have-- wanna have control over the workflow or, or other thing.
- 1:08
And so you can think of us more as like Kubernetes or AWS Lambda in the sense that we can run arbitrary containers or arbitrary code. We do focus on Python right now, might add other languages in the future.
- 1:19
Unlike system like Kubernetes, we're fully managed, so we, we run all the infrastructure. We have a big pool of thousands of GPUs and CPUs, uh, but we let you run all kinds of applications in our cloud and, uh, this could really be anything.
- 1:35
In that sense, we're not an AI API. We don't have one model or ten models that we put behind an API doing next token prediction. You can really run anything, which puts a little bit more onus on the developer to, to build this thing, but it also makes this a lot more powerful.
- 1:50
In particular, when I think about platforms and how they make you productive and makes it fun to write code, a lot of my experience is that it comes down to fast feedback loops.
- 2:02
So in order to make engineers fast and make them more productive, you wanna have this like super fast feedback loop that let you iterate on code very quickly. I think cloud has been a phenomenal invention and lets us build things with, you know, far more powerful things, but it's arguably a step backwards in terms of developer experience.
- 2:19
And, and thinking a lot about this problem, what, what I realized was in order to solve this, we had to build our own system to start containers in the cloud very fast.
- 2:28
'Cause if you can start containers in the cloud very fast, you can take code that r- that the user is building locally and execute in the cloud, maybe inside a custom image, running on a GPU, whatever, uh, and, and have that sort of fast feedback loop that you like when you run things c- locally.
- 2:43
Uh, as it turns out, solving container cold start in a distributed system is a very, very deep rabbit hole. We had to build our own scheduler, we had to build it all, all-- build our own file system, and, and many other things.
- 2:55
So we, we, we set out on a multi-year journey that we still haven't completed, building a lot of this very, very core, very foundational infrastructure. Modal today, you can think of it as two facets.
- 3:04
One is a big resource pool. We run thousands of GPUs, different types, H100s, A100s, L4s, T4s, you name it. And the only way to access those is through a Python SDK.
- 3:16
We might add other languages in the future, like I mentioned, but right now it's Python. And, and the reason we started with Python is obviously that Python is such a dominant language in AI, machine learning, and, and data applications.
- 3:28
One way to think about Modal is that it's a serverless framework that basically lets you take any Python function and turn that into serverless function. And so you do that by applying this decorator, as you can see in this code sample.
- 3:40
I'll show you a little bit more examples in a second. Uh, people use Modal for very large scale applications, but also small scale applications. The, the biggest use case is most likely GenAI inference.
- 3:52
Uh, we in particular have seen a lot of traction within Diffusion models. So for instance, AI-generated music, video, images, but also a lot of batch jobs, a lot of, for instance, processing very large scale medical images or doing computer vision on, on frames of videos.
- 4:12
Um, seeing a lot of traction in computational bio, things like protein folding, both things running on GPUs or, or but also CPUs. Of course, LLMs. You can't talk about GenAI without mentioning LLMs.
- 4:23
We have a lot of fine-tuning applications, batch embeddings, uh, of course inference as well. Uh, some of our customers, one customer I'm, um, I always think is incredibly cool is, is Suno.
- 4:35
They, uh, do AI-generated music, uh, and, and run a lot of their inference on Modal. Uh, but we have many other use cases for, for Modal. Uh, some running at very large scale, uh, doing all kinds of different, different applications.
- 4:50
Modal, it's a little bit abstract to talk about Modal without going into code, so I'm gonna do some live coding. Uh, so let's jump into a terminal, and I'll show you exactly-- uh, try to give you an idea of like what it looks like in code.
- 5:03
So let's look at a very, very basic Modal application. Uh, Modal, basically, one way to think about it is we take Python functions and turn them into things that run in the cloud.
- 5:14
Uh, there's a very simple function in it called square, which run- returns a square of a number and also prints some stuff to standard error. And this decorator that we apply, app.function,
- 5:25
takes that and turns that into a serverless function running in the cloud. And there's a few different ways to invoke this thing, but we have a little thing here that basically makes sure to trigger it from our laptop when we run it from the command line.
- 5:36
So we're gonna do that. So Modal has a little command line interface where basically it lets you run things interactively. And what happens when we run this thing is we take the code, we stick it in a container, we execute it in the cloud.
- 5:49
As it's executing, it streams the output back. And the whole point of this is like we wanna make it fast and feel like we're almost developing things locally. It's almost as fast as running things locally.
- 5:58
And this extends to things like, let's say you wanna edit this thing,
- 6:02
uh, and just, you know, print something else.
- 6:05
And in- instead of having to rebuild a container, push up the container to the cloud, download logs, et cetera, with this slow feedback loop, we just-- it just picks up the latest code, right, and rebuilds the container automatically and all these things, right?
- 6:17
And so while you're, like, building applications and rewriting code, you can always just run things in the cloud very, very fast. So far, this is, uh, just showcases like the sort of iteration speed, but also, let's, let's look at the power of Modal.
- 6:30
Like, what can you do with Modal? Like, what kinds of stuff can you... Can we get to scale? Can we run things on, on, on other types of hardware?
- 6:36
So let's, let's actually run this on an H100, and, and the way in Modal you do that is by saying just on the function decorator, you say GPU equals one G-- H100.
- 6:47
Uh, we have a bunch of other types, as I mentioned. We have A-A100s and T4s and all kinds of other ones. But let's run this on an A-H100, which is NVIDIA's flagship.
- 6:56
Um, and we can get access to an H100 in a couple of seconds. This is obviously not using the H100, uh, but we're running it in a container that has access to an H100.
- 7:07
So let's say we wanna actually access it. Now we need to probably install some software, right? So we might wanna install Torch in this case. Uh, and there's a few different ways you can do that in Modal.
- 7:17
You can give us a Dockerfile. You can also point to Docker image. But the easiest thing to do that is to basically define the entire compute environment in code.
- 7:26
So we're gonna define the con-con- the container image using Modal's Python SDK. So we're gonna say image equals modal.Image.debian_slim() as the base image, and we're gonna pip install torch.
- 7:39
And then we're gonna use this image on this function, and we're gonna import torch. And just to show that it works, we're gonna print
- 7:47
torch.cuda.getdevice_name(). And hopefully this works when I run this. We'll delete this line. And when I run this thing, hopefully it will print something like we're running on an H100.
- 8:00
And, um, as you can see, it's still very fast, but slightly slower this time, uh, because loading Torch takes a little bit of extra overhead. And we'll talk about in a second what we've done to, to reduce that overhead, but, but it takes maybe about a second to initialize Torch.
- 8:17
Uh, okay, cool. So now we can run stuff on H-H100s. Let's try to run things on a lot of H100s. Uh, and so let's try to scale things out a little bit.
- 8:27
Uh, in Modal, any function can... You can map over any function in Modal just in code. So instead of calling just a single function invocation, we're gonna fan out and do a thousand, or maybe let's do ten thousand function invocation.
- 8:40
And you can do this in code by just saying we're gonna map over five thousand. I said ten thousand, actually, so let's do that. And we're gonna unpack the, the iterator, and let's, um, print x just to show some progress.
- 8:57
And what Modal does when you fan out is that it's gonna spin up as many containers as possible. And so you can see we're already running five containers, six containers, eight containers.
- 9:08
Uh, it makes it very easy to, to fan out and start, you know, even hundreds of containers or even thousands of containers running on GPUs. If we keep this running for several minutes, we can easily scale up to very large, um, number.
- 9:21
Um, so this gives you basically the ability to take something like, you know, that needs a lot of compute and, you know, something like a, a batch job and fan out, spin up thousands of containers, parallelize over it, and, and get results much faster.
- 9:37
Uh, we're gonna s- take a look at the
- 9:40
UI for a second. Uh, Modal also has a UI, uh, that you can access if you go to the, uh, website. Uh, the, the URL is printed in the console.
- 9:52
So let's take a look at that. Uh, so we can see the app details in our UI. There's all kinds of interesting things here. Modal has a pretty rich UI that lets you see container metrics, logs, uh, lets you set up users, and many other things.
- 10:08
Uh, so if you zoom in, for instance, on the number of containers, we can see here we spun up eighteen containers at peak. As I mentioned, if we had kept going, we would reach a much larger number.
- 10:17
Uh, we got up to eighteen containers at this point. Can look at the CPU utilization, GPU, et cetera. Um, could look at GPU temperature, thirty-three Celsius. Uh, even the, the watt consumption.
- 10:30
So there's a lot, a lot of other things here. We can look at app logs and many other things. Um,
- 10:36
okay, let's switch back to the terminal for a second and see some other stuff. There's a lot of stuff, so I'm not gonna go into every single possibility of how to use Modal.
- 10:48
But one thing I didn't show that I think is interesting and very valuable is you can also deploy these things. So, so far we only showed how to run things interactively, which means we have sort of, you know, we run things from our laptop.
- 11:00
But if I take this code and deploy this using modal deploy,
- 11:06
uh, we get this persistent endpoint. And what's nice about that is now we have this thing we can call from any other context in Python. And I'm just gonna show this using my REPL.
- 11:16
If we import modal, and if we do lookup like this,
- 11:22
uh, we get this handle to this remote function. So let's call this, and the first time we're gonna call it, we're gonna have incur a cold start. So it's gonna take a couple seconds 'cause the container has to start up.
- 11:32
And remember, we're, we're importing Torch, and we're running this on an H100, so it takes a little bit of extra time. The container keeps running for a few, for, for, for sixty seconds by default, and then it shuts down.
- 11:41
So the... Now it's actually idle. So if we call this again, typically it'll be a little bit faster. And, uh, we're obviously, you know, wasting an enormous amount of, uh, FLOPS using a GPU to calculate the square root of a number.
- 11:53
Uh, but, but this showcases, you know, how you can easily take things and deploy it, uh, even on very powerful hardware and, uh, and build these serverless endpoints. For instance, doing inference and, and, you know...
- 12:04
And Modal handles all the scaling. So when you invoke this function multiple times, we'll just scale up using more and more containers and shut down. Um- And many other things you can do with Modal.
- 12:14
You can, uh, set up distributed file systems that you can mount to each container, so you can, like, exchange information using the file system. You can s-set up web endpoints, you can set up cron jobs, and many other things.
- 12:27
Uh, so this hopefully gives you a little bit more of an idea of, like, what Modal looks like from an engineering perspective. Like, what does it look like when you're interacting through code with Modal?
- 12:36
Let's talk a bit about how Modal works under the hood. And as I mentioned, Modal, in order to deliver on this developer experience that I always wanted to have, we had to go down this very deep rabbit hole and build a lot of custom infrastructure ourself, and that's the only way we felt that we can make it
- 12:52
fast enough. We couldn't use Kubernetes, we couldn't use Docker, so we had to build a lot of this stuff ourselves. And it should be pointed out we're standing on shoulders of giants here.
- 13:01
Uh, we're using a fantastic container runtime called gVisor, uh, that gives us isolation, but we had to build a lot of stuff around it. Uh, we had to build our own scheduler and many other things, but we're obviously using a lot of the existing things in Linux and, and other systems, and we're using fantastic cloud tools as
- 13:18
well. Uh, in order to deliver the developer experience that we wanted to, as I mentioned, and the feedback loop that we wanted too, we had to figure out container cold start.
- 13:27
And container cold start, starting containers fast in a distributed system is a hard problem. So let's talk about what containers are to start with. Containers are, and this is my super crude, unfair generalization of what a container is or a container image, it's basically two things.
- 13:44
It's a root file system, so that's like the slash that you have in, in Linux that contains all the data on your drive. And then it's a bunch of stuff to iso-isolate processes so they can't tamper with each other.
- 13:56
Uh, there are many inefficiencies with how container images are stored and how container images are transferred. In particular, one of the issues is that there's a lot of junk.
- 14:08
There's a lot of stuff we're never gonna read. Like, many container images has Perl installed by default, man pages, locale information, time zone information for Uzbekistan. You're never gonna read this stuff, so we're sending all this data back and forth and, and, and the, the core thing here is we wanna start containers on a remote file, on
- 14:25
a remote worker very fast. We wanna minimize the amount of data that has to be transferred. We wanna do as little as possible. The other inefficiency is that there's a lot of redundancy in this.
- 14:35
Uh, a lot of the files that are being transferred back and forth are actually the same files. So if you grab just, like, three very different, uh, container images like I did in this case, and you look at the files, it actually turns out to be mostly the same files to a very large extent.
- 14:49
So with those two tricks, with those two observations, there's a number of tricks we can do and, and we... So we built what's called a content address storage. And this is not a new invention.
- 14:59
This is not something we came up with, but it's rarely used in production systems. Notably, AWS Lambda actually uses the same technique. And the idea is that instead of storing the images directly, we store the images, the container images, as just a bunch of metadata that points to blobs.
- 15:18
And for each blob, we compute a checksum or a hash value, and then we use that to deduplicate all the blobs, 'cause there's an enormous amount of redundancy in these blobs.
- 15:29
And this means the container images themselves are actually just little pieces of metadata, and in many cases, we can cache a very large percentage of the container images, and we can also avoid pulling data that we, we're not gonna need by lazy loading a lot of the data on access.
- 15:49
Uh, this is tricky because container cold start, in particular with Python, is very latency sensitive because we end up doing a lot of very sequential file accesses. So in many cases, when a container starts up in Modal or, or in any, in Python in any case, uh, it requires reading every single module, every single Python module, which
- 16:12
is many, in many cases it ends up being several thousand Python modules. Each one of them requires accessing the file system, and so what we can't allow is that to take several milliseconds, 'cause if you're doing something that takes several milliseconds and you're doing it a thousand times, it ends up taking several seconds, and we wanna avoid
- 16:31
that. So there's a lot of tricks that we have to do in order to basically get this down below a second. We do a lot of prefetching. We do a lot of task tracing.
- 16:40
We look at, you know, historical runs and see what types of files was accessed last time it ran. And then building these containers is, is obviously also another whole challenge.
- 16:51
We, we basically built our own container image builders. Uh, another technique that we're also more recently started leveraging is we can snapshot the CPU memory. So we talked about how we snapshot the container images, and we, we, we, we cache a lot of the data, which means, like, when you're loading it, you don't have to fetch a
- 17:09
lot of data. But what if we can avoid loading the data in the first place? What if we can just, like, revert to the, the memory state, the CPU memory, the RAM of, of, uh, a container?
- 17:20
And as it turns out, gVisor actually supports this, and that's another way that arguably supersedes a lot of the previous stuff. Uh, in practice, they end up kind of both reinforcing this, this container cold start.
- 17:31
But this lets us cut down even more dramatically, uh, th-things like Stable Diffusion we can now start in, in a couple of seconds, even though it involves loading very, very large, uh, model weights like, you know, five or 10 gigabytes.
- 17:44
Uh, we're also looking at GPU snapshotting, which will make things even more, even faster, which is very exciting. Um, and so doing all these things, you know, owning the entire stack, owning the file system, you know, building a storage system.
- 18:00
I didn't talk about the storage system. We, we basically use R2, and we run in many different regions, and so we use both the CDN and the, the, the R2.
- 18:08
And, and, and so all these optimizations together means we- Kind of solved the problem of container cold start. And let's remember, what it-- why did we originally set out to start this thing is because we wanted to deliver good developer experience.
- 18:24
As it turns out, it's good for other things too. So container cold start is also good because it enables serverless. So what does serverless mean? Uh, it means a lot of different things.
- 18:35
I think part of l- why sometimes I avoid the term serverless is that it has so many different definitions. But the, the promise of serverless was always don't provision more than you actually need.
- 18:46
Just, just, you know, only ch- pay for capacity you're actually using. And so especially with GPUs, which are very expensive, as it turns out, you can pack-- take a lot of different users, pull them together, and give people dynamically the resources they need and get dramatically better utilization.
- 19:09
And so that in turn means we can get lower cost. It means there's no capacity planning. It, it also, because we can pull a lot of these users, the, the variance, the, the total variance goes down, the relative variance, uh, which means we can run a much more predictable set of, of resource pools, uh, the, the total
- 19:28
capacity that we run. Uh, which is another problem, by the way. So we need to run thousands of GPUs. We use a lot of different cloud vendors. We use a lot of different regions.
- 19:37
We scale up and down continuously. Uh, in fact, we actually end up solving a mixed integer programming problem to, to do this, uh, minimizing the total cost spend. Uh, and, and this is some of the stuff we have to do for, for our customers so they don't have to think about it.
- 19:51
So through Modal, you can come in, you can request 100 GPUs. Under the hood, there's enormous amount of work that we had to put in in order to get the capacity somewhere in the world, uh, you know, spinning up GPUs if needed.
- 20:04
Uh, but in many cases, it happens instantaneously because we can maintain a buffer that makes it very fast to get access to these compute resources for any customer. Um, this was very technical, but just to kind of go back and look at a high level again, uh, why do people like Modal?
- 20:21
People pick Modal in-- because they can run their own code. We're not an AI API, so to speak. You can run almost anything with Modal. Uh, we make it possible to iterate very quickly.
- 20:32
We're fully usage based. So when you run things in Modal, you only pay for the time the containers are actually active. You have to never think about capacity. You don't have to, you know, go out and buy, you know, hundreds of GPUs or thousands of GPUs.
- 20:45
We can get you that within, you know, seconds or at least minutes. Uh, so there's a lot of things, the sort of burden of infrastructure, building your own internal platform, setting up Kubernetes, setting up, you know, Docker and all these things.
- 20:57
You don't have to think about this with Modal.
- 21:00
Um, how do you try Modal? It's actually very simple. Uh, you go to your terminal and you do pip install modal. The, the Python client automatically, you know, configures itself to connect to, uh, Modal, and you can immediately start running stuff because we give everyone $30 a month, uh, per, per month of free credits.
- 21:17
If you are a startup, uh, we can give you up to $50,000 in credits in order for you to get started.
- 21:25
Thank you, and I really hope you enjoyed this. And if you have any questions, feel free to reach out, [REDACTED:email_address]. Uh, you can also follow me on Twitter, [REDACTED:username], or check out my blog, erikbern.com.