# Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/qeDPKbWjsuk-keynote-ai-developer-experience-doesnt-have-suck) · [Video](https://www.youtube.com/watch?v=qeDPKbWjsuk)

## Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Modal’s edit–run loop connects Python functions to remote GPUs, but making that loop fast requires rethinking container storage, initialization, and shared capacity.

Familiarity with Python functions, decorators, and basic container concepts will help you follow the examples.

### Write the application, not its infrastructure

[0:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=0s)

How do you deploy models, scale them out, and run large batch jobs without spending your development time managing infrastructure? [Erik Bernhardsson](https://erikbern.com/), who previously built Spotify’s music recommendation system, describes that as the founding problem for [Modal](https://modal.com/). He started the company during the pandemic, four or five years before this talk, to make building data and machine-learning applications enjoyable again.

The intended user wants to write code: perhaps a custom model, perhaps a workflow around an existing one. Modal is closer to Kubernetes or AWS Lambda than to a fixed-model inference API: it runs arbitrary code and containers, while managing the underlying pool of thousands of GPUs and CPUs. The tradeoff is straightforward. Developers retain control over the application, but must implement more of it themselves. Python is the supported language in the platform described here.

### Fast feedback starts below the SDK

[1:50](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=110s)

**A fast edit–run loop depends on fast container startup.** Cloud infrastructure provides more powerful machines, but putting those machines behind image builds, uploads, and remote execution can make development feel slower than working locally. Modal’s goal is to take locally authored code, execute it remotely in the right image and on the right hardware, and return feedback quickly enough to preserve the rhythm of local development.

That requirement led to a custom scheduler and file system, part of a multiyear infrastructure effort Bernhardsson describes as still unfinished. The resulting platform has two faces: a resource pool containing H100, A100, L4, and T4 GPUs, and a Python SDK that exposes that pool. Python was the starting point because of its dominance in AI, machine learning, and data work. At the application boundary, a decorator turns a Python function into a serverless function.

![Slide titled “We wanted fast feedback loops” links making cloud development feel local to fast container cold starts and a custom container engine and scheduler.](https://ai.engineer/api/talk-essay-media/17932c6672fd8cc3c1955d5d76b158c00923f1832f5a602b382621a198bb4b8c)

Fast feedback loops require fast container cold starts.

### One execution platform, many workloads

[3:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=220s)

Bernhardsson identifies generative inference as probably Modal’s largest use case, but the workload range extends well beyond serving language models:

- **Diffusion and media:** generating music, video, and images.
- **Batch processing:** processing medical images and applying computer vision to video frames.
- **Computational biology:** protein folding and other workloads on GPUs or CPUs.
- **Language models:** fine-tuning, batch embeddings, and inference.

[Suno](https://modal.com/blog/suno-case-study) is the concrete customer example: Bernhardsson says it runs much of its AI music inference on Modal. The common requirement is flexible compute under application code, rather than a single prescribed model API.

![Slide titled “People use Modal at large scale today” lists workloads beside a green infrastructure illustration and the speaker’s video.](https://ai.engineer/api/talk-essay-media/b7934263b21395985d2af09f2a6b5ab46a816027a57141f3dd6a6c2688a2b44a)

Modal workloads include generative inference, batch preprocessing, computational biology, and LLM fine-tuning.

### A square function crosses the cloud boundary

[4:50](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=290s)

The live demonstration starts with a deliberately small application: `square` returns a number’s square and writes a message to standard error. The `app.function` decorator makes it remotely executable; a local entrypoint triggers it from the laptop. The essential structure is:

```python
import sys
import modal

app = modal.App("square-demo")

@app.function()
def square(x: int) -> int:
    print(f"Squaring {x}", file=sys.stderr)
    return x * x

@app.local_entrypoint()
def main():
    print(square.remote(12))
```

The arithmetic stays ordinary Python. The decorator and `.remote()` call establish where it runs.

With that file saved as `square_demo.py`, the interactive command is:

```bash
modal run square_demo.py
```

Modal packages the code into a container, executes it in the cloud, and streams output back to the terminal. Bernhardsson then changes a print statement and runs it again. The new code is picked up and the container rebuilt automatically, removing the manual cycle of rebuilding an image, pushing it, and retrieving logs. He describes the experience as nearly local in speed; the demonstration emphasizes iteration rather than presenting a timed comparison.

### Request hardware and define its environment

[6:30](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=390s)

The next change requests an H100 through the function decorator. A100s and T4s are other available choices. Bernhardsson reports getting access to an H100 in a couple of seconds in this demonstration. Merely attaching a GPU does not move Python arithmetic onto it: `x * x` still does not use the accelerator.

To inspect the device, the function needs PyTorch installed. Modal accepts a Dockerfile or an existing Docker image, but the demonstration defines the environment directly in Python: start with `modal.Image.debian_slim()`, install `torch`, and attach the image to the function. Extending the same example gives:

```python
import sys
import modal

app = modal.App("square-demo")
image = modal.Image.debian_slim().pip_install("torch")

@app.function(image=image, gpu="H100")
def square(x: int) -> int:
    import torch

    print(torch.cuda.get_device_name(), file=sys.stderr)
    print(f"Squaring {x}", file=sys.stderr)
    return x * x

@app.local_entrypoint()
def main():
    print(square.remote(12))
```

[`torch.cuda.get_device_name()`](https://docs.pytorch.org/docs/stable/generated/torch.cuda.get_device_name.html) returns the CUDA device’s name. It establishes hardware access without pretending that squaring an integer is useful GPU work. Bernhardsson estimates roughly one second of Torch initialization overhead in this run. That remaining initialization cost becomes important later.

### Map the function, then inspect the actual scale

[8:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=497s)

Once a function can run remotely, batch parallelism becomes a change in how it is called. After considering smaller batches, Bernhardsson settles on 10,000 invocations. Replace the local entrypoint with a mapping call and consume its result iterator:

```python
@app.local_entrypoint()
def main():
    for result in square.map(range(10_000)):
        print(result)
```

Modal fans the work out across containers. During the ramp, Bernhardsson points out five, then six, then eight running containers. He describes hundreds or thousands of GPU containers as possible with a longer run; that larger scale is not reached in this demonstration. The mechanism is the useful part: independent batch inputs become parallel function invocations without application code explicitly provisioning each worker.

![Terminal displays numeric output, a running container status line, and a progress bar labeled 7239/10000, with the speaker below.](https://ai.engineer/api/talk-essay-media/efb4a2147b55f128dc28ad041bda7564c84dda2864829691bdac4b9e3238b61e)

The mapped batch run shows progress at 7239/10000.

The console also prints a link to the application dashboard. There, the run can be inspected through app details, container metrics, and logs; the UI also includes user management. The demonstrated run peaks at 18 containers. Bernhardsson then examines CPU and GPU utilization, a GPU temperature reading of 33°C, and power consumption. These views connect the convenient mapping call to the resources it actually created.

### Keep the function available, not the container

[10:48](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=648s)

Interactive runs are initiated from the laptop. Deployment makes the function persistently addressable from another Python context:

```bash
modal deploy square_demo.py
```

In the recording, Bernhardsson opens a REPL, imports `modal`, and uses a lookup operation to obtain a remote function handle. The important distinction is between a **persistent callable endpoint** and a permanently running container. The deployed function remains available even when its execution container has shut down.

The first deployed call incurs a cold start, including Torch import in an H100-backed container. Bernhardsson describes a default 60-second idle retention period before shutdown. A subsequent call can reuse the warm container and is typically faster. The tiny arithmetic workload still wastes the GPU’s compute capacity; it demonstrates deployment and lifecycle behavior, not acceleration. More invocations can create more containers, followed by scale-down.

The same platform also supports distributed file systems mounted into containers for exchanging information, web endpoints, and cron jobs. Remote function calls are therefore one interface to a broader application environment, rather than the only way to organize work.

### Start containers by moving less data

[12:36](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=756s)

The infrastructure explanation begins with the speed requirement. Bernhardsson says Kubernetes and Docker could not deliver the desired startup behavior for Modal’s internal execution stack, so the team built its own scheduler and surrounding machinery. That does not conflict with accepting Docker images as inputs. The execution system uses [gVisor](https://gvisor.dev/docs/user_guide/checkpoint_restore/) for isolation, alongside existing Linux facilities and cloud tools.

For this explanation, a container can be reduced to two pieces: a root file system, and process isolation that prevents workloads from interfering with one another. The file-system side contains two opportunities to avoid work. First, an image may contain data the application never reads—Perl, manual pages, locale information, or Uzbekistan time-zone data. Second, different images contain many identical files. Bernhardsson’s comparison of three otherwise different images finds extensive overlap. Moving every byte of every image independently wastes both bandwidth and startup time.

Modal uses **content-addressed storage** to exploit that overlap. An image becomes metadata pointing to blobs; each blob is identified by a checksum or hash, allowing identical content to be deduplicated. Bernhardsson notes that AWS Lambda uses the same broad technique.

| Observation | Storage response | Startup benefit |
|---|---|---|
| Images share identical content | Address blobs by content hash | Reuse cached blobs across images |
| Many files are never read | Load data on access | Avoid unnecessary transfers |
| Images can reference shared blobs | Store image metadata separately | Keep the image description small |

Caching and lazy loading solve different parts of the problem: caching avoids fetching content already nearby, while lazy loading avoids fetching content that is not needed at all.

### Predict reads, then bypass initialization

[15:49](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=949s)

Lazy loading introduces another problem: Python startup can require reading several thousand modules, with many file accesses occurring sequentially. A small delay on each read accumulates. Bernhardsson explains that millisecond-scale accesses repeated a thousand times can add seconds to startup. Saving transfer volume is not enough if each required file introduces another remote round trip.

Modal therefore uses **prefetching** and **task tracing**. Historical runs reveal which files were accessed, allowing the system to anticipate those reads on a later start. Bernhardsson describes these techniques as part of the effort to get startup below a second. They preserve demand loading’s avoidance of unnecessary data while reducing the latency of predictable reads.

The next step goes beyond fetching files efficiently. Modal builds its own container images, but also snapshots CPU memory. Instead of loading data and repeating initialization, it can restore the container’s RAM state through gVisor’s checkpoint/restore support. Image caching and memory restoration reinforce one another: one makes the underlying data available cheaply; the other avoids recreating already initialized process state.

Bernhardsson reports starting **Stable Diffusion** in a couple of seconds despite roughly 5–10 GB of model weights. The talk does not specify the model version, hardware, cache conditions, or latency percentile for that claim. GPU snapshotting is still under investigation at this point in the recording. CPU memory restoration should therefore not be read as a claim that the demonstrated system restores GPU state.

These optimizations sit on a distributed storage system that Bernhardsson says uses R2 and a CDN across multiple regions. Together, the scheduler, file system, caching, prefetching, and snapshots bring the discussion back to the original goal: making remote execution fast enough to support an enjoyable development loop.

### Fast startup changes the economics of shared GPUs

[18:24](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1104s)

Fast cold starts also make serverless resource allocation more practical. Bernhardsson frames the promise as provisioning only what is needed and paying for used capacity. With expensive GPUs, pooling many users and assigning resources dynamically can improve utilization and reduce cost. Pooling also reduces relative demand variance, making aggregate demand more predictable than individual customers’ needs.

The capacity-planning problem moves to the provider. Modal operates thousands of GPUs across cloud vendors and regions, scaling its fleet up and down. Bernhardsson says the system solves a **mixed integer programming** problem to minimize total capacity spend. A request for 100 GPUs may require acquiring capacity somewhere in the world and starting machines; a maintained buffer can instead satisfy the request immediately. The simple function interface rests on continuous procurement and allocation work beneath it.

### From an infrastructure platform to an install command

[20:21](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1221s)

The resulting contract is to run your own code, iterate quickly, and let the provider manage capacity. Bernhardsson describes billing as covering the time containers are active, and says hundreds or thousands of GPUs can become available within seconds or minutes. Active-container billing is not the same as billing only while useful computation runs: [retained idle resources can still incur charges](https://modal.com/docs/guide/cold-start). The developer avoids operating an internal platform built around Kubernetes, Docker, and GPU capacity planning.

The talk closes with `pip install modal` and describes the client as configuring its connection automatically. For a present-day setup, the [installation guide](https://modal.com/docs/guide) also requires an account and authentication:

```bash
pip install modal
modal setup
```

The recording’s Python-only access and REPL lookup belong to its SDK era; current documentation also supports JavaScript/TypeScript and Go clients for calling Functions and managing resources, while Python remains primary for implementing Functions. At the talk, Bernhardsson offers $30 per month in free credits and up to $50,000 in startup credits. Those are the recorded offers, rather than a statement of current startup eligibility. The long infrastructure journey ends at a deliberately small entry point: install the client, write a function, and let the platform supply its execution environment.

![Slide titled “How do you try Modal?” shows “pip install modal,” $30/month free credits for all users, and $10k–50k Modal credits for startups.](https://ai.engineer/api/talk-essay-media/d42f5f0695725cccc4fbcc347ec4fc21e81858b34edc57372fc05bf9b908beb8)

Getting started with Modal: pip install modal.

## Resources

- [Get started with Modal](https://modal.com/docs/guide) — Current setup instructions and a complete Python example defining an image, requesting a GPU, and running code remotely.
- [Fast, lazy container loading in Modal.com](https://modal.com/blog/jono-containers-talk) — Engineering explanation of content-addressed images, FUSE, shared caches, and the latency costs of fetching container data.
- [Memory snapshots: Checkpoint/restore for sub-second startup](https://modal.com/blog/mem-snapshots) — Explains CPU snapshot restoration, its interaction with lazy loading, and measured startup improvements for example workloads.
- [gVisor checkpoint/restore](https://gvisor.dev/docs/user_guide/checkpoint_restore/) — Official documentation for saving and restoring container execution state in gVisor.
- [On-demand Container Loading in AWS Lambda](https://www.usenix.org/conference/atc23/presentation/brooker) — A July 2023 systems paper describing Lambda's container storage, caching, deduplication, and demand-loading architecture.
- [Suno's inference and batch processing on Modal](https://modal.com/blog/suno-case-study) — Modal's customer case study describes how Suno moved from batch preprocessing to deployed inference and autoscaling.
- [CPU and GPU Memory Snapshots](https://modal.com/docs/guide/memory-snapshots) — Current snapshot configuration, lifecycle hooks, and limitations, including alpha GPU snapshot support.
- [Control cold starts and warm-container retention](https://modal.com/docs/guide/cold-start) — Explains queueing, initialization, warm-container settings, and the cost of retaining idle resources.

## Slides

### A tool for data, AI and ML

Modal began as an effort to make deploying and scaling models and running batch jobs more enjoyable, so developers could focus on code. Generative AI later emerged as a fitting use case.

[0:26](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=26s) [Slide describing a general-purpose, productive and fun tool, with generative AI as a use case.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3456efc3ce2db3d09275c41c02c4f0c3976d879d976392cd3ff30f89390fb04.webp)

All I wanted was a great tool for data/AI/ML stuff

• Something general purpose that lets you build a wide range of stuff
• Something that makes you productive
• Something that is fun to use

... turned out Gen AI was a perfect use case

Modal

### Who is Modal for?

Modal targets high-code workflows where developers want control over models and code. It runs the infrastructure, while developers build the application themselves.

[0:53](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=53s) [Slide describing Modal’s high-code use cases and managed AI and ML infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/4569b3d222d32bf7aefcfdfff28b1431b928c5205bcd987c706176d59311e87d.webp)

Who is Modal for?

Modal is for high-code use cases. For people running their own models, or custom workflows, or need high control over the code.

Modal is not an “AI API”.

Think of Modal as Kubernetes or AWS Lambda, but:
• Focused on AI/ML use cases
• Awesome developer experience
• Fully managed

Embedded code:
import modal
[illegible]

Modal

### Fast feedback loops

Modal built its own container engine and scheduler for fast container starts, so developers can run locally written code in the cloud while keeping a fast feedback loop.

[1:48](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=108s) [Slide explaining that fast container cold starts help cloud development feel local.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d6341f4c94e616a63cbcbc1de2f7da6990b419706a1a5f6f9b3d6d6f1be5bf4.webp)

We wanted fast feedback loops

• The #1 design criteria was to make cloud development feel like local
• This meant we needed to build super fast container cold start
  ◦ This meant we had to build our own container engine and scheduler and all kinds of stuff

Modal

### Building foundational infrastructure

Fast container cold starts led Modal to build its own scheduler, container runtime, and file system. The foundational infrastructure became a multi-year effort that is still ongoing.

[2:42](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=162s) [Slide listing a scheduler, container runtime and file system alongside a function-call dashboard.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fbd62ec2a63dcdf17cbd18f22a43a4a195747ba94e1cb719248dc075da659d9.webp)

This took us into a deep rabbit hole

We built our own:
• Scheduler
• Container runtime
• File system
• ... a lot more

... but let’s revisit this in a sec!

Modal

Interface text:
Live App
Function Calls
1 day
Started | Startup | Execution | Status
[illegible] | 0s | 133ms | 200 OK
[illegible] | 0s | 257ms | 400 Bad Request
[illegible] | 0s | 101ms | 200 OK
[illegible] | 0s | 133ms | 200 OK
[illegible] | 0s | 142ms | 200 OK
[illegible] | 0s | 202ms | 200 OK
[illegible] | 1.22s | 128ms | 200 OK

### What Modal is today

Modal combines a pool of thousands of GPUs and CPUs with a Python SDK. The example applies a decorator to turn a Python function into a serverless function.

[3:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=180s) [Slide describing a GPU and CPU resource pool and a Python SDK, with a short code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/689743b2b6d5f9dad2135391f3dc6675aa57501b9fd09fd4acf6311a87a5691e.webp)

What Modal is today

1. A big resource pool (thousands of GPUs and CPUs)
2. A Python SDK that makes it easy to run stuff

import modal

app = modal.App()

@app.function()
def my_cloud_function():
    print("hello, world!")

Modal

### Modal use cases

Modal’s largest use case is likely generative AI inference, including diffusion models for music, video and images. Other workloads include data preprocessing, computational biology and LLM fine-tuning.

[3:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=220s) [Slide listing generative AI inference, batch processing, computational biology and LLM fine-tuning.](https://assets.ai.engineer/sites/aiecode2025/sha256/c395ae5cf8ae5f831459179f5ff4e9a6f19ba803e7e769b15c78ae5098c64dcf.webp)

People use Modal at large scale today

• Gen AI inference, in particular diffusion models
• Batch jobs to preprocess data for training
• Computational bio (protein folding, sequence alignment, ...)
• LLM fine-tuning
• ... much more

Embedded dashboard: [illegible]

Modal

### Modal customers

Suno runs much of its AI music inference on Modal, while Substack replaced its SageMaker stack with Modal pipelines. The slide also names customers including Cursor, Mistral AI and Quora.

[4:28](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=268s) [Slide featuring Suno and Substack alongside other customer logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/660c3dedcc75c691ea5e695e09a590b33616fea819cfab3920c3631cd63c4543.webp)

What are some customers?

SUNO
AI-generated songs powered by Modal

substack
Replaced SageMaker stack with Modal pipelines

ramp | Quora | substack | Cartesia | CURSOR | SUNO | Mistral AI | contextual-ai

+ 1000s of other ones

Modal

### Demo time

The speaker moves to live coding to show what a basic Modal application looks like in Python.

[4:48](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=288s) [Demo introduction slide with a green illustration of a dog.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7cef204a5483bce9a8b0457ac2f7baff2e714f78db4b5b3900ab1e0bedfad06.webp)

Demo time

Modal

10

### Modal square function

@app.function() turns square into a cloud function; the local entrypoint calls square.remote(42) and prints the result.

[5:05](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=305s) [Python editor showing a Modal function that squares a number and a local entrypoint that calls it remotely.](https://assets.ai.engineer/sites/aiecode2025/sha256/d7fcfbd2af0db2dedc51106e2e6afe9151d9c0f814ff199f1d3c9b703f862972.webp)

import modal

app = modal.App("example-get-started")

@app.function()
def square(x):
    print("This code is running on a remote worker!")
    return x**2

@app.local_entrypoint()
def main():
    print("the square is", square.remote(42))

Interface text:
modal-ai-engineer-2025 — emacs get_started.py — 84x22
File Edit Options Buffers Tools Python Help
get_started.py  All  L15  (Python ElDoc)
Erik Bernhardsson's screen
Erik Bernhardsson

### Running the Modal square function

Modal runs the square function in a cloud container and streams its output back to the terminal. The call returns 1764.

[5:35](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=335s) [Terminal showing a command to run the Modal example.](https://assets.ai.engineer/sites/aiecode2025/sha256/cee5d0cda710221a1c563d894926ab1543d15d7bbd4232795d889a480b7d9e4e.webp)

erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py

Interface text:
modal-ai-engineer-2025 — modal run get_started.py — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

[5:44](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=344s) [Terminal shows a completed remote function run returning 1764.](https://assets.ai.engineer/sites/aiecode2025/sha256/7abaa116403b2e3a44bd62c017b8e96946f0d93275cc95fc676ba089ce7cde9f.webp)

erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
✓ Created objects.
  Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
  Created function square.
the square is 1764
Stopping app - local entrypoint completed.
This code is running on a remote worker!
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py

Interface text:
modal-ai-engineer-2025 — -zsh — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

### Rerunning edited code

Erik edits and reruns the Modal example. The new run still returns 1764 and prints a new greeting; he says Modal picks up the latest code and rebuilds the container automatically.

[6:12](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=372s) [Terminal shows two completed runs, including a new greeting and the square result.](https://assets.ai.engineer/sites/aiecode2025/sha256/54dd103f67567666cd5d0c9652b725b4fcf1c84c874f4e984a2c26ef0952f124.webp)

https://modal.com/apps/modal-labs/main/ap-[illegible]
✓ Created objects.
├── Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
└── Created function square.
the square is 1764
Stopping app - local entrypoint completed.
This code is running on a remote worker!
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
✓ Created objects.
├── Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
└── Created function square.
the square is 1764
Stopping app - local entrypoint completed.
hi all ai engineers
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 %

Interface text:
modal-ai-engineer-2025 — -zsh — 84×22
Erik Bernhardsson's screen
Erik Bernhardsson

### Initializing another run

Erik starts another run with access to an H100. The square calculation itself does not use the GPU; the terminal shows one container initializing.

[6:56](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=416s) [Terminal shows an earlier completed run and a new container initializing.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ac18d7cd1ff4ecd34286c19836d1c4ea6f4f9e3cca7a8de6f13d269e150ac7d.webp)

https://modal.com/apps/modal-labs/main/ap-3uzR6pZ13RZ8vTq1a0Ke9A
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-zm7Ftd8C8000zJtbOZLMci
✓ Created objects.
├── Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
└── Created function square.
the square is 1764
Stopping app - local entrypoint completed.
hi all ai engineers
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-zm7Ftd8C8000zJtbOZLMci
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-YmotOue5jKCOWM4n72aG5A
✓ Created objects.
├── Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
└── Created function square.
Loading images (1 containers initializing)... View app at
https://modal.com/apps/modal-labs/main/ap-YmotOue5jKCOWM4n72aG5A

Interface text:
modal-ai-engineer-2025 — modal run get_started.py — 84×22
Erik Bernhardsson’s screen
Erik Bernhardsson

### Requesting an H100

Bernhardsson requests an H100 for the remote square function. The calculation runs in a container with GPU access, but does not itself use the GPU.

[7:07](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=427s) [Python code requests an H100 for a remote square function.](https://assets.ai.engineer/sites/aiecode2025/sha256/6168fccf754482791a0ff4de6101ac15d783185e861348c4abba6c8a29e1317f.webp)

import modal

app = modal.App("example-get-started")

@app.function(gpu="h100")
def square(x):
    print("hi all ai engineers")
    return x**2

@app.local_entrypoint()
def main():
    print("the square is", square.remote(42))

Interface text:
modal-ai-engineer-2025 — emacs get_started.py — 84×22
File Edit Options Buffers Tools Python Help
[illegible] F1 get_started.py All L5 (Python ElDoc)
Erik Bernhardsson's screen
Erik Bernhardsson

### Adding Torch to the GPU image

Bernhardsson builds a Debian slim image with Torch, attaches it to the H100 function, and adds a print statement to inspect the GPU device name.

[7:41](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=461s) [Python code installs Torch in a Modal image and prints the GPU device name from the remote function.](https://assets.ai.engineer/sites/aiecode2025/sha256/044b241921fc65f5a8cf8c03e3f56214642769c720048a2fef2682843be316fc.webp)

import modal

app = modal.App("example-get-started")

image = modal.Image.debian_slim().pip_install("torch")

@app.function(gpu="h100", image=image)
def square(x):
    import torch
    print(f"{torch.cuda.get_device_name()=}")

    return x**2

@app.local_entrypoint()
def main():
    print("the square is", square.remote(42))

Interface text:
modal-ai-engineer-2025 — emacs get_started.py — 84x22
File Edit Options Buffers Tools Python Help
[illegible] F1 get_started.py All L12 (Python ElDoc)
Erik Bernhardsson's screen
Erik Bernhardsson

### Inspecting the H100 run result

The remote square function reports an NVIDIA H100 and returns 1764 despite a missing NumPy warning. Bernhardsson says initializing Torch adds about a second of overhead.

[8:04](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=484s) [Terminal reports an NVIDIA H100 device, a missing NumPy warning, and the square result.](https://assets.ai.engineer/sites/aiecode2025/sha256/71b29bba16df08f40d2fe69884a806343cbcf5f9f91ff236ef4ec5d88c1e5fc5.webp)

Created function square.
the square is 1764
Stopping app - local entrypoint completed.
hi all ai engineers
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
✓ Created objects.
  Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
  Created function square.
/usr/local/lib/python3.12/site-packages/torch/_subclasses/functional_tensor.py:295:
UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
torch.cuda.get_device_name()='NVIDIA H100 80GB HBM3'
the square is 1764
Stopping app - local entrypoint completed.
Runner terminated.

Interface text:
modal-ai-engineer-2025 — modal run get_started.py — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

[8:07](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=487s) [Terminal reports an H100 GPU and a square result of 1764.](https://assets.ai.engineer/sites/aiecode2025/sha256/918cbeb37156fa8d0a2d1d599395833eb2df4458f6eee7ca4acd6d4330bd1d91.webp)

Stopping app - local entrypoint completed.
hi all ai engineers
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py
erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal run get_started.py
✓ Initialized. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
✓ Created objects.
  Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
  Created function square.
/usr/local/lib/python3.12/site-packages/torch/_subclasses/functional_tensor.py:295:
UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
  cpu = _conversion_method_template(device=torch.device("cpu"))
torch.cuda.get_device_name()='NVIDIA H100 80GB HBM3'
the square is 1764
Stopping app - local entrypoint completed.
Runner terminated.
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/ap-[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 % emacs get_started.py

Interface text:
modal-ai-engineer-2025 — -zsh — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

### Parallel run progress

Bernhardsson maps 10,000 calls across containers and prints the results. The progress bar reaches 9,413/10,000 at 0:00:02.

[9:03](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=543s) [Terminal shows progress at 9,413 of 10,000 calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/75501fe87d54fcfa3257601e88bff41bf7ff7446b3ed6c6c7f61f59abe5e7e0b.webp)

9449
9450
9451
9452
9453
9454
9465
9324
9439
9440
9441
9442
9443
9444
9445
9446

Running (0/18 containers active)... View app at https://modal.com/apps/modal-labs/
square 9413/10000 0:00:02

Interface text:
modal-ai-engineer-2025 — modal run get_started.py — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

### Parallel run completed

The parallel run finishes and prints a link to its Modal app details. Bernhardsson describes using containers to fan out compute-heavy batch jobs and return results faster.

[9:28](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=568s) [Terminal displays final output and reports that the app completed.](https://assets.ai.engineer/sites/aiecode2025/sha256/f346a8e2fac404b44e55c0abf2a0714d3dd89ccb24ee78270cb5f6514c2c1390.webp)

9991
9992
9993
9994
9995
9996
9863
9864
9865
9879
9880
9881
9882
9883
9884
8340
Stopping app — local entrypoint completed.
✓ App completed. View run at
https://modal.com/apps/modal-labs/main/[illegible]
erikbern@Eriks-Laptop modal-ai-engineer-2025 %

Interface text:
modal-ai-engineer-2025 — -zsh — 84x22
Erik Bernhardsson's screen
Erik Bernhardsson

### Calling a deployed function

Erik deploys square, looks it up by app name in Python, and calls it remotely. The calls return 15129 for 123 and 550012485999025 for 23452345.

[12:26](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=746s) [Terminal showing deployment, function lookup, and remote calls returning squared values.](https://assets.ai.engineer/sites/aiecode2025/sha256/f6f738ce877959c8056f356d1fdca7fe689524b86a94c5e2d1619adebdb0d962.webp)

erikbern@Eriks-Laptop modal-ai-engineer-2025 % modal deploy get_started.py
✓ Created objects.
├── Created mount /Users/erikbern/tmp/modal-ai-engineer-2025/get_started.py
└── Created function square.
✓ App deployed in 0.732s! 🎉

View Deployment: https://modal.com/apps/modal-labs/main/deployed/example-get-started

erikbern@Eriks-Laptop modal-ai-engineer-2025 % python3
Python 3.12.4 (v3.12.4:8e8a4baf65, Jun 6 2024, 17:33:18) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import modal
>>> f = modal.Function.from_name("example-get-started", "square")
>>> f.remote(123)
15129
>>> f.remote(123)
15129
>>> f.remote(23452345)
550012485999025
>>>

Interface text:
modal-ai-engineer-2025 — Python — 84×22
Erik Bernhardsson's screen
Erik Bernhardsson

### Modal's custom infrastructure

Modal built its own scheduler, file system and image builder alongside a gVisor-based runtime. Bernhardsson says gVisor provides isolation while custom infrastructure helps deliver speed.

[12:43](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=763s) [Slide lists a container runtime, scheduler, file system, and image builder.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4703888a46ddbe7c66c129495f8a86dcebe5441cc135195225c9e19c01fe2f0.webp)

Modal is fast because we built our own infrastructure

• Built our own container runtime (based on gVisor)
• Built our own scheduler
• Built our own file system for fast container startup
• Built our own container image builder

Modal

Interface text:
Erik Bernhardsson
Erik Bernhardsson's screen

### Unused files in container images

The Python image has 29,772 files, including thousands of documentation, locale and man-page files. Bernhardsson wants to avoid transferring unread data so containers can start quickly on remote workers.

[13:57](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=837s) [Slide uses a Python container image to illustrate the quantity of files that may never be read.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fabb1c0a7e9538b62bc083e9e1670f4df88ea2afa3408d2425444238124d192.webp)

Inefficiency 1: Most container images have a lot of junk

Eg the python container image from Dockerhub:
• 870MB large
• 29,772 files
  ◦ /usr/share/locale: 1,553 files
  ◦ /usr/share/doc: 3,210 files
  ◦ /usr/share/perl: 1,389 files
  ◦ /usr/share/man: 3,050 files
• Most of this is never actually read by the container!

Modal

Interface text:
Erik Bernhardsson
Erik Bernhardsson's screen

### Duplicated files across container images

The diagram shows 8,716 files shared by all three images. Bernhardsson suggests skipping unread files and caching duplicates to reduce data transfer.

[14:30](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=870s) [Slide explains unused and duplicated container image data alongside a diagram of overlapping file sets.](https://assets.ai.engineer/sites/aiecode2025/sha256/266d33a3dd714192cc545805392f1be7b3a7ef2691bf6bc7f98b774c9f1f9614.webp)

Inefficiency 2: Most container images store the same files

Based on two observations:
• Most of container images are not actually read – no need to transfer this data
• Most data inside container images is duplicated across images – can cache this

pytorch/pytorch
18,240 unique files
6,415
10,695
8,716
2,228
78,287 unique files
25,438 unique files
smy20011/dreambooth
huggingface/transformers-pytorch-gpu

Modal

Interface text:
Erik Bernhardsson
Erik Bernhardsson's screen

### Cloud illustration

Bernhardsson returns to Modal’s appeal: developers can run their own code, iterate quickly, and pay only for the time containers are active.

[20:10](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1210s) [Turquoise clouds surround tall geometric forms on a green slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/aedeff7811c29a788b7e4c2291d90d70b49ca113bcc413e5808eb9b4046c902b.webp)

25

Interface text:
Erik Bernhardsson

### How to try Modal

Install Modal with pip; its Python client configures the connection. The slide offers all users $30/month in free credits and startups $10k–50k in Modal credits.

[21:03](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1263s) [Slide shows the Modal installation command and describes free monthly and startup credits.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f8b03a382d79b505b4bc17f2d23428ce42388a0d72a0e2346b5668a97ac6482.webp)

How do you try Modal?

pip install modal

All users get $30/month free credits!
We also offer all startups $10k-50k Modal credits

Modal

Interface text:
Erik Bernhardsson
Erik Bernhardsson's screen

### Dog illustration

Bernhardsson closes the talk with a stylized green and yellow dog portrait.

[21:34](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1294s) [A stylized dog portrait in green and yellow on a green slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e5b39937f69085d466c4facea17a43ea5bce38e22f04ef713f9d6c54b3b0043.webp)

27

Interface text:
Erik Bernhardsson

## Full transcript

[0:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=0s) Hi, my name is Erik [REDACTED:username]. It's great to be here virtually. Uh, who am I? I am the CEO of a company called Modal. We are based here in New York.

[0:10](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=10s) Most of my background is in data, AI, machine learning, and in particular, I was at Spotify for many years and built a music recommendation system there. I did leave about 10 years ago and did all kinds of other stuff in between.

[0:21](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=21s) But started Modal about four or five years ago during the pandemic, and the mission I had at that point was to build an infrastructure platform for data, AI, machine learning in a way that takes all the-- that, that makes it fun again to write these applications.

[0:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=40s) To, to basically to, to deploy models, scale them out, run large-scale batch jobs, making it possible to focus on writing code and not have to deal with the infrastructure.

[0:49](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=49s) As it turned out, GenAI was a perfect use case for this. We just didn't know it at that time. Uh, Modal is very much focused on high code use cases.

[0:57](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=57s) What that means is we focus on people who wanna write their own code, in particular writing their own models, but also in many cases, using existing models in a way where you have-- wanna have control over the workflow or, or other thing.

[1:08](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=68s) And so you can think of us more as like Kubernetes or AWS Lambda in the sense that we can run arbitrary containers or arbitrary code. We do focus on Python right now, might add other languages in the future.

[1:19](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=79s) Unlike system like Kubernetes, we're fully managed, so we, we run all the infrastructure. We have a big pool of thousands of GPUs and CPUs, uh, but we let you run all kinds of applications in our cloud and, uh, this could really be anything.

[1:35](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=95s) In that sense, we're not an AI API. We don't have one model or ten models that we put behind an API doing next token prediction. You can really run anything, which puts a little bit more onus on the developer to, to build this thing, but it also makes this a lot more powerful.

[1:50](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=110s) In particular, when I think about platforms and how they make you productive and makes it fun to write code, a lot of my experience is that it comes down to fast feedback loops.

[2:02](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=122s) So in order to make engineers fast and make them more productive, you wanna have this like super fast feedback loop that let you iterate on code very quickly. I think cloud has been a phenomenal invention and lets us build things with, you know, far more powerful things, but it's arguably a step backwards in terms of developer experience.

[2:19](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=139s) And, and thinking a lot about this problem, what, what I realized was in order to solve this, we had to build our own system to start containers in the cloud very fast.

[2:28](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=148s) 'Cause if you can start containers in the cloud very fast, you can take code that r- that the user is building locally and execute in the cloud, maybe inside a custom image, running on a GPU, whatever, uh, and, and have that sort of fast feedback loop that you like when you run things c- locally.

[2:43](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=163s) Uh, as it turns out, solving container cold start in a distributed system is a very, very deep rabbit hole. We had to build our own scheduler, we had to build it all, all-- build our own file system, and, and many other things.

[2:55](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=175s) So we, we, we set out on a multi-year journey that we still haven't completed, building a lot of this very, very core, very foundational infrastructure. Modal today, you can think of it as two facets.

[3:04](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=184s) One is a big resource pool. We run thousands of GPUs, different types, H100s, A100s, L4s, T4s, you name it. And the only way to access those is through a Python SDK.

[3:16](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=196s) We might add other languages in the future, like I mentioned, but right now it's Python. And, and the reason we started with Python is obviously that Python is such a dominant language in AI, machine learning, and, and data applications.

[3:28](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=208s) One way to think about Modal is that it's a serverless framework that basically lets you take any Python function and turn that into serverless function. And so you do that by applying this decorator, as you can see in this code sample.

[3:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=220s) I'll show you a little bit more examples in a second. Uh, people use Modal for very large scale applications, but also small scale applications. The, the biggest use case is most likely GenAI inference.

[3:52](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=232s) Uh, we in particular have seen a lot of traction within Diffusion models. So for instance, AI-generated music, video, images, but also a lot of batch jobs, a lot of, for instance, processing very large scale medical images or doing computer vision on, on frames of videos.

[4:12](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=252s) Um, seeing a lot of traction in computational bio, things like protein folding, both things running on GPUs or, or but also CPUs. Of course, LLMs. You can't talk about GenAI without mentioning LLMs.

[4:23](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=263s) We have a lot of fine-tuning applications, batch embeddings, uh, of course inference as well. Uh, some of our customers, one customer I'm, um, I always think is incredibly cool is, is Suno.

[4:35](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=275s) They, uh, do AI-generated music, uh, and, and run a lot of their inference on Modal. Uh, but we have many other use cases for, for Modal. Uh, some running at very large scale, uh, doing all kinds of different, different applications.

[4:50](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=290s) Modal, it's a little bit abstract to talk about Modal without going into code, so I'm gonna do some live coding. Uh, so let's jump into a terminal, and I'll show you exactly-- uh, try to give you an idea of like what it looks like in code.

[5:03](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=303s) So let's look at a very, very basic Modal application. Uh, Modal, basically, one way to think about it is we take Python functions and turn them into things that run in the cloud.

[5:14](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=314s) Uh, there's a very simple function in it called square, which run- returns a square of a number and also prints some stuff to standard error. And this decorator that we apply, app.function,

[5:25](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=325s) takes that and turns that into a serverless function running in the cloud. And there's a few different ways to invoke this thing, but we have a little thing here that basically makes sure to trigger it from our laptop when we run it from the command line.

[5:36](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=336s) So we're gonna do that. So Modal has a little command line interface where basically it lets you run things interactively. And what happens when we run this thing is we take the code, we stick it in a container, we execute it in the cloud.

[5:49](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=349s) As it's executing, it streams the output back. And the whole point of this is like we wanna make it fast and feel like we're almost developing things locally. It's almost as fast as running things locally.

[5:58](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=358s) And this extends to things like, let's say you wanna edit this thing,

[6:02](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=362s) uh, and just, you know, print something else.

[6:05](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=365s) And in- instead of having to rebuild a container, push up the container to the cloud, download logs, et cetera, with this slow feedback loop, we just-- it just picks up the latest code, right, and rebuilds the container automatically and all these things, right?

[6:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=377s) And so while you're, like, building applications and rewriting code, you can always just run things in the cloud very, very fast. So far, this is, uh, just showcases like the sort of iteration speed, but also, let's, let's look at the power of Modal.

[6:30](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=390s) Like, what can you do with Modal? Like, what kinds of stuff can you... Can we get to scale? Can we run things on, on, on other types of hardware?

[6:36](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=396s) So let's, let's actually run this on an H100, and, and the way in Modal you do that is by saying just on the function decorator, you say gpu equals one G-- H100.

[6:47](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=407s) Uh, we have a bunch of other types, as I mentioned. We have A-A100s and T4s and all kinds of other ones. But let's run this on an A-H100, which is NVIDIA's flagship.

[6:56](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=416s) Um, and we can get access to an H100 in a couple of seconds. This is obviously not using the H100, uh, but we're running it in a container that has access to an H100.

[7:07](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=427s) So let's say we wanna actually access it. Now we need to probably install some software, right? So we might wanna install Torch in this case. Uh, and there's a few different ways you can do that in Modal.

[7:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=437s) You can give us a Dockerfile. You can also point to Docker image. But the easiest thing to do that is to basically define the entire compute environment in code.

[7:26](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=446s) So we're gonna define the con-con- the container image using Modal's Python SDK. So we're gonna say image equals modal.Image.debian_slim() as the base image, and we're gonna pip install torch.

[7:39](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=459s) And then we're gonna use this image on this function, and we're gonna import torch. And just to show that it works, we're gonna print

[7:47](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=467s) torch.cuda.getdevice_name(). And hopefully this works when I run this. We'll delete this line. And when I run this thing, hopefully it will print something like we're running on an H100.

[8:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=480s) And, um, as you can see, it's still very fast, but slightly slower this time, uh, because loading Torch takes a little bit of extra overhead. And we'll talk about in a second what we've done to, to reduce that overhead, but, but it takes maybe about a second to initialize Torch.

[8:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=497s) Uh, okay, cool. So now we can run stuff on H-H100s. Let's try to run things on a lot of H100s. Uh, and so let's try to scale things out a little bit.

[8:27](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=507s) Uh, in Modal, any function can... You can map over any function in Modal just in code. So instead of calling just a single function invocation, we're gonna fan out and do a thousand, or maybe let's do ten thousand function invocation.

[8:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=520s) And you can do this in code by just saying we're gonna map over five thousand. I said ten thousand, actually, so let's do that. And we're gonna unpack the, the iterator, and let's, um, print x just to show some progress.

[8:57](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=537s) And what Modal does when you fan out is that it's gonna spin up as many containers as possible. And so you can see we're already running five containers, six containers, eight containers.

[9:08](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=548s) Uh, it makes it very easy to, to fan out and start, you know, even hundreds of containers or even thousands of containers running on GPUs. If we keep this running for several minutes, we can easily scale up to very large, um, number.

[9:21](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=561s) Um, so this gives you basically the ability to take something like, you know, that needs a lot of compute and, you know, something like a, a batch job and fan out, spin up thousands of containers, parallelize over it, and, and get results much faster.

[9:37](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=577s) Uh, we're gonna s- take a look at the

[9:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=580s) UI for a second. Uh, Modal also has a UI, uh, that you can access if you go to the, uh, website. Uh, the, the URL is printed in the console.

[9:52](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=592s) So let's take a look at that. Uh, so we can see the app details in our UI. There's all kinds of interesting things here. Modal has a pretty rich UI that lets you see container metrics, logs, uh, lets you set up users, and many other things.

[10:08](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=608s) Uh, so if you zoom in, for instance, on the number of containers, we can see here we spun up eighteen containers at peak. As I mentioned, if we had kept going, we would reach a much larger number.

[10:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=617s) Uh, we got up to eighteen containers at this point. Can look at the CPU utilization, GPU, et cetera. Um, could look at GPU temperature, thirty-three Celsius. Uh, even the, the watt consumption.

[10:30](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=630s) So there's a lot, a lot of other things here. We can look at app logs and many other things. Um,

[10:36](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=636s) okay, let's switch back to the terminal for a second and see some other stuff. There's a lot of stuff, so I'm not gonna go into every single possibility of how to use Modal.

[10:48](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=648s) But one thing I didn't show that I think is interesting and very valuable is you can also deploy these things. So, so far we only showed how to run things interactively, which means we have sort of, you know, we run things from our laptop.

[11:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=660s) But if I take this code and deploy this using modal deploy,

[11:06](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=666s) uh, we get this persistent endpoint. And what's nice about that is now we have this thing we can call from any other context in Python. And I'm just gonna show this using my REPL.

[11:16](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=676s) If we import modal, and if we do lookup like this,

[11:22](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=682s) uh, we get this handle to this remote function. So let's call this, and the first time we're gonna call it, we're gonna have incur a cold start. So it's gonna take a couple seconds 'cause the container has to start up.

[11:32](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=692s) And remember, we're, we're importing torch, and we're running this on an H100, so it takes a little bit of extra time. The container keeps running for a few, for, for, for sixty seconds by default, and then it shuts down.

[11:41](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=701s) So the... Now it's actually idle. So if we call this again, typically it'll be a little bit faster. And, uh, we're obviously, you know, wasting an enormous amount of, uh, FLOPS using a GPU to calculate the square root of a number.

[11:53](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=713s) Uh, but, but this showcases, you know, how you can easily take things and deploy it, uh, even on very powerful hardware and, uh, and build these serverless endpoints. For instance, doing inference and, and, you know...

[12:04](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=724s) And Modal handles all the scaling. So when you invoke this function multiple times, we'll just scale up using more and more containers and shut down. Um- And many other things you can do with Modal.

[12:14](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=734s) You can, uh, set up distributed file systems that you can mount to each container, so you can, like, exchange information using the file system. You can s-set up web endpoints, you can set up cron jobs, and many other things.

[12:27](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=747s) Uh, so this hopefully gives you a little bit more of an idea of, like, what Modal looks like from an engineering perspective. Like, what does it look like when you're interacting through code with Modal?

[12:36](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=756s) Let's talk a bit about how Modal works under the hood. And as I mentioned, Modal, in order to deliver on this developer experience that I always wanted to have, we had to go down this very deep rabbit hole and build a lot of custom infrastructure ourself, and that's the only way we felt that we can make it

[12:52](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=772s) fast enough. We couldn't use Kubernetes, we couldn't use Docker, so we had to build a lot of this stuff ourselves. And it should be pointed out we're standing on shoulders of giants here.

[13:01](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=781s) Uh, we're using a fantastic container runtime called gVisor, uh, that gives us isolation, but we had to build a lot of stuff around it. Uh, we had to build our own scheduler and many other things, but we're obviously using a lot of the existing things in Linux and, and other systems, and we're using fantastic cloud tools as

[13:18](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=798s) well. Uh, in order to deliver the developer experience that we wanted to, as I mentioned, and the feedback loop that we wanted too, we had to figure out container cold start.

[13:27](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=807s) And container cold start, starting containers fast in a distributed system is a hard problem. So let's talk about what containers are to start with. Containers are, and this is my super crude, unfair generalization of what a container is or a container image, it's basically two things.

[13:44](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=824s) It's a root file system, so that's like the slash that you have in, in Linux that contains all the data on your drive. And then it's a bunch of stuff to iso-isolate processes so they can't tamper with each other.

[13:56](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=836s) Uh, there are many inefficiencies with how container images are stored and how container images are transferred. In particular, one of the issues is that there's a lot of junk.

[14:08](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=848s) There's a lot of stuff we're never gonna read. Like, many container images has Perl installed by default, man pages, locale information, time zone information for Uzbekistan. You're never gonna read this stuff, so we're sending all this data back and forth and, and, and the, the core thing here is we wanna start containers on a remote file, on

[14:25](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=865s) a remote worker very fast. We wanna minimize the amount of data that has to be transferred. We wanna do as little as possible. The other inefficiency is that there's a lot of redundancy in this.

[14:35](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=875s) Uh, a lot of the files that are being transferred back and forth are actually the same files. So if you grab just, like, three very different, uh, container images like I did in this case, and you look at the files, it actually turns out to be mostly the same files to a very large extent.

[14:49](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=889s) So with those two tricks, with those two observations, there's a number of tricks we can do and, and we... So we built what's called a content-addressed storage. And this is not a new invention.

[14:59](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=899s) This is not something we came up with, but it's rarely used in production systems. Notably, AWS Lambda actually uses the same technique. And the idea is that instead of storing the images directly, we store the images, the container images, as just a bunch of metadata that points to blobs.

[15:18](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=918s) And for each blob, we compute a checksum or a hash value, and then we use that to deduplicate all the blobs, 'cause there's an enormous amount of redundancy in these blobs.

[15:29](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=929s) And this means the container images themselves are actually just little pieces of metadata, and in many cases, we can cache a very large percentage of the container images, and we can also avoid pulling data that we, we're not gonna need by lazy loading a lot of the data on access.

[15:49](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=949s) Uh, this is tricky because container cold start, in particular with Python, is very latency sensitive because we end up doing a lot of very sequential file accesses. So in many cases, when a container starts up in Modal or, or in any, in Python in any case, uh, it requires reading every single module, every single Python module, which

[16:12](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=972s) is many, in many cases it ends up being several thousand Python modules. Each one of them requires accessing the file system, and so what we can't allow is that to take several milliseconds, 'cause if you're doing something that takes several milliseconds and you're doing it a thousand times, it ends up taking several seconds, and we wanna avoid

[16:31](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=991s) that. So there's a lot of tricks that we have to do in order to basically get this down below a second. We do a lot of prefetching. We do a lot of task tracing.

[16:40](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1000s) We look at, you know, historical runs and see what types of files was accessed last time it ran. And then building these containers is, is obviously also another whole challenge.

[16:51](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1011s) We, we basically built our own container image builders. Uh, another technique that we're also more recently started leveraging is we can snapshot the CPU memory. So we talked about how we snapshot the container images, and we, we, we, we cache a lot of the data, which means, like, when you're loading it, you don't have to fetch a

[17:09](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1029s) lot of data. But what if we can avoid loading the data in the first place? What if we can just, like, revert to the, the memory state, the CPU memory, the RAM of, of, uh, a container?

[17:20](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1040s) And as it turns out, gVisor actually supports this, and that's another way that arguably supersedes a lot of the previous stuff. Uh, in practice, they end up kind of both reinforcing this, this container cold start.

[17:31](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1051s) But this lets us cut down even more dramatically, uh, th-things like Stable Diffusion we can now start in, in a couple of seconds, even though it involves loading very, very large, uh, model weights like, you know, five or 10 gigabytes.

[17:44](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1064s) Uh, we're also looking at GPU snapshotting, which will make things even more, even faster, which is very exciting. Um, and so doing all these things, you know, owning the entire stack, owning the file system, you know, building a storage system.

[18:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1080s) I didn't talk about the storage system. We, we basically use R2, and we run in many different regions, and so we use both the CDN and the, the, the R2.

[18:08](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1088s) And, and, and so all these optimizations together means we- Kind of solved the problem of container cold start. And let's remember, what it-- why did we originally set out to start this thing is because we wanted to deliver good developer experience.

[18:24](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1104s) As it turns out, it's good for other things too. So container cold start is also good because it enables serverless. So what does serverless mean? Uh, it means a lot of different things.

[18:35](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1115s) I think part of l- why sometimes I avoid the term serverless is that it has so many different definitions. But the, the promise of serverless was always don't provision more than you actually need.

[18:46](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1126s) Just, just, you know, only ch- pay for capacity you're actually using. And so especially with GPUs, which are very expensive, as it turns out, you can pack-- take a lot of different users, pull them together, and give people dynamically the resources they need and get dramatically better utilization.

[19:09](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1149s) And so that in turn means we can get lower cost. It means there's no capacity planning. It, it also, because we can pool a lot of these users, the, the variance, the, the total variance goes down, the relative variance, uh, which means we can run a much more predictable set of, of resource pools, uh, the, the total

[19:28](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1168s) capacity that we run. Uh, which is another problem, by the way. So we need to run thousands of GPUs. We use a lot of different cloud vendors. We use a lot of different regions.

[19:37](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1177s) We scale up and down continuously. Uh, in fact, we actually end up solving a mixed integer programming problem to, to do this, uh, minimizing the total cost spend. Uh, and, and this is some of the stuff we have to do for, for our customers so they don't have to think about it.

[19:51](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1191s) So through Modal, you can come in, you can request 100 GPUs. Under the hood, there's enormous amount of work that we had to put in in order to get the capacity somewhere in the world, uh, you know, spinning up GPUs if needed.

[20:04](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1204s) Uh, but in many cases, it happens instantaneously because we can maintain a buffer that makes it very fast to get access to these compute resources for any customer. Um, this was very technical, but just to kind of go back and look at a high level again, uh, why do people like Modal?

[20:21](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1221s) People pick Modal in-- because they can run their own code. We're not an AI API, so to speak. You can run almost anything with Modal. Uh, we make it possible to iterate very quickly.

[20:32](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1232s) We're fully usage based. So when you run things in Modal, you only pay for the time the containers are actually active. You have to never think about capacity. You don't have to, you know, go out and buy, you know, hundreds of GPUs or thousands of GPUs.

[20:45](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1245s) We can get you that within, you know, seconds or at least minutes. Uh, so there's a lot of things, the sort of burden of infrastructure, building your own internal platform, setting up Kubernetes, setting up, you know, Docker and all these things.

[20:57](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1257s) You don't have to think about this with Modal.

[21:00](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1260s) Um, how do you try Modal? It's actually very simple. Uh, you go to your terminal and you do pip install modal. The, the Python client automatically, you know, configures itself to connect to, uh, Modal, and you can immediately start running stuff because we give everyone $30 a month, uh, per, per month of free credits.

[21:17](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1277s) If you are a startup, uh, we can give you up to $50,000 in credits in order for you to get started.

[21:25](https://www.youtube.com/watch?v=qeDPKbWjsuk&t=1285s) Thank you, and I really hope you enjoyed this. And if you have any questions, feel free to reach out, [REDACTED:email_address]. Uh, you can also follow me on Twitter, [REDACTED:username], or check out my blog, erikbern.com.
