Serverless AI infrastructure
Modal
Modal provides cloud infrastructure for developers running AI inference, training, batch processing and agents. Its Modal SDK lets teams define application logic, container images, hardware and storage in Python, then execute workloads with autoscaling and usage-based pricing. Developers can serve language, image and audio models, fine-tune models, or run parallel data jobs. Modal Sandboxes provides isolated environments for executing untrusted code, including coding agents and reinforcement-learning rollouts.
Work on Modal began in 2021, and the platform became generally available in 2023. Its founders are CEO Erik Bernhardsson and CTO Akshat Bubna. Modal’s engineering focuses on reducing the startup work that makes elastic GPU computing difficult: its custom filesystem loads container files on demand, while CPU and CUDA checkpoint restoration reuse initialized application state. These mechanisms let inference replicas start without repeating much of their setup, helping capacity respond to variable demand.
By May 2026, the company reported more than 1 billion cumulative sandbox launches, with Sandboxes contributing over one-third of revenue. It also reported exceeding $300 million in annualized revenue. That month, Modal raised a $355 million Series C led by General Catalyst and Redpoint at a $4.65 billion post-money valuation.
5 talks
Newest first
What We Learned From A Year of Building With LLMs
Eugene Yan · Hamel Husain · Jason Liu · Dr Bryan Bischof · Charles Frye · Shreya Shankar
AI Engineer World's Fair 2024 · 35:21
3 speakers at AIE
Affiliations reflect their AIE appearances, not necessarily current employment.
Start here
- What every AI engineer needs to know about GPUs
Start here to understand how GPU parallelism and low-precision matrix multiplication shape decoding and inference infrastructure.
Charles FryeAI Engineer World's Fair 2025
- Taking Reinforcement Learning Cross Datacenter
Learn why finite-precision rounding can leave many rollout-visible weights unchanged between updates, making sparse synchronization useful.
Nan JiangAI Engineer World's Fair 2026
- What Lies Beneath the API — Benjamin Cowen, Modal
Follow the infrastructure needed for custom models, from serverless training and sandboxed reinforcement-learning rollouts to autoscaling inference.
Benjamin CowenAI Engineer Europe 2026
- How fast are LLM inference engines anyway?
Modal's Charles Frye examines how increasingly capable open-weight models and modern inference engines make self-hosted LLM deployment practical.
Charles FryeAI Engineer World's Fair 2025
Messages from the stage
Match inference benchmarks to the workload
Frye's talks connect tensor cores, memory bandwidth, and numerical precision to engine selection. Throughput and first-token latency objectives provide different ways to assess vLLM, SGLang, and TensorRT-LLM.
Separate training from rollout infrastructure
Jiang describes keeping tightly coupled training together while distributing rollout workers geographically. Sparse synchronization of served policy weights reduces the need to transfer full checkpoints between those systems.
Model choices sit inside production decisions
Cowen frames fine-tuning around application-specific performance and inference economics. The joint discussion featuring Frye adds provider switching, team workflows, and production safeguards to the broader decision about how to build with LLMs.
Affiliations reflect each recorded session, not necessarily current employment.