# What Lies Beneath the API — Benjamin Cowen, Modal

Benjamin Cowen

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/HvZXAOZ3iv8-what-lies-beneath-api) · [Video](https://www.youtube.com/watch?v=HvZXAOZ3iv8)

## What Lies Beneath the API

Fine-tuning becomes worth considering when a product needs its own economics, performance targets, and business logic—and when its data and evaluations can support the move.

Familiarity with model APIs, basic Python, and evaluating AI application outputs will help you follow the training and serving examples.

### When does your application become a custom domain?

[0:25](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=25s)

When does an application become specialized enough that fine-tuning is worth the effort? A general-purpose model can get a product started quickly, but a maturing product may need better performance on a narrower task or a different cost structure. That is the decision Benjamin Cowen opens with: when to move from consuming a model to shaping one around the service you provide.

Cowen approaches this question through his work at Modal, a general-purpose serverless compute platform providing functions and hardened sandboxes for code execution. The workloads span physics simulations, quantum chemistry, voice processing, LLMs, and agents. Growing demand for large-scale reinforcement learning adds another reason to think about where an application sits on the spectrum between a managed model API and a fully custom stack.

### From caveman mode to infrastructure ownership

[1:50](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=110s)

Frontier APIs make it possible to build useful applications exceptionally quickly, without first operating model infrastructure. In the API-only setup Cowen describes, prompting is the main customization surface; the contrast is with controlling training and serving yourself, rather than a claim that every hosted model service lacks fine-tuning.

Consider his caveman-mode example: ask the model to speak tersely, and shorter answers consume fewer tokens. That can reduce spending, but it does not establish that the economics will survive a startup growing a hundredfold or a thousandfold. An enterprise contract can also introduce specific latency and throughput requirements—or a quality metric that encodes the customer's business logic. Those requirements reach beyond the wording of a prompt.

![Slide titled “Use a frontier API,” listing fast starts, no infrastructure overhead, and good models, alongside reasons to shift: performance control, model differentiation, and rate limits.](https://ai.engineer/api/talk-essay-media/c8527c841938b68c768431ac1f82fa91f10e1cd5a832b6a4e32af856df5a6da9)

When frontier APIs work well, and reasons to move beyond them.

Traditionally, gaining that control meant jumping to the other end of the spectrum. Training has different compute and scaling characteristics from production inference. Running both on a large cluster requires isolating training resources from serving resources, then assigning infrastructure work to infrastructure engineers—or to AI engineers and scientists who would otherwise work on the model.

| Approach | What it makes easy | What you take on |
|---|---|---|
| Frontier API | Fast application development | Provider constraints on customization and capacity |
| Self-managed cluster | Deep model and runtime control | Resource isolation and stack operations |
| Serverless custom workloads | Custom compute without managing a cluster | Training and serving decisions |

The emerging middle ground is a cloud platform that makes custom workloads easier to operate. Its purpose is to reduce the infrastructure responsibility attached to model differentiation.

### Optimize for the service you actually provide

[4:10](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=250s)

Cowen reports that Intercom's fine-tuned model beat its frontier API baseline at one-fifth the cost. He does not specify the evaluation, workload, or cost-accounting conditions, so this is an attributed customer example rather than a general performance guarantee. His Pinterest example likewise invokes orders-of-magnitude savings without an exact multiplier; that comparison concerns cost, not an orders-of-magnitude improvement in model quality.

The rationale he attributes to Decagon is more broadly useful than either number: frontier labs pursue models that perform well across many tasks, while a business needs its model to excel at its own service. **The objective is business-specific performance**, which may involve a particular workflow, response behavior, or operational constraint.

Open-source training libraries make the algorithm accessible without requiring the same team to manage a cluster. Combined with managed compute, they offer a way to reach toward custom training while retaining the fast iteration that made APIs attractive in the first place. Algorithm control and iteration speed are the two properties Cowen wants to preserve together.

### Separate the reason to train from readiness to train

[5:47](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=347s)

Cowen's expectation is that a differentiated product will eventually develop domain-specific requirements. That is a judgment about product development, not a universal deadline. The useful question is which observable signals would justify customization for your application.

- **Unit economics:** API spending exceeds what customers pay, even after reducing unnecessary output. A customized inference endpoint becomes worth investigating.
- **Latency and throughput:** The service needs performance characteristics that the current endpoint cannot deliver.
- **Evaluation progress:** Results have plateaued, suggesting that further prompting may not produce the improvement the product needs.

These signals identify a problem worth solving; they do not by themselves establish that fine-tuning is the solution.

**Data and mature evaluations are prerequisites.** If you have not been collecting useful data and cannot reliably evaluate the result, Cowen advises against training yet. Poor inputs remain poor inputs after a training run.

Existing product work can supply much of what is missing. An agent harness provides an environment in which a model can practice delivering the service through reinforcement learning. Evaluations and records of what worked or failed provide potential training data. The harness supplies interactions; the evaluation process supplies evidence about their outcomes.

Modern libraries also remove much of the need to calculate gradients or implement linear algebra by hand. Managed infrastructure reduces another part of the burden. The opportunity is to build on the product's existing harness, data, and evaluations rather than treating training as an entirely separate engineering discipline.

### Start small, then fan out training experiments

[8:21](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=501s)

The training example begins with code slides after the embedded video fails to play. Cowen describes a supervised fine-tuning setup in roughly 300 lines of Python. That is the size of the example, not an estimate of all the work required to curate data, evaluate a model, and operate it. With curated data and an account on Modal or another serverless platform, the setup becomes a practical starting point; he points to the [Modal examples repository](https://github.com/modal-labs/modal-examples).

Serverless scaling is useful for more than inference. In **hyperparameter search**, different configurations can run in separate containers requested on demand. Instead of treating every minute on a fixed cluster as scarce, the team can launch parallel trials and stop the ones that cease to look promising.

![Slide titled “SFT in 300 lines of code,” with text about data readiness, open source libraries, serverless infrastructure, and parallel hyperparameter sweeps beside a code screenshot.](https://ai.engineer/api/talk-essay-media/cffefe4207f0eaa8f2069444841bec417eca4711cbcbeb9ab291a55b2336c420)

SFT in 300 lines of code, with a linked example and training-code screenshot.

A small Python example makes the pruning decision concrete. Suppose three running trials report the same evaluation metric at a comparable checkpoint, where higher is better:

```python
from dataclasses import dataclass, replace

@dataclass(frozen=True)
class Trial:
    name: str
    learning_rate: float
    eval_score: float
    status: str = "running"

def propose_pruning(trials: list[Trial], minimum_score: float) -> list[Trial]:
    return [
        replace(trial, status="stop_requested")
        if trial.status == "running" and trial.eval_score < minimum_score
        else trial
        for trial in trials
    ]

trials = [
    Trial("a", 1e-5, 0.72),
    Trial("b", 3e-5, 0.41),
    Trial("c", 1e-4, 0.68),
]

for trial in propose_pruning(trials, minimum_score=0.60):
    print(trial.name, trial.status)
```

Trial `b` becomes `stop_requested`; `a` and `c` remain `running`. This records a proposed action. The orchestration layer must actually terminate the selected container to stop its compute consumption. The search-and-prune loop is what Cowen likens to a meta evolutionary algorithm: explore alternatives, discard weak candidates, and keep resources on promising ones.

### Reinforcement learning creates a parallel execution workload

[9:47](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=587s)

Reinforcement learning is longstanding, but its current tooling makes it accessible to engineers who did not previously specialize in it. Cowen describes an RL example in roughly 300 lines of code. As with the supervised example, the claim concerns a compact setup built on libraries, not the total effort of developing a reliable training system.

The execution pattern explains why serverless infrastructure is useful here. A model needs repeated opportunities to practice. These interactions and evaluations form **rollouts**, many of which can run independently. Modal exposes unified APIs for sandboxes and GPU containers or clusters, bringing the environments used for practice and the compute used for model work into the same programming surface. The slide presents this as a GRPO reinforcement-learning example.

![Slide titled “GRPO (RL) in 300 lines of code,” with a statement about unified GPU and Sandbox APIs, a repository link, and a code screenshot.](https://ai.engineer/api/talk-essay-media/ebfbf9bc712d132d6cbf8d0d8ca0265189c4e4d12332071cccc81455e2982589)

GRPO reinforcement learning example using unified GPU and Sandbox APIs.

Cowen reports customers scaling to 50,000–100,000 sandboxes for reinforcement learning during the preceding quarter. He does not specify whether those counts were concurrent, how long they ran, or a measured throughput result. The architectural point is the large parallel demand created by rollouts, with open-source example code available to illustrate the setup.

### Serve the model—and prepare before you need to train

[10:55](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=655s)

Training leaves one essential job unfinished: serving the model. The inference slide presents a vLLM example in 200 lines of code. Cowen also names SGLang, Triton Inference Server, and custom Python inference workflows as options. These perform the serving role hidden behind a frontier API; he explicitly does not claim to know whether frontier providers use vLLM.

![Slide titled “vLLM inference in 200 lines of code,” with a repository link and a tall code screenshot beside the speaker inset.](https://ai.engineer/api/talk-essay-media/79dc342e2847ef9d2e61fe2e92c3a9f2a327a044d331fa079c083bc45146d44f)

vLLM inference in 200 lines of code.

On Modal or another suitable serverless platform, the serving workload can autoscale with incoming traffic. This completes the lifecycle: train the model, put it behind an inference service, and scale that service to demand. It also returns to the original decision. Having an accessible path through the stack is not a reason to train immediately.

The practical horizon may be six months or a year rather than a distant future project. Those are planning examples, not deadlines. Decide how you will recognize the moment when customization is justified, then collect the data and develop the evaluations that will let you act on that decision. Preparation makes training an available option before cost, latency, or product quality turns it into an urgent one.

## Resources

- [Modal examples repository](https://github.com/modal-labs/modal-examples) — Official Python examples for running applications on Modal, with installation and execution instructions.
- [Modal training and serving examples](https://modal.com/docs/examples) — Examples covering fine-tuning, reinforcement learning, parallel evaluations, hyperparameter searches, and model serving.
- [The age of vertical models is here](https://ideas.fin.ai/p/fin-apex-1-0-model-announcement) — Intercom's account of deploying a specialized answering model for Fin's customer-service conversations.
- [Pinterest's Q3 2025 discussion of specialized AI](https://s204.q4cdn.com/369458543/files/doc_financials/2025/q3/PINS-Q3-2025-Transcript-2.pdf) — Page 9 discusses early cost comparisons between fine-tuned open-source models and proprietary models for Pinterest's visual-AI workloads.
- [How Decagon shipped real-time voice AI on Modal](https://modal.com/blog/decagon-case-study) — A case study combining specialized model training, speculative decoding, and serving-engine optimization for voice agents.
- [Modal's million-sandbox demonstration](https://modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds) — A July 2026 engineering account of rebuilding sandbox scheduling and demonstrating one million concurrent sandboxes.

## Slides

### AI Engineer Europe

The recording opens with the AI Engineer Europe logo.

[0:00](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=0s) [AI Engineer Europe logo in white on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ccdd81af89578fbdc369ab18584ea0dfb2b059eca41534beca1de745ba07dc8.webp)

AI Engineer
EUROPE

### Presenting Sponsor

Google DeepMind is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=5s) [Presenting sponsor Google DeepMind on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/22a68544b30de78510df1a9b6df16b3387e2d47afa7615dada9c2d67451498dc.webp)

PRESENTING SPONSOR
Google DeepMind

### Platinum Sponsors

Braintrust, WorkOS and OpenAI are the platinum sponsors.

[0:10](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=10s) [Platinum sponsors Braintrust, WorkOS and OpenAI on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/f68085d3cdb22c094cae3551c0214abe663bfb5d4464af890ea9ff6e5f88ab77.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### What Lies Beneath the API: When You Should Fine-Tune Your Own Model

Benjamin Cowen asks when fine-tuning becomes worthwhile as companies and their products mature and specialize, seeking better performance and cost.

[0:43](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=43s) [Modal title slide for Benjamin Cowen's talk on when to fine-tune a model.](https://assets.ai.engineer/sites/aiecode2025/sha256/519f74f30d4f0b189c06e8cb339a2d2bfb978cbdd2a9ff38c84b76e071a08cdc.webp)

Modal

What Lies Beneath the API:
When you should fine-tune your own model

Benjamin Cowen, PhD

CONFIDENTIAL - DO NOT DISTRIBUTE
1

AIE
Google DeepMind

### Modal 101

Modal combines functions and sandboxes with storage, application state and networking, powered by a custom runtime and capacity orchestration. Cowen draws on work across AI applications to introduce recurring patterns.

[1:13](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=73s) [Modal 101 slide describing functions, sandboxes, storage, application state and networking, supported by a custom runtime and capacity orchestration.](https://assets.ai.engineer/sites/aiecode2025/sha256/72cae7ca1b9691967f66c68862003d2cdcac6bc7dce9877d8a2d58484b69cdc8.webp)

AI-NATIVE INFRASTRUCTURE
Modal 101

As a general purpose platform, we’re seeing some interesting patterns.

Functions
Serverless compute that scales with your workload.

Sandboxes
Secure execution for untrusted code.

Storage
Efficient, scalable data loading.

Application State
Shared data collections across your container fleet.

Networking
Securely connect your containers to the world

POWERED BY

Custom Runtime
Performance optimized cold starts and file loading

Capacity Orchestration
Quickly and dynamically scale container fleets globally

Modal
CONFIDENTIAL - DO NOT DISTRIBUTE
2

Other visible text:
AIE
Braintrust
WorkOS
OpenAI

### The Model Spectrum: Frontier API

Frontier APIs offer a fast start, but limited customization and unpredictable performance. Scratch servers offer full control, with cluster management and software maintenance left to the user.

[1:52](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=112s) [A spectrum compares frontier APIs with scratch servers, highlighting fast startup and the limitations of customization and performance.](https://assets.ai.engineer/sites/aiecode2025/sha256/52a816bf24b1fb270ba6895e4689948ae36eb3add1ab08ea6a4ad12fd2582a67.webp)

The Model Spectrum

Frontier API ↔ Scratch Server

Frontier API:
✓ Start fast
× No customization
× Unpredictable performance

Scratch Server:
✓ Full control
× Cluster management
× Have to maintain the software yourself

CONFIDENTIAL - DO NOT DISTRIBUTE
3

AIE
AI Engineer EUROPE

### Use a frontier API

Frontier APIs offer a fast start with little infrastructure overhead. As requirements grow, teams may need more control over cost, throughput, latency, or custom metrics—or run into rate limits.

[1:56](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=116s) [A slide lists frontier API benefits and reasons to shift, including performance control, model differentiation, and rate limits.](https://assets.ai.engineer/sites/aiecode2025/sha256/25f7f7dcb8225af5768a139d285f983c2fd8761c5028a388dbee992143fe4a8c.webp)

Use a frontier API
Move fast. Ship something.

Works for
• Get started fast
• No infrastructure overhead
• Very good models!

When you might want to shift
• You need performance control at scale
  ◦ Cost, throughput, latency, custom metric
• You need model differentiation
• You’re hitting rate limits

CONFIDENTIAL - DO NOT DISTRIBUTE
4

AIE
AI Engineer
EUROPE

### The Model Spectrum: Scratch Server

Cowen says companies seeking model differentiation through fine-tuning traditionally jump from frontier APIs to running their own servers. That brings full control, along with cluster management and software maintenance.

[3:02](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=182s) [The spectrum compares frontier APIs with scratch servers, highlighting control, cluster management, and software maintenance on the server side.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6bc51f3ab761bd68db2fa98217457b436270b3927941abded4d7a0d297d6f54.webp)

The Model Spectrum

Frontier API ↔ Scratch Server

Frontier API:
✓ Start fast
⊗ No customization
⊗ Unpredictable performance

Scratch Server:
✓ Full control
⊗ Cluster management
⊗ Have to maintain the software yourself

CONFIDENTIAL - DO NOT DISTRIBUTE
5

AIE
Engineering the future of AI

### Build it yourself on Kubernetes

Kubernetes offers full control, but teams must manage the entire stack and rebalance resources for training, which likely scales differently from production apps.

[3:11](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=191s) [A slide describes full stack ownership and the resource and orchestration demands of training workloads on Kubernetes.](https://assets.ai.engineer/sites/aiecode2025/sha256/51fdcf0f6035d0a6e29e9ed146eca9af39913b7c95265e7d9a16245d6a56ccc2.webp)

Build it yourself on Kubernetes

Maximum control. Full ownership. But now you're in the infrastructure business.

Works for
• Traditional approach for gaining full control
• Fully tunable, you own every part of the stack

What it means
• You’re responsible for every part of the stack.
• You need to rebalance resources to account for training workloads
  ○ New GPU reservations and orchestration
  ○ Training likely has different scaling characteristics than your production apps

CONFIDENTIAL - DO NOT DISTRIBUTE
6

AIE
Engineering the future of AI
Modal

### Rise of domain-specific models

The slide says Intercom’s Fin Apex beats GPT-5.4 at one-fifth the cost with a 73.1% resolution rate; Decagon says its needs diverge from major labs’ priorities.

[4:13](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=253s) [A slide discusses fine-tuning, model performance, and domain-specific systems, with a quotation from Decagon.](https://assets.ai.engineer/sites/aiecode2025/sha256/08744da6f7030d11830dd1dc0d7954fc32960367a5cf826a4180fa657f698a12.webp)

Rise of domain-specific models

Fine-tuning on proprietary data is replacing API dependency

• Fine-tuning open-source models on proprietary data. Intercom's Fin Apex beats GPT-5.4 at 1/5 the cost — 73.1% resolution rate
• Owning model performance. Pinterest CEO: "orders of magnitude reduction in cost" using fine-tuned open-source vs. frontier APIs
• Treating models as raw material, not product. The model is the transistor. The fine-tuned, domain-specific system is the product

“As the major labs push toward broader reasoning capabilities, the gap between what they prioritize and what [Decagon] actually requires has only widened.”

Max Lu, Head of Decagon Labs
Decagon

Modal
8

Interface text:
AIE
AI Engineer

### When to train a domain-specific model

Consider training when evals plateau despite prompt work, you need lower latency or higher throughput, or unit economics fail to scale. Cowen says collected data and mature evals are also needed.

[5:43](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=343s) [Slide lists signals for training and the data, evaluation, and engineering resources needed to get started.](https://assets.ai.engineer/sites/aiecode2025/sha256/74d3ff0fc5909d9a09070de5c71e0f1e008b6fdca70be6397dc6dc32d6d25ae3.webp)

It’s a matter of time until your product becomes “domain-specific.”

What are some signals indicating it’s time to train?
✓ Evals are plateauing despite prompt work
✓ You need lower latency or higher throughput
✓ Unit economics aren’t scaling
✕ Still developing core functionality
✕ Haven’t collected data

So what do you need to get started?
✓ Agent harness
✓ Eval suite
✓ AI Engineers
✓ Data
✕ Infrastructure experts
✕ Cluster reservation

In building your product, you may have already done the hard part.

CONFIDENTIAL - DO NOT DISTRIBUTE
18
AIE
AI Engineer EUROPE

### SFT in 300 lines of code

Once data is in place, open-source libraries and serverless infrastructure make parallel hyperparameter sweeps trivial. Cowen illustrates SFT with a YOLO training function and says it can fit in 300 lines of Python.

[8:19](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=499s) [Slide describing serverless hyperparameter sweeps alongside a training code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/01291ef48b9b00ac784a1a20beb78a1ed4face053d7ef2158b105fc166afb635.webp)

SFT in 300 lines of code

Once you have your data in place, open source libraries and serverless infra make parallel hyperparameter sweeps trivial.

https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/yolo/finetune_yolo.py

@app.function(
    gpu=TRAIN_GPU,
    cpu=TRAIN_CPU_COUNT,
    timeout=60 * MINUTES,
)
def train(
    model_id: str,
    dataset: DatasetConfig,
    model_size="yolov10m.pt",
    quick_check=False,
):
    from ultralytics import YOLO

    volume.reload()  # make sure volume is synced

    model_path = volume_path / "runs" / model_id
    model_path.mkdir(parents=True, exist_ok=True)

    data_path = volume_path / "dataset" / dataset.id / "data.yaml"

    model = YOLO(model_size)
    model.train(
        # dataset config
        data=data_path,
        fraction=0.4
        if not quick_check
        else 0.04,  # fraction of dataset to use for training/validation
        # optimization config
        device=list(range(TRAIN_GPU_COUNT)),  # use the GPU(s)
        epochs=8 if not quick_check else 1,  # pass over entire dataset this [illegible]
        batch=0.95,  # automatic batch size to target fraction of GPU util
        seed=117,  # set seed for reproducibility
        # data processing config
        workers=max(
            TRAIN_CPU_COUNT // TRAIN_GPU_COUNT, 1
        ),  # split CPUs evenly across GPUs
        cache=False,  # cache preprocessed images in RAM?
        # model saving config
        project=f"{volume_path}/runs",
        name=model_id,
        exist_ok=True,  # overwrite previous model if it exists
        verbose=True,  # detailed logs
    )

AIE
Braintrust
WorkOS
OpenAI

### SFT in 300 lines of code

Cowen describes fanning out on-demand containers for hyperparameter tuning and stopping unpromising runs. The slide links a YOLO training example.

[9:43](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=583s) [Slide describing parallel hyperparameter sweeps alongside a YOLO training code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0306b81bcb098412f6543c519cc83b9f8c90b2b4ca44d88d3fb22df62640fde.webp)

SFT in 300 lines of code

Once you have your data in place, open source libraries and serverless infra make parallel hyperparameter sweeps trivial.

https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/yolo/finetune_yolo.py

@app.function(
    gpu=TRAIN_GPU,
    cpu=TRAIN_CPU_COUNT,
    timeout=60 * MINUTES,
)
def train(
    model_id: str,
    dataset: DatasetConfig,
    model_size="yolov10m.pt",
    quick_check=False,
):
    from ultralytics import YOLO

    volume.reload()  # make sure volume is synced

    model_path = volume_path / "runs" / model_id
    model_path.mkdir(parents=True, exist_ok=True)

    data_path = volume_path / "dataset" / dataset.id / "data.yaml"

    model = YOLO(model_size)
    model.train(
        # dataset config
        data=data_path,
        fraction=0.4
        if not quick_check
        else 0.04,  # fraction of dataset to use for training/validation
        # optimization config
        device=list(range(TRAIN_GPU_COUNT)),  # use the GPU(s)
        epochs=8 if not quick_check else 1,  # pass over entire dataset this [illegible]
        batch=0.95,  # automatic batch size to target fraction of GPU util
        seed=117,  # set seed for reproducibility
        # data processing config
        workers=max(
            TRAIN_CPU_COUNT // TRAIN_GPU_COUNT, 1
        ),  # split CPUs evenly across GPUs
        cache=False,  # cache preprocessed images in RAM?
        # model saving config
        project=f"{volume_path}/runs",
        name=model_id,
        exist_ok=True,  # overwrite previous model if it exists
        verbose=True,  # detailed logs
    )

AIE

Engineering the future of AI

### GRPO in 300 lines of code

Modal combines GPU containers and sandboxes for parallel RL rollouts. Cowen says customers scaled up to 50,000 and 100,000 sandboxes for RL in the last quarter; the linked example trains GRPO in 300 lines.

[9:48](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=588s) [Slide describing GPU and sandbox APIs for reinforcement learning, alongside a GRPO training example.](https://assets.ai.engineer/sites/aiecode2025/sha256/fdfec4df93c924c79e3f72432a06fd1171ec3b7e51ebc40ff9b12c421403dde1.webp)

GRPO (RL) in 300 lines of code.

Unified GPU + Sandbox APIs makes large scale RL feasible.

https://github.com/modal-labs/modal-examples/tree/main/06_gpu_and_ml/reinforcement-learning

@app.function(
    image=image,
    gpu="H100",
    timeout=60 * 60 * 24,  # 24 hours
    secrets=[modal.Secret.from_name("wandb-secret")],
    volumes={"/models": checkpoints_volume},
)
def start_grpo_trainer(use_vllm=False, vllm_mode=None):
    dataset: Dataset = load_dataset(
        "OpenCoder-LLM/opc-sft-stage2", "educational_instruct", split="train"
    )
    dataset = dataset.rename_column(
        "instruction", "prompt"
    )  # Needed for the GRPO trainer
    dataset = dataset.rename_column("testcase", "testcases")
    dataset = dataset.select(range(128))  # To simplify testing.
    training_args: GRPOConfig = GRPOConfig(
        output_dir=str(MODELS_DIR),
        report_to="wandb",
        use_vllm=use_vllm,
        vllm_mode=vllm_mode,
        save_steps=1,
        max_steps=5,  # To simplify testing. Remove for production use cases.
    )
    trainer = GRPOTrainer(
        model="Qwen/Qwen2-0.5B-Instruct",
        reward_funcs=reward_helper_function,
        args=training_args,
        train_dataset=dataset,
    )
    trainer.train()

AIE
CONFIDENTIAL - DO NOT DISTRIBUTE
12
Engineering the future of AI

Interface text:
Copy

### vLLM inference in 200 lines of code

After training, the example serves a model with vLLM: a Modal web server builds the command, configures GPU parallelism and launches it as a subprocess.

[10:53](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=653s) [Slide showing a vLLM inference example, a code excerpt, and a link to the example repository.](https://assets.ai.engineer/sites/aiecode2025/sha256/9eecad7839ed7998fa0a2f9a4c337049510bfe33d8947b90b97b59839e5f820f.webp)

vLLM inference in 200 lines of code.

https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_inference.py

@modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES)
def serve():
    import json
    import subprocess

    cmd = [
        "vllm", "serve", MODEL_NAME,
        "--revision", MODEL_REVISION,
        "--served-model-name", MODEL_NAME,
        "llm", "--host", "0.0.0.0", "--port", str(VLLM_PORT),
        "--uvicorn-log-level=info", "--async-scheduling",
    ]

    # enforce-eager disables both Torch compilation and CUDA graph captu[illegible]
    # default is no-enforce-eager. see the --compilation-config flag for [illegible]
    cmd += ["--enforce-eager" if FAST_BOOT else "--no-enforce-eager"]

    # assume multiple GPUs are for splitting up large matrix multiplicat[illegible]
    cmd += ["--tensor-parallel-size", str(N_GPU)]

    # add model-specific configuration
    cmd += [
        # skip multimedia support, just language
        "--limit-mm-per-prompt",
        f"'{json.dumps({'image': 0, 'video': 0, 'audio': 0})}'",
        # enable reasoning and tool use
        "--enable-auto-tool-choice",
        "--reasoning-parser gemma4",
        "--tool-call-parser gemma4",
    ]

    print(*cmd)

    subprocess.Popen(" ".join(cmd), shell=True)

AIE
CONFIDENTIAL - DO NOT DISTRIBUTE
AI Engineer EUROPE

### Modal contact and code

Modal invites attendees to visit its booth or reach out for code. Cowen suggests preparing for possible model training by collecting data and developing evals.

[11:33](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=693s) [Modal closing slide inviting attendees to visit the booth or get in touch for code, with a startup webpage and contact address.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff13c96a4e67880eed46a6e3bccd7b58d81b4a4ca117516f1b3756a0a2e5afe7.webp)

Modal

Stop by our booth or reach out for code!

modal.com/startups
[redacted]

CONFIDENTIAL - DO NOT DISTRIBUTE
14

AIE
Engineering the future of AI

## Full transcript

[0:00](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=0s) [upbeat music] Yeah.

[0:15](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=15s) So good afternoon, and thanks for coming to this session. I know there's a lot to choose from. Uh, my name is Ben Cowen. I'm a Forward Deployed Machine Learning Engineer in Modal.

[0:25](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=25s) And, uh, I wanna talk about an interesting pattern that we've been seeing, uh, in AI application development. I'll give you the punchline now. Um, it's about, uh, as companies mature and their products mature and specialize, we're seeing more and more turn to fine-tuning to get impr- increased performance, better cost, and so forth.

[0:47](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=47s) So this brings up an interesting question of when does your application step over the line into a custom domain? When is fine-tuning worth it?

[1:00](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=60s) Um, so if you're not familiar with Modal, uh, we're a general purpose serverless compute platform. Uh, we provide sort of basic building blocks, like serverless functions and, uh, like hardened sandboxes for code execution.

[1:15](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=75s) And so as an FDE on a general purpose platform, I, I've had the opportunity to work with sort of a extremely wide range of AI applications. So from physics simulations to, uh, quantum chemistry and of course like voice processing, LLMs and agents.

[1:34](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=94s) A really interesting, um, one that's been really blowing up is large-scale reinforcement learning. Um, and so we've started to think about, um, some of these customers as where, where are they on the model spectrum?

[1:50](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=110s) So on one side of the spectrum is the frontier API. And the frontier APIs, uh, I think everyone here would agree has unlocked a completely new era of s-- uh, you know, accelerated growth.

[2:06](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=126s) Um, people can build basically anything exceptionally fast. Um, they're amazing. Um, but, uh, you can't customize it at all, uh, beyond prompt engineering. And so, um, you might y-- I mean, uh, I love the, uh, the whole caveman mode thing.

[2:23](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=143s) If you tell your LLM to speak like a caveman, you can reduce your tokens by like a lot. But that's not gonna scale if your startup, you know, hundred Xs or a thousand Xs, right?

[2:36](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=156s) Um, another, uh, kind of interesting thing we see is when, uh, startups win large enterprise contracts with very specific latency or throughput requirements. Um, there's very little ability to customize for those things, let alone if you have a custom metric that encapsulates your business logic.

[2:57](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=177s) Okay. So to get this model differentiation, a lot of companies turn to fine-tuning. And what that has meant traditionally is this huge jump to the other end of the spectrum, right?

[3:11](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=191s) We-- Training has a very different like scaling and compute characteristic to most production workloads. So if you wanna train and serve a production workload, in the past you have to get a big cluster.

[3:24](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=204s) Now you have to isolate that, those resources from your production resources. You're gonna need infrastructure engineers, or your AI engineers are gonna be working on infrastructure, maybe even your scientists.

[3:37](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=217s) So with this kind of extremely customized, uh, you know, powerful option, you also have this massive responsibility for the entire stack. Um, and so, uh, you might g-have a guess who I would recommend for this, but there's sort of a middle ground that's emerging.

[3:57](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=237s) There's a new type of cloud provider that makes this a lot easier. And, um, you know, we're, we're building this to address, uh, this problem that we're seeing, right?

[4:10](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=250s) So leader after leader in the space are announcing or publishing that they've fine-tuned and gotten incredible results. Okay. So Intercom is beating their frontier API at one-fifth the cost.

[4:27](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=267s) Um, Pinterest says orders of magnitude. I wish I knew the exact amount. But I think, uh, one of our customers, Decagon, has summed this up really well, which is that, um, basically the frontier labs probably don't have the exact same goal as you, right?

[4:44](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=284s) They want their models to win on everything possible. And, um, we want our models to win at our business logic, right? You wanna be the best at what you provide your customer.

[4:57](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=297s) Um, and so, yeah, I'm happy to announce that it's actually a lot easier than you might think to train a model. Um, with-- There's some incredible open source libraries out there now that make this extremely accessible.

[5:14](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=314s) They give you, uh, full control over the algorithm, right? So you get to kind of reach across the spectrum to doing it yourself at the algorithm level without having to also manage the cluster and so forth.

[5:29](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=329s) And the most important thing is that this retains the fast iteration cycles of the frontier end of the spectrum. Um, so that's basically our entire mission is to give you algorithm control and fast iteration.

[5:47](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=347s) Um, so this is my hot take that it's just a matter of time until your product steps into being domain specific, right? So in some sense, like if you have a differentiated product, it is custom, right?

[6:02](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=362s) So, uh- When exactly you cross that line, that's s- a decision you have to make, but it's something we'd love to talk to you about. Um, so I, I have here a few signals that might indicate that, you know, you're getting close to that time.

[6:20](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=380s) So if you've, uh, moved to caveman mode and you're still paying more for your API than your customers are paying you, that might be a signal that your economics aren't scaling, right?

[6:32](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=392s) And that you, you could probably benefit from a customized inference endpoint. Um,

[6:39](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=399s) same for latency and throughput, right? So if you, if you are plateauing on your evals, that is a-- that's a signal that you might get something out of fine-tuning a model.

[6:50](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=410s) Um, there's a decades-old adage in training that if you have garbage data, it's garbage in, garbage out. So if you haven't been collecting data and you don't have mature evals, it's probably not time to train.

[7:05](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=425s) You need to collect the data. Um, that said, uh, this is one of the main takeaways that I'd love for everyone here to walk out with, is that if you have built a product, you probably have at least touched all of the things you need to train if you haven't already done it.

[7:24](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=444s) Okay? If you've built an agent harness, then you have what you need to have a new model, you know, learn through reinforcement learning how to provide your service, right?

[7:36](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=456s) So if you're evaluating your products, um, and collecting that data on what's working and what's not, then you have training data to train your model. Um, and with, you know, the advent of, uh, serverless compute platforms and, uh, these open source libraries, I don't know, like, uh, a lot of us when we started training models, we were

[8:00](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=480s) taking the gradient by hand and like implementing the linear algebra. You don't, you don't have to do that anymore unless you have a freaky model, um, which if you do, I'd love to talk to you about it.

[8:12](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=492s) Um, but yeah, you don't need the infrastructure experts, um, and so forth. So this is an exciting time.

[8:21](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=501s) Um, I knew the vi-video wouldn't play. Anyway, so, uh, the next couple slides are just some snippets of code. I don't expect-- Yeah, you can't even really read it, but the-- I just wanna, uh, k-kind of illustrate what it looks like to set up a training algorithm today.

[8:39](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=519s) It's not, uh, a gigantic monorepo with thousands of lines of code. You can do supervised fine-tuning in three hundred lines of Python.

[8:49](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=529s) Okay, so once you have your data curated, uh, once you have an account on Modal or some other serverless platform, you can get started really fast, and this code is on our examples repository.

[9:02](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=542s) Um, the, the thing in this video, it's just showing how, um, we can scale containers really fast. And so just to bridge, bridge these concepts a little bit, people usually associate serverless with, uh, inference.

[9:18](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=558s) But with training it can be really, really handy too, uh, for doing something like hyperparameter tuning, right? You, you can-- You don't have to, uh, you know, every minute on your cluster isn't sacred anymore.

[9:33](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=573s) You can fan out to a bunch of containers, get them on demand. As soon as it's not promising, kill it. And y- It's kind of, uh, almost like an, a meta evolutionary algorithm at that point.

[9:47](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=587s) Um, so it's, it's a exciting time to be doing that. And the same goes for reinforcement learning. Um, a lot of us who got our, you know, graduate degrees in machine learning in the last ten years, uh, didn't do reinforcement learning, right?

[10:01](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=601s) This is relatively, I mean, it's actually really old, but the stuff we're using today is pr- kind of new. But they have these libraries too. Um, you can do it in three hundred lines of code.

[10:13](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=613s) Um, and something interesting about Modal part- in particular is we have sort of unified APIs for sandboxes and GPU containers or clusters. Um, so what this means in a nutshell, when you're training a model with RL, it needs to sort of practice a lot and so this is massively embarrassingly parallel kind of evaluation thing called a rollout.

[10:38](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=638s) And so we have one of the most amazing things in the last quarter has been customers scaling up to fifty thousand, a hundred thousand sandboxes just to do RL.

[10:48](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=648s) Um, and you can do it too. Uh, the code is open source.

[10:55](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=655s) And then I'd be remiss not to mention what comes after the training. You have to serve the model, right? And this is what the a-- the frontier API is doing, you know, under the hood.

[11:07](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=667s) Um, well, they pro-- I don't know if they use vLLM, but my point is that you can do it too, and it's actually not that much code. vLLM, SGLang, Triton Inference Server, or a custom inference workflow with just Python, um, on Modal or other, you know, serverless platforms, you can autoscale all of this stuff, uh, to match

[11:30](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=690s) your traffic as it's coming in. So yeah. So, uh, just to kind of sum everything up what I'm saying here, I'm not saying go train your model right now.

[11:42](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=702s) I'm saying it's not something that is like, "Oh, I'll do that in ten years." You might, you might wanna train your model in one year, right? You might wanna do it in six months.

[11:53](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=713s) So start thinking about what is... when am I gonna know, okay, it's time to train my model, and how can I prepare for that moment by collecting data, developing your evals.

[12:06](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=726s) And yeah, I'd love to, you know, come by our booth. We're like at kind of the end over on that side. Uh, love to talk to you more about this, or you can reach out, uh, at my email here.

[12:18](https://www.youtube.com/watch?v=HvZXAOZ3iv8&t=738s) That's it. [clapping] [outro music]
