What Lies Beneath the API — Benjamin Cowen, Modal

When does an application become specialized enough that fine-tuning is worth the effort? A general-purpose model can get a product started quickly, but a maturing product may need better performance on a narrower task or a different cost structure. That is the decision Benjamin Cowen opens with: when to move from consuming a model to shaping one around the service you provide.

Cowen approaches this question through his work at Modal, a general-purpose serverless compute platform providing functions and hardened sandboxes for code execution. The workloads span physics simulations, quantum chemistry, voice processing, LLMs, and agents. Growing demand for large-scale reinforcement learning adds another reason to think about where an application sits on the spectrum between a managed model API and a fully custom stack.

Frontier APIs make it possible to build useful applications exceptionally quickly, without first operating model infrastructure. In the API-only setup Cowen describes, prompting is the main customization surface; the contrast is with controlling training and serving yourself, rather than a claim that every hosted model service lacks fine-tuning.

Consider his caveman-mode example: ask the model to speak tersely, and shorter answers consume fewer tokens. That can reduce spending, but it does not establish that the economics will survive a startup growing a hundredfold or a thousandfold. An enterprise contract can also introduce specific latency and throughput requirements—or a quality metric that encodes the customer's business logic. Those requirements reach beyond the wording of a prompt.

Traditionally, gaining that control meant jumping to the other end of the spectrum. Training has different compute and scaling characteristics from production inference. Running both on a large cluster requires isolating training resources from serving resources, then assigning infrastructure work to infrastructure engineers—or to AI engineers and scientists who would otherwise work on the model.

The emerging middle ground is a cloud platform that makes custom workloads easier to operate. Its purpose is to reduce the infrastructure responsibility attached to model differentiation.

Cowen reports that Intercom's fine-tuned model beat its frontier API baseline at one-fifth the cost. He does not specify the evaluation, workload, or cost-accounting conditions, so this is an attributed customer example rather than a general performance guarantee. His Pinterest example likewise invokes orders-of-magnitude savings without an exact multiplier; that comparison concerns cost, not an orders-of-magnitude improvement in model quality.

The rationale he attributes to Decagon is more broadly useful than either number: frontier labs pursue models that perform well across many tasks, while a business needs its model to excel at its own service. The objective is business-specific performance, which may involve a particular workflow, response behavior, or operational constraint.

Open-source training libraries make the algorithm accessible without requiring the same team to manage a cluster. Combined with managed compute, they offer a way to reach toward custom training while retaining the fast iteration that made APIs attractive in the first place. Algorithm control and iteration speed are the two properties Cowen wants to preserve together.

Cowen's expectation is that a differentiated product will eventually develop domain-specific requirements. That is a judgment about product development, not a universal deadline. The useful question is which observable signals would justify customization for your application.

• Unit economics: API spending exceeds what customers pay, even after reducing unnecessary output. A customized inference endpoint becomes worth investigating.
• Latency and throughput: The service needs performance characteristics that the current endpoint cannot deliver.
• Evaluation progress: Results have plateaued, suggesting that further prompting may not produce the improvement the product needs.

These signals identify a problem worth solving; they do not by themselves establish that fine-tuning is the solution.

Data and mature evaluations are prerequisites. If you have not been collecting useful data and cannot reliably evaluate the result, Cowen advises against training yet. Poor inputs remain poor inputs after a training run.

Existing product work can supply much of what is missing. An agent harness provides an environment in which a model can practice delivering the service through reinforcement learning. Evaluations and records of what worked or failed provide potential training data. The harness supplies interactions; the evaluation process supplies evidence about their outcomes.

Modern libraries also remove much of the need to calculate gradients or implement linear algebra by hand. Managed infrastructure reduces another part of the burden. The opportunity is to build on the product's existing harness, data, and evaluations rather than treating training as an entirely separate engineering discipline.

The training example begins with code slides after the embedded video fails to play. Cowen describes a supervised fine-tuning setup in roughly 300 lines of Python. That is the size of the example, not an estimate of all the work required to curate data, evaluate a model, and operate it. With curated data and an account on Modal or another serverless platform, the setup becomes a practical starting point; he points to the Modal examples repository.

Serverless scaling is useful for more than inference. In hyperparameter search, different configurations can run in separate containers requested on demand. Instead of treating every minute on a fixed cluster as scarce, the team can launch parallel trials and stop the ones that cease to look promising.

A small Python example makes the pruning decision concrete. Suppose three running trials report the same evaluation metric at a comparable checkpoint, where higher is better:

from dataclasses import dataclass, replace

@dataclass(frozen=True)
class Trial:
    name: str
    learning_rate: float
    eval_score: float
    status: str = "running"

def propose_pruning(trials: list[Trial], minimum_score: float) -> list[Trial]:
    return [
        replace(trial, status="stop_requested")
        if trial.status == "running" and trial.eval_score < minimum_score
        else trial
        for trial in trials
    ]

trials = [
    Trial("a", 1e-5, 0.72),
    Trial("b", 3e-5, 0.41),
    Trial("c", 1e-4, 0.68),
]

for trial in propose_pruning(trials, minimum_score=0.60):
    print(trial.name, trial.status)

Trial b becomes stop_requested; a and c remain running. This records a proposed action. The orchestration layer must actually terminate the selected container to stop its compute consumption. The search-and-prune loop is what Cowen likens to a meta evolutionary algorithm: explore alternatives, discard weak candidates, and keep resources on promising ones.

Reinforcement learning is longstanding, but its current tooling makes it accessible to engineers who did not previously specialize in it. Cowen describes an RL example in roughly 300 lines of code. As with the supervised example, the claim concerns a compact setup built on libraries, not the total effort of developing a reliable training system.

The execution pattern explains why serverless infrastructure is useful here. A model needs repeated opportunities to practice. These interactions and evaluations form rollouts, many of which can run independently. Modal exposes unified APIs for sandboxes and GPU containers or clusters, bringing the environments used for practice and the compute used for model work into the same programming surface. The slide presents this as a GRPO reinforcement-learning example.

Cowen reports customers scaling to 50,000–100,000 sandboxes for reinforcement learning during the preceding quarter. He does not specify whether those counts were concurrent, how long they ran, or a measured throughput result. The architectural point is the large parallel demand created by rollouts, with open-source example code available to illustrate the setup.

Training leaves one essential job unfinished: serving the model. The inference slide presents a vLLM example in 200 lines of code. Cowen also names SGLang, Triton Inference Server, and custom Python inference workflows as options. These perform the serving role hidden behind a frontier API; he explicitly does not claim to know whether frontier providers use vLLM.

On Modal or another suitable serverless platform, the serving workload can autoscale with incoming traffic. This completes the lifecycle: train the model, put it behind an inference service, and scale that service to demand. It also returns to the original decision. Having an accessible path through the stack is not a reason to train immediately.

The practical horizon may be six months or a year rather than a distant future project. Those are planning examples, not deadlines. Decide how you will recognize the moment when customization is justified, then collect the data and develop the evaluations that will let you act on that decision. Preparation makes training an available option before cost, latency, or product quality turns it into an urgent one.