Scaling Reinforcement Learning: Lessons from Trillion-Token Deployments at Fortune 500s

How do you keep improving a language model once the demo works but production requirements remain unmet? AdaptiveML’s enterprise work centers on that problem: building, evaluating, and serving specialized models for customers including AT&T, Manulife, and CCS. Alessandro Cappelli, its co-founder and chief customer officer, traces this focus to his team’s experience training Falcon. In his account, reinforcement learning was the missing ingredient between an open model and the production behavior of proprietary offerings.

The misleading assumption is that creating an MVP is the hard part and production is merely the last mile. Cappelli cites a 95% failure-to-production figure for GenAI pilots, without identifying its source. Treat that as his framing, not an established universal failure rate: lack of financial return and failure to deploy are different outcomes.

Consider the maintenance loop after testing reveals a defect. In the prompt-based workflow Cappelli describes, you change the system prompt to correct it, only to introduce a different defect. With supervised fine-tuning (SFT), the corresponding intervention is to revise the training dataset. That may be expensive, and deployment does not end the need for revisions. Neither intervention, by itself, specifies how each new defect becomes a measured improvement.

The MVP is the first mile of an ongoing model lifecycle. The remaining work is to collect client feedback, business metrics, and environmental rewards, then use them to retrain, evaluate, and refine the model. Cappelli presents RL as the systematic mechanism for closing that loop; his stronger claim that only RL can do so is the thesis of the talk, rather than a demonstrated exclusion of other methods.

Prompting, SFT, and RL all steer model behavior. Cappelli’s comparison is about how much useful behavior each can produce at a given model size. He describes RL matching SFT performance with a smaller model, which would move the deployment decision from buying more general capability toward training the capability the task actually needs. The performance–cost chart is conceptual: it supplies no task, benchmark conditions, or numerical results establishing that advantage.

At enterprise adoption levels, serving expense can determine whether a useful feature becomes commonplace. Cappelli says AT&T spends millions of dollars summarizing customer–agent transcripts, without specifying an accounting period. His point is that a repeated, narrow task can justify a specialized model because even a modest change in serving cost matters across the workload.

Latency creates a separate constraint. For speech-to-speech customer support, Cappelli describes half a second as an upper bound and roughly one-third of a second as preferable. He associates that target with models around the 10B parameter range, naming Gemma, Mistral, and Qwen. These are his deployment targets, not hardware-independent limits: model size alone does not establish end-to-end speech latency.

The final benefit is control over the deployed solution. A model trained on the business’s own data can be evaluated and released on the business’s schedule, instead of changing behavior when an upstream provider updates its model. Together, the proposed benefits are lower serving cost, stronger task performance, lower latency, and ownership of the specialized system.

Summarization, classification, and OCR already raise questions of cost and reliability. Agents add longer workflows and actions that affect external systems. An agent may read customer data or modify a database, so an error can change something beyond the generated answer. Cappelli asks what happens if an agent uses ten times the tokens of a summarization workflow; this is an illustrative cost scenario, not a measured deployment result.

That added complexity also gives RL a concrete training setting. An agent acts in an environment, receives responses to its actions, and is assessed on the resulting behavior. Rather than treating every output as an isolated answer, training can use the interaction between the model and the systems it operates.

There are two starting points:

• Reuse an existing workflow. Cappelli describes Manulife as already having an agent environment. The training system can plug a model such as Qwen 3.5 into that workflow instead of recreating it.
• Construct an environment. When no suitable workflow exists, mock the tools and, if the task involves a conversation, provide a simulated user. Another LLM can play that user.

In both cases, the environment supplies the interaction; the reward defines which interactions count as successful.

Success might mean achieving a business outcome, improving a KPI, being helpful, or following the organization’s tone and vocabulary guidelines. An LLM judge can assess criteria that are difficult to check mechanically. Cappelli also points to a workshop by his colleagues Laetitia and João on connecting model training to an existing environment.

A company may have plenty of business data but little suitable agent training data. What is missing is often a trajectory: a sequence of model decisions, tool interactions, and responses showing how to complete that company’s task. Generic examples from elsewhere need not capture its tools or operating conditions.

Once an environment and reward exist, they also form a synthetic data pipeline:

1. Run candidate agent interactions in the environment.
2. Score the resulting trajectories using the reward.
3. Use rejection sampling to retain suitable examples.
4. Use the retained dataset to bootstrap initial model training.

The reward supplies a selection criterion, so dataset construction no longer depends entirely on people writing successful demonstrations.

A small Python example makes the selection step explicit. These teaching records represent already-scored candidate trajectories; filtering them prepares a dataset but does not itself train a model.

from dataclasses import dataclass

@dataclass(frozen=True)
class Trajectory:
    id: str
    request: str
    actions: tuple[str, ...]
    reward: float

candidates = [
    Trajectory(
        id="run-a",
        request="Where is my order?",
        actions=("lookup_order", "report_status"),
        reward=1.0,
    ),
    Trajectory(
        id="run-b",
        request="Where is my order?",
        actions=("guess_status",),
        reward=0.0,
    ),
]

minimum_reward = 1.0
training_dataset = [
    trajectory
    for trajectory in candidates
    if trajectory.reward >= minimum_reward
]

The selected record preserves the request and action sequence. The quality of that dataset still depends on what the reward considers good.

Existing customer conversations can improve the environment even when they are not agent demonstrations. Give those transcripts to the mock user, or train the mock user on them, and it can reproduce repeated requests, difficult exchanges, and people who do not follow a tidy script. Cappelli’s medical-supply example includes callers in panic. In such scenarios, the desired behavior might be escalation to a human or contacting emergency services—illustrations of behavior the environment should represent, not a validated emergency-response protocol.

ChatGPT helped make reinforcement learning from human feedback, or RLHF, widely known. But keeping humans involved does not have to mean repeatedly commissioning large annotation campaigns. Cappelli describes those campaigns as expensive and difficult to sustain. His alternative places human effort in defining what the reward should recognize.

Different requirements call for different reward sources:

For CCS, the medical-supply customer, Cappelli identifies containment rate as a business objective: the share of calls the model handles end to end. That objective sits alongside the earlier examples where escalation is appropriate; the talk does not specify how the reward combines those competing demands.

Humans define the judge’s rubrics and system prompt, and choose scenarios that resemble actual interactions. The judge can then apply those criteria across generated trajectories. Cappelli estimates this rubric and scenario work at minutes to hours rather than weeks of repeated annotation, based on his experience. The human role shifts toward specifying and checking the evaluation standard.

Accelerating the lifecycle means more than shortening a training run. Cappelli presents Adaptive Engine as an integrated platform for evaluating, tuning, observing, and serving models. The purpose of joining those operations is to detect behavioral defects before and after deployment, then act on them through another measured iteration. These are the platform capabilities he describes, rather than independently established deployment results.

The starting model remains a choice: Cappelli names the Gemma, Mistral, and Qwen families as foundations for specialization. The operational challenge comes after that choice. RL is harder to implement than editing a prompt or assembling an instruction-tuning dataset.

Cappelli describes a common LLM PPO setup as orchestrating four language models simultaneously. That is a description of a particular orchestration pattern, not a universal requirement of Proximal Policy Optimization (PPO). Adaptive Engine’s proposed division of labor is for customers to define the rubrics while the platform manages the training machinery.

Prebuilt recipes are intended to remove the need for each customer to implement algorithms such as Group Sequence Policy Optimization (GSPO) and assemble the training process themselves. This is the infrastructure behind Cappelli’s production thesis: make feedback-driven improvement repeatable by managing the surrounding evaluation and serving lifecycle as well as the optimization algorithm.

The audience question makes the feedback problem more specific. Citing Cursor’s online RL work on Tab completion, the questioner asks about implicit production signals such as accepting or rejecting a suggestion. Unlike a training setup that samples several outputs for a prompt, a production interaction may expose only one output and one response. Should the system replay the problem to generate variations, or can that single observed signal directly support training?

Cappelli answers by distinguishing stages in the availability of feedback. With roughly 10–20 early human feedback examples, his team typically refines the LLM judge’s description of good and bad behavior. Those examples help clarify how the existing evaluation criteria should apply.

Initially, the judge can be a prompted large model; Cappelli gives Qwen 235B as an example. With thousands of production feedback examples, his team moves toward training reward models that can scale that feedback into further LLM training. The progression is from people clarifying a prompted evaluator to production data supporting a learned evaluator.

Implicit feedback still requires a use-case-specific interpretation. Cappelli proposes building the reward model accordingly and comparing alternative training approaches against a defined evaluation using the available data. He does not explicitly choose replay over direct training from a single event. The practical endpoint is an empirical decision: select the approach that improves the behavior the evaluation is intended to measure.