AI Engineer Europe 2026
Scaling Reinforcement Learning: Lessons from Trillion-Token Deployments at Fortune 500s
Read the talk
Reinforcement Learning Across the Production Model Lifecycle
Getting a model to production requires more than a convincing demo: it requires a repeatable way to turn defects, business outcomes, and user feedback into better behavior.
From a talk by Alessandro Cappelli
Before you start: Familiarity with language-model prompting and fine-tuning is helpful; reinforcement learning concepts are introduced as they arise.
What happens after the demo works?
How do you keep improving a language model once the demo works but production requirements remain unmet? AdaptiveML’s enterprise work centers on that problem: building, evaluating, and serving specialized models for customers including AT&T, Manulife, and CCS. Alessandro Cappelli, its co-founder and chief customer officer, traces this focus to his team’s experience training Falcon. In his account, reinforcement learning was the missing ingredient between an open model and the production behavior of proprietary offerings.
The misleading assumption is that creating an MVP is the hard part and production is merely the last mile. Cappelli cites a 95% failure-to-production figure for GenAI pilots, without identifying its source. Treat that as his framing, not an established universal failure rate: lack of financial return and failure to deploy are different outcomes.
Consider the maintenance loop after testing reveals a defect. In the prompt-based workflow Cappelli describes, you change the system prompt to correct it, only to introduce a different defect. With supervised fine-tuning (SFT), the corresponding intervention is to revise the training dataset. That may be expensive, and deployment does not end the need for revisions. Neither intervention, by itself, specifies how each new defect becomes a measured improvement.
The MVP is the first mile of an ongoing model lifecycle. The remaining work is to collect client feedback, business metrics, and environmental rewards, then use them to retrain, evaluate, and refine the model. Cappelli presents RL as the systematic mechanism for closing that loop; his stronger claim that only RL can do so is the thesis of the talk, rather than a demonstrated exclusion of other methods.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialization changes the serving economics
Prompting, SFT, and RL all steer model behavior. Cappelli’s comparison is about how much useful behavior each can produce at a given model size. He describes RL matching SFT performance with a smaller model, which would move the deployment decision from buying more general capability toward training the capability the task actually needs. The performance–cost chart is conceptual: it supplies no task, benchmark conditions, or numerical results establishing that advantage.
At enterprise adoption levels, serving expense can determine whether a useful feature becomes commonplace. Cappelli says AT&T spends millions of dollars summarizing customer–agent transcripts, without specifying an accounting period. His point is that a repeated, narrow task can justify a specialized model because even a modest change in serving cost matters across the workload.
Latency creates a separate constraint. For speech-to-speech customer support, Cappelli describes half a second as an upper bound and roughly one-third of a second as preferable. He associates that target with models around the 10B parameter range, naming Gemma, Mistral, and Qwen. These are his deployment targets, not hardware-independent limits: model size alone does not establish end-to-end speech latency.
The final benefit is control over the deployed solution. A model trained on the business’s own data can be evaluated and released on the business’s schedule, instead of changing behavior when an upstream provider updates its model. Together, the proposed benefits are lower serving cost, stronger task performance, lower latency, and ownership of the specialized system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train agents where their actions have consequences
Summarization, classification, and OCR already raise questions of cost and reliability. Agents add longer workflows and actions that affect external systems. An agent may read customer data or modify a database, so an error can change something beyond the generated answer. Cappelli asks what happens if an agent uses ten times the tokens of a summarization workflow; this is an illustrative cost scenario, not a measured deployment result.
That added complexity also gives RL a concrete training setting. An agent acts in an environment, receives responses to its actions, and is assessed on the resulting behavior. Rather than treating every output as an isolated answer, training can use the interaction between the model and the systems it operates.
There are two starting points:
- Reuse an existing workflow. Cappelli describes Manulife as already having an agent environment. The training system can plug a model such as Qwen 3.5 into that workflow instead of recreating it.
- Construct an environment. When no suitable workflow exists, mock the tools and, if the task involves a conversation, provide a simulated user. Another LLM can play that user.
In both cases, the environment supplies the interaction; the reward defines which interactions count as successful.
Success might mean achieving a business outcome, improving a KPI, being helpful, or following the organization’s tone and vocabulary guidelines. An LLM judge can assess criteria that are difficult to check mechanically. Cappelli also points to a workshop by his colleagues Laetitia and João on connecting model training to an existing environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The environment becomes a source of training data
A company may have plenty of business data but little suitable agent training data. What is missing is often a trajectory: a sequence of model decisions, tool interactions, and responses showing how to complete that company’s task. Generic examples from elsewhere need not capture its tools or operating conditions.
Once an environment and reward exist, they also form a synthetic data pipeline:
- Run candidate agent interactions in the environment.
- Score the resulting trajectories using the reward.
- Use rejection sampling to retain suitable examples.
- Use the retained dataset to bootstrap initial model training.
The reward supplies a selection criterion, so dataset construction no longer depends entirely on people writing successful demonstrations.
A small Python example makes the selection step explicit. These teaching records represent already-scored candidate trajectories; filtering them prepares a dataset but does not itself train a model.
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Trajectory:
id: str
request: str
actions: tuple[str, ...]
reward: float
candidates = [
Trajectory(
id="run-a",
request="Where is my order?",
actions=("lookup_order", "report_status"),
reward=1.0,
),
Trajectory(
id="run-b",
request="Where is my order?",
actions=("guess_status",),
reward=0.0,
),
]
minimum_reward = 1.0
training_dataset = [
trajectory
for trajectory in candidates
if trajectory.reward >= minimum_reward
]
The selected record preserves the request and action sequence. The quality of that dataset still depends on what the reward considers good.
Existing customer conversations can improve the environment even when they are not agent demonstrations. Give those transcripts to the mock user, or train the mock user on them, and it can reproduce repeated requests, difficult exchanges, and people who do not follow a tidy script. Cappelli’s medical-supply example includes callers in panic. In such scenarios, the desired behavior might be escalation to a human or contacting emergency services—illustrations of behavior the environment should represent, not a validated emergency-response protocol.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put human judgment into the reward definition
ChatGPT helped make reinforcement learning from human feedback, or RLHF, widely known. But keeping humans involved does not have to mean repeatedly commissioning large annotation campaigns. Cappelli describes those campaigns as expensive and difficult to sustain. His alternative places human effort in defining what the reward should recognize.
Different requirements call for different reward sources:
| Reward source | Example | What it assesses |
|---|---|---|
| Mechanical check | Code runs; syntax is valid | A directly testable property |
| Business outcome | Customer-support containment | Whether the model completes the call |
| LLM judge | Tone and business requirements | Open-ended behavioral criteria |
For CCS, the medical-supply customer, Cappelli identifies containment rate as a business objective: the share of calls the model handles end to end. That objective sits alongside the earlier examples where escalation is appropriate; the talk does not specify how the reward combines those competing demands.
Humans define the judge’s rubrics and system prompt, and choose scenarios that resemble actual interactions. The judge can then apply those criteria across generated trajectories. Cappelli estimates this rubric and scenario work at minutes to hours rather than weeks of repeated annotation, based on his experience. The human role shifts toward specifying and checking the evaluation standard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Operationalize evaluation, training, and serving together
Accelerating the lifecycle means more than shortening a training run. Cappelli presents Adaptive Engine as an integrated platform for evaluating, tuning, observing, and serving models. The purpose of joining those operations is to detect behavioral defects before and after deployment, then act on them through another measured iteration. These are the platform capabilities he describes, rather than independently established deployment results.
The starting model remains a choice: Cappelli names the Gemma, Mistral, and Qwen families as foundations for specialization. The operational challenge comes after that choice. RL is harder to implement than editing a prompt or assembling an instruction-tuning dataset.
Cappelli describes a common LLM PPO setup as orchestrating four language models simultaneously. That is a description of a particular orchestration pattern, not a universal requirement of Proximal Policy Optimization (PPO). Adaptive Engine’s proposed division of labor is for customers to define the rubrics while the platform manages the training machinery.
Prebuilt recipes are intended to remove the need for each customer to implement algorithms such as Group Sequence Policy Optimization (GSPO) and assemble the training process themselves. This is the infrastructure behind Cappelli’s production thesis: make feedback-driven improvement repeatable by managing the surrounding evaluation and serving lifecycle as well as the optimization algorithm.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a single feedback event to a learned reward
The audience question makes the feedback problem more specific. Citing Cursor’s online RL work on Tab completion, the questioner asks about implicit production signals such as accepting or rejecting a suggestion. Unlike a training setup that samples several outputs for a prompt, a production interaction may expose only one output and one response. Should the system replay the problem to generate variations, or can that single observed signal directly support training?
Cappelli answers by distinguishing stages in the availability of feedback. With roughly 10–20 early human feedback examples, his team typically refines the LLM judge’s description of good and bad behavior. Those examples help clarify how the existing evaluation criteria should apply.
Initially, the judge can be a prompted large model; Cappelli gives Qwen 235B as an example. With thousands of production feedback examples, his team moves toward training reward models that can scale that feedback into further LLM training. The progression is from people clarifying a prompted evaluator to production data supporting a learned evaluator.
Implicit feedback still requires a use-case-specific interpretation. Cappelli proposes building the reward model accordingly and comparing alternative training approaches against a defined evaluation using the available data. He does not explicitly choose replay over direct training from a single event. The practical endpoint is an empirical decision: select the approach that improves the behavior the evaluation is intended to measure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Cursor explains how accepted and rejected suggestions become rewards, and why its training loop continually collects fresh production data.
The original GSPO paper describes sequence-level policy optimization for training language models.
Further reading
Adaptive ML's v0.7 guide shows how to configure models, datasets, graders, and training recipes.
A public mirror of the July 2025 preliminary report on enterprise AI adoption, deployment, and financial impact.
Updates since the talk
A May 2026 vendor case study describes AT&T's specialized models for call summarization, sensitive-data classification, and fraud review.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Hello, everyone.
- 0:15
Uh, I am Alessandro Cappelli. I'm co-founder and chief customer officer at AdaptiveML.
- 0:23
At AdaptiveML, we build an RL ops platform, as in reinforcement learning operation, that allows large enterprises like AT&T, Manulife, CCS to build, evaluate, and serve in production their own specialized large, large language models.
- 0:45
Um, I'm here to show you how reinforcement learning, RL, is not just any other algorithm for post-training, but is an, an algorithm that at its core will bring models to production.
- 1:01
Around three years ago, uh, I was part of a team that trained Falcon. Falcon, around three years ago, was one of the most widely adopted, uh, open source models.
- 1:13
And we realized with, you know, my team that is actually the core founding team of AdaptiveML, that the gap that was missing between bringing an open source model to production versus proprietary models like of frontier labs like OpenAI was actually reinforcement learning.
- 1:34
Ninety-five percent of GenAI pilots fail to reach production. Why that is the case? Uh, I believe it's what we call the myth of the last mile. Here you see, let's say, a time description of what does it get to get to production, which, uh, I believe i-it's false.
- 1:53
And this idea is that the hard part is to get to an MVP is to, you know, come up with a demo that looks nice in front of stakeholders, in front of your colleagues, and that is the, the hard part.
- 2:04
And then, you know, there will be just the last mile, where the last mile will actually get the model into production. The issue is that most MVP are built on top of proprietary models or are built on top of open source model using instruction fine-tuning.
- 2:18
Both of these solutions won't let you systematically improve your solution. They don't integrate in a really systematic and mathematical way what are the defects you might find, uh, in the journey to get to production.
- 2:32
I will give you an example. If you test, let's say, your solution and there will be some defects and you're using a proprietary models, all you can do is to change the system prompt.
- 2:43
Now, you change the system prompt in one direction, you might have other defects, and there is no really like a really scientific systematic way to improve that system prompt nicely in a way that you can easily monitor.
- 2:55
Likewise, for instruction fine-tuning. Best you can do is to iterate over the dataset. That might be expensive. And what after production? Will you keep creating a new dataset every single week?
- 3:07
This is what I believe is a more realistic view of what getting to production and beyond actually looks like. Getting to an MVP is not easy, but it's just the first mile.
- 3:20
What actually the real journey, the real marathon is to get from an MVP to production and beyond. And the secret to do that is to accelerate model lifecycle. Is to be able to integrate every single feedback you can get from a variety of sources to keep improving your solution.
- 3:39
This continuous retraining, refinement, and improvement driven by real client feedback, business metrics, and environmental reward is unlocked in a systematic way only by reinforcement learning.
- 3:52
Reinforcement learning, as I mentioned before, my entire point is that by almost by design, by nature, allows to integrate feedbacks in a almost a mathematical way. But reinforcement learning is not just like-- not just that.
- 4:06
Compared to other post-training techniques or steering behavior techniques like, you know, prompting and instruction fine-tuning, they all, let's say, reach the same goal, which is steer a model's behavior,
- 4:20
but they're not equally effective. Reinforcement learning is disproportionally more effective than instruction fine-tuning and likewise versus, versus, uh, prompting. Reinforcement learning unlocks, let's say, outsized performance. So what does it mean?
- 4:35
What you can see in the plot that, you know, you can get the same performance with a RL with respect to SFT with a much smaller model. What it unlocks that actually help you to get to production.
- 4:48
RL enabled scale at adoption. What do I mean by that? As you train smaller model, those models will be cheaper to serve at scale, and the tokenomics of your use case will eventually make sense.
- 5:02
When you are a big enterprise like AT&T, any use case, any feature that you want to be a commodity, either for internal employees or, you know, any, you know, facing customer features you might think at scale will cost you millions of dollars.
- 5:18
As an example, AT&T, they summarize every single transcript that might happen between, you know, a customer and an agent. Just summarizing that costs them millions of dollars.
- 5:31
If you can train a model that is much smaller than, you know, uh, you know, a, a ChatGPT or a Sonnet, you will save money. Another things you unlock is that smaller model will be faster.
- 5:44
Now, not all use cases require speed, but many of them, they have
- 5:49
a threshold of latency That is not just something nice to have, it's a constraint that will prevent you to get into production. Let's say you have a model for customer support that is powering, you know, a speech-to-speech system.
- 6:04
You can go above half of a second. And I would say half of a second is already weird. When you're talking to someone and you g-- and you have to wait half of a second, that's already weird.
- 6:12
Ideally, it should be a third of a second, and a third of a second is something you will never get if you're re-- using large language models. You need to use small models.
- 6:20
Might be, you know, the latest Gemma, the latest Mistral, the latest Qwen of, you know, in that ten B family, but you can use m-much larger models. The last thing you unlock is ownership.
- 6:31
You will own the data that you give to the model 'cause the model will be trained on your own business data, and you will own the solution. So you don't need to worry about, you know, the latest update of the model that may shift performance underneath your feet.
- 6:48
Everything I've said so far, it is true for any use case you might think. I've mentioned summarization. Could have been classification, uh, could have been OCR, could have been anything you can think of.
- 7:01
And reinforcement learning is already the better choice. But now we are in the era of agents, and agents actually make everything more complicated. Agents require more tokens, more complexity.
- 7:13
There is less room for errors because now agents will have access to the data, will change things in the database connected to, you know, either, you know, internal employees or clients you, you might have.
- 7:25
So all of that raises the standard of what can be brought into production, and it raises further question on whether the tokenomics of an agent actually makes sense or not.
- 7:36
I mentioned before, you know, just for a summarization use case, you might spend millions. Imagine if you scale an agent to actually ten X the number of tokens.
- 7:48
RL advantage that already existed only widens when it comes to, you know, training agents. 'Cause RL at its core was actually made to train robots, you know, to, to train agents, uh, to, to leave an environment.
- 8:02
And environments are where, you know, agents actually behave. So RL naturally fits in a narrative where you want to train a model to be, uh, a good agent. Now, there are two scenarios.
- 8:15
Uh, either you already have an agent in place. Uh, as an example, you know, like, um, we work for, uh, uh, Manulife, and Manulife already have agents, so they already have a NACA workflow that has been settled.
- 8:28
Like, we don't need to recreate it on our side. You can directly plug a model that you might train. Can be, you know, the latest Qwen 3.5 and you can directly train the model on an environment that already exists.
- 8:42
If such environment doesn't exist, it can still be built. You can still mock the tools and you can mock if you need one for this specific case, a mock user.
- 8:51
Like if you want to create a chatbot that has access to tools, a mock user might be, uh, an LLM. What about the reward? The reward will be just, you know, any business outcome, any KPIs, any LLM as a judge that might define what success looks like to you.
- 9:07
Like, was the agent helpful? Was the agent useful? Was the agent using, you know, a tone and a vocabulary that is following, uh, business guidelines?
- 9:17
On this topic, uh, two colleagues of mine, Laetitia and João, they recorded a workshop that you might find, you know, on our engineering website that will show you exactly how you can actually train a model by plugging into an existing, uh, environment.
- 9:33
When I talk to clients, one of the main, you know, sources of doubt on whether they will ever get to an MVP or to production is because they don't have the data to do it.
- 9:42
Data was already an issue before agents. After agent is even more of an issue because agents' training data doesn't exist in the wild. Like, there's no such a dataset you can scrape from the web where, you know, an agent is using tool.
- 9:55
Uh, you don't have such, uh, such a dataset. The nice thing is that when you train a model with reinforcement learning and you have an environment and you have a reward in place, basically you just build what is, uh,
- 10:11
as a by-product of your, of your environment, you created a synthetic dataset pipeline. Basically, as you have an environment, you can literally create trajectories that are good because the reward that you put in place will tell you what is good and what is not.
- 10:26
So basically, you can do rejection sampling and create a dataset that you can use to bootstrap, you know, the first training of a model. And the nice thing is that even though many company don't have the data, uh, exa-- the exact data that is required to train agents, they still have a lot of datasets that can be
- 10:43
leveraged to improve the entire experience in the environment. Such data might be, you know, real transcript between a customer and an agent that can be given to the mock user.
- 10:53
The mock user can even be trained on that to be the actual realistic person that might be annoying, that might ask things three times in a row. Uh, we work with, you know, like with, with customers like, uh, medical supply, where people might call, they might be in panic.
- 11:07
So the correct behavior might be, you know, "I will escalate you, uh, to a, uh, to a human agent," or, "We'll call nine one one for you." And that kind of, let's say, dirty real conversation is something that can be easily mocked by using proprietary, uh, datasets.
- 11:24
Where is the human in the loop? Um, RL became famous, uh, in the LLM world thanks to ChatGPT because OpenAI published a blog post where it was saying, "We did RLHF," so reinforcement learning from human feedback.
- 11:40
But sometimes the human in the loop, which is nice to hear, sometimes what actually hides behind are expensive annotation campaign. So in my experience, nobody wants to run a, an annotation campaign.
- 11:52
It is either ex-expensive or it is really useless because the reality is that people don't wanna do it. But you still want to keep a human in the loop.
- 12:00
So where does the human-in-the-loop come in the equation that I just showed you? When you train with RL, the most important thing you want to do is to build a reward signal.
- 12:11
A reward signal might come from different sources. Might be a systematic reward when it comes from, you know, does the code run? Is the syntax is correct? It can come from direct KPIs or bit- business outcomes.
- 12:24
One of our clients, CCS, the medical supply company I was mentioning before, as, as a customer support system that as any other customer support system, what is trying to maximize is containment rate.
- 12:36
Like how many calls are actually brought end-to-end by the model. And that reward, that percentage of calls that actually brought end-to-end is something you can directly maximize. Many other things like was the tone correct?
- 12:48
Were the business requirement followed? It's a bit of an open-ended question when it comes to systematic reward, but that issue can be solved with LLMs as judges. So the human-in-the-loop is helping just by defining the rubrics, defining the system prompt to these LLM as judges, and defining these scenarios, as in making sure that it's just aligned with
- 13:09
what they see. But this, let's say, activity that, that the human will do will take from few minutes to hours, but they will not take weeks, and you don't have to do it iteratively
- 13:21
dozens of times. For everything I mentioned before, RL not being just one algorithm, but the one algorithm that industrializes bringing model into production. In the last two years at Adaptive, we built Adaptive Engine that is an RL ops platform to evaluate, tune, and serve the best LLMs for your business.
- 13:44
The Adaptive Engine is an holistic platform where you can observe, train, and serve at once. When I mentioned at the very beginning that the goal is to accelerate the life cycles, that doesn't mean to accelerate training per se, right?
- 14:00
You also want to evaluate, be sure that the model is actually behaving, and you want a systematic way to find defect pre and post-production and to act accordingly. This is something you can do only if you have a systematic, holistic approach.
- 14:17
Our models are built on top of the best open source models. Any, let's say, m- most open source model you can think of that are available, like the latest Gemma, uh, Gemma Four that you heard a few days be- uh, ago.
- 14:30
The latest Minstrel, the latest Wen, they're all, you know, available in your company, and depending on, you know, the model of your preference, you can start building on, on top of it.
- 14:40
And finally, what, what's the catch with RL? The only catch with RL is that reinforcement learning is actually hard. Reinforcement learning is not as easy as changing a system prompt, and it's not as easy as just building a dataset for instruction fine-tuning.
- 14:54
Uh, one of the most famous RL algorithms, which is a PPO, uh, requires orchestrating not one but four large language models at the same time. That is where, you know, Adaptive Engine shines because we let you define the rubrics and the rest, but we take care, we take care of the complexity of reinforcement learning by exposing a
- 15:15
series of pre-built recipes for you. So you don't need to implement the latest algorithm, say, GSPO, and you don't need to build like, you know, the training recipe to run an actual training.
- 15:28
So once again, RL is the one algorithm that will let you bring model into production in a systematic and industrialized way, and all of that is possible with the Adaptive Engine.
- 15:41
Thank you very much for your attention. [audience clapping]
- 15:50
I have a question about, um, uh, some of the human feedback portion that can get incorporated into RL.
- 15:56
Yeah.
- 15:56
Uh, for example, um, last year Cursor had a blog post where they outlined how they take human feedback from production data-
- 16:05
Yeah
- 16:05
... such as whether or not the tab completion is accepted or not.
- 16:09
Yeah.
- 16:09
And they constantly use it to update the model.
- 16:12
Yeah.
- 16:12
Um, in settings like this where an LLM is at play, um, like in a more traditional LLM RL style, you would do like several rollouts per prompt and pick the ones that, uh, work and train on just these samples.
- 16:26
Yeah.
- 16:26
When it's human feedback and there's a single signal-- uh, a single signal, um, do you do, uh, like replays to, um, have many variations of outputs for that problem and train on it?
- 16:38
Or is it effective to just have a single, uh, implicit feedback from production train on as a reward signal?
- 16:45
So I would say, you know, like what, what you ask is how do we, you know, uh, leverage a human feedback? Uh, I would say there's two, let's say, scenarios.
- 16:55
There's a scenario where, you know, sometimes the human feedback is like, uh, it doesn't come from production, right? It comes from, you know, uh, like ten to twenty feedbacks.
- 17:04
I- in that scenario, what we do is that we basically usually use it to improve the LLM as judges, as in that is good, that is bad. Like how does it fit into the current description of, you know, of what you are trying to do?
- 17:18
And the nice thing is that as you go to production, then you will have thousands of such feedbacks. So what we do is that we usually, rather than using-- for LLM as judges, at the very beginning, we just use prompted a really big large language model, say, uh, when, uh, twen- two hundred and thirty-five B.
- 17:33
As we move to production, we have so much data that what we do is that we use this data to train reward models so that we can basically scale that, uh, human feedback in, uh, like to actually train actively the, the LLM.
- 17:46
Got it.
- 17:46
Uh, and then, you know, with respect to, you know, to the question when these feedbacks are not as explicit, but more like implicit, I think it really depends on the specific use case, and then we can build a reward model, uh, accordingly.
- 17:59
You know, because we already have the data, we can have do two different scenarios to see which kind of training actually gives you the best, uh, output, give, uh, the best performance given a certain evaluation.
- 18:10
Got it. Thank you.
- 18:11
You're welcome. Okay. And, um, thank you very much.
- 18:18
Thank you. [audience clapping] [outro music]