Scaling Reinforcement Learning: Lessons from Trillion-Token Deployments at Fortune 500s

Talk transcript

1. 0:00

   [on-hold electronic music] Hello, everyone.

2. 0:15

   Uh, I am Alessandro Cappelli. I'm co-founder and chief customer officer at AdaptiveML.

3. 0:23

   At AdaptiveML, we build an RL ops platform, as in reinforcement learning operation, that allows large enterprises like AT&T, Manulife, CCS to build, evaluate, and serve in production their own specialized large, large language models.

4. 0:45

   Um, I'm here to show you how reinforcement learning, RL, is not just any other algorithm for post-training, but is an, an algorithm that at its core will bring models to production.

5. 1:01

   Around three years ago, uh, I was part of a team that trained Falcon. Falcon, around three years ago, was one of the most widely adopted, uh, open source models.

6. 1:13

   And we realized with, you know, my team that is actually the core founding team of AdaptiveML, that the gap that was missing between bringing an open source model to production versus proprietary models like of frontier labs like OpenAI was actually reinforcement learning.

7. 1:34

   Ninety-five percent of GenAI pilots fail to reach production. Why that is the case? Uh, I believe it's what we call the myth of the last mile. Here you see, let's say, a time description of what does it get to get to production, which, uh, I believe i-it's false.

8. 1:53

   And this idea is that the hard part is to get to an MVP is to, you know, come up with a demo that looks nice in front of stakeholders, in front of your colleagues, and that is the, the hard part.

9. 2:04

   And then, you know, there will be just the last mile, where the last mile will actually get the model into production. The issue is that most MVP are built on top of proprietary models or are built on top of open source model using instruction fine-tuning.

10. 2:18

    Both of these solutions won't let you systematically improve your solution. They don't integrate in a really systematic and mathematical way what are the defects you might find, uh, in the journey to get to production.

11. 2:32

    I will give you an example. If you test, let's say, your solution and there will be some defects and you're using a proprietary models, all you can do is to change the system prompt.

12. 2:43

    Now, you change the system prompt in one direction, you might have other defects, and there is no really like a really scientific systematic way to improve that system prompt nicely in a way that you can easily monitor.

13. 2:55

    Likewise, for instruction fine-tuning. Best you can do is to iterate over the dataset. That might be expensive. And what after production? Will you keep creating a new dataset every single week?

14. 3:07

    This is what I believe is a more realistic view of what getting to production and beyond actually looks like. Getting to an MVP is not easy, but it's just the first mile.

15. 3:20

    What actually the real journey, the real marathon is to get from an MVP to production and beyond. And the secret to do that is to accelerate model lifecycle. Is to be able to integrate every single feedback you can get from a variety of sources to keep improving your solution.

16. 3:39

    This continuous retraining, refinement, and improvement driven by real client feedback, business metrics, and environmental reward is unlocked in a systematic way only by reinforcement learning.

17. 3:52

    Reinforcement learning, as I mentioned before, my entire point is that by almost by design, by nature, allows to integrate feedbacks in a almost a mathematical way. But reinforcement learning is not just like-- not just that.

18. 4:06

    Compared to other post-training techniques or steering behavior techniques like, you know, prompting and instruction fine-tuning, they all, let's say, reach the same goal, which is steer a model's behavior,

19. 4:20

    but they're not equally effective. Reinforcement learning is disproportionally more effective than instruction fine-tuning and likewise versus, versus, uh, prompting. Reinforcement learning unlocks, let's say, outsized performance. So what does it mean?

20. 4:35

    What you can see in the plot that, you know, you can get the same performance with a RL with respect to SFT with a much smaller model. What it unlocks that actually help you to get to production.

21. 4:48

    RL enabled scale at adoption. What do I mean by that? As you train smaller model, those models will be cheaper to serve at scale, and the tokenomics of your use case will eventually make sense.

22. 5:02

    When you are a big enterprise like AT&T, any use case, any feature that you want to be a commodity, either for internal employees or, you know, any, you know, facing customer features you might think at scale will cost you millions of dollars.

23. 5:18

    As an example, AT&T, they summarize every single transcript that might happen between, you know, a customer and an agent. Just summarizing that costs them millions of dollars.

24. 5:31

    If you can train a model that is much smaller than, you know, uh, you know, a, a ChatGPT or a Sonnet, you will save money. Another things you unlock is that smaller model will be faster.

25. 5:44

    Now, not all use cases require speed, but many of them, they have

26. 5:49

    a threshold of latency That is not just something nice to have, it's a constraint that will prevent you to get into production. Let's say you have a model for customer support that is powering, you know, a speech-to-speech system.

27. 6:04

    You can go above half of a second. And I would say half of a second is already weird. When you're talking to someone and you g-- and you have to wait half of a second, that's already weird.

28. 6:12

    Ideally, it should be a third of a second, and a third of a second is something you will never get if you're re-- using large language models. You need to use small models.

29. 6:20

    Might be, you know, the latest Gemma, the latest Mistral, the latest Qwen of, you know, in that ten B family, but you can use m-much larger models. The last thing you unlock is ownership.

30. 6:31

    You will own the data that you give to the model 'cause the model will be trained on your own business data, and you will own the solution. So you don't need to worry about, you know, the latest update of the model that may shift performance underneath your feet.

31. 6:48

    Everything I've said so far, it is true for any use case you might think. I've mentioned summarization. Could have been classification, uh, could have been OCR, could have been anything you can think of.

32. 7:01

    And reinforcement learning is already the better choice. But now we are in the era of agents, and agents actually make everything more complicated. Agents require more tokens, more complexity.

33. 7:13

    There is less room for errors because now agents will have access to the data, will change things in the database connected to, you know, either, you know, internal employees or clients you, you might have.

34. 7:25

    So all of that raises the standard of what can be brought into production, and it raises further question on whether the tokenomics of an agent actually makes sense or not.

35. 7:36

    I mentioned before, you know, just for a summarization use case, you might spend millions. Imagine if you scale an agent to actually ten X the number of tokens.

36. 7:48

    RL advantage that already existed only widens when it comes to, you know, training agents. 'Cause RL at its core was actually made to train robots, you know, to, to train agents, uh, to, to leave an environment.

37. 8:02

    And environments are where, you know, agents actually behave. So RL naturally fits in a narrative where you want to train a model to be, uh, a good agent. Now, there are two scenarios.

38. 8:15

    Uh, either you already have an agent in place. Uh, as an example, you know, like, um, we work for, uh, uh, Manulife, and Manulife already have agents, so they already have a NACA workflow that has been settled.

39. 8:28

    Like, we don't need to recreate it on our side. You can directly plug a model that you might train. Can be, you know, the latest Qwen 3.5 and you can directly train the model on an environment that already exists.

40. 8:42

    If such environment doesn't exist, it can still be built. You can still mock the tools and you can mock if you need one for this specific case, a mock user.

41. 8:51

    Like if you want to create a chatbot that has access to tools, a mock user might be, uh, an LLM. What about the reward? The reward will be just, you know, any business outcome, any KPIs, any LLM as a judge that might define what success looks like to you.

42. 9:07

    Like, was the agent helpful? Was the agent useful? Was the agent using, you know, a tone and a vocabulary that is following, uh, business guidelines?

43. 9:17

    On this topic, uh, two colleagues of mine, Laetitia and João, they recorded a workshop that you might find, you know, on our engineering website that will show you exactly how you can actually train a model by plugging into an existing, uh, environment.

44. 9:33

    When I talk to clients, one of the main, you know, sources of doubt on whether they will ever get to an MVP or to production is because they don't have the data to do it.

45. 9:42

    Data was already an issue before agents. After agent is even more of an issue because agents' training data doesn't exist in the wild. Like, there's no such a dataset you can scrape from the web where, you know, an agent is using tool.

46. 9:55

    Uh, you don't have such, uh, such a dataset. The nice thing is that when you train a model with reinforcement learning and you have an environment and you have a reward in place, basically you just build what is, uh,

47. 10:11

    as a by-product of your, of your environment, you created a synthetic dataset pipeline. Basically, as you have an environment, you can literally create trajectories that are good because the reward that you put in place will tell you what is good and what is not.

48. 10:26

    So basically, you can do rejection sampling and create a dataset that you can use to bootstrap, you know, the first training of a model. And the nice thing is that even though many company don't have the data, uh, exa-- the exact data that is required to train agents, they still have a lot of datasets that can be

49. 10:43

    leveraged to improve the entire experience in the environment. Such data might be, you know, real transcript between a customer and an agent that can be given to the mock user.

50. 10:53

    The mock user can even be trained on that to be the actual realistic person that might be annoying, that might ask things three times in a row. Uh, we work with, you know, like with, with customers like, uh, medical supply, where people might call, they might be in panic.

51. 11:07

    So the correct behavior might be, you know, "I will escalate you, uh, to a, uh, to a human agent," or, "We'll call nine one one for you." And that kind of, let's say, dirty real conversation is something that can be easily mocked by using proprietary, uh, datasets.

52. 11:24

    Where is the human in the loop? Um, RL became famous, uh, in the LLM world thanks to ChatGPT because OpenAI published a blog post where it was saying, "We did RLHF," so reinforcement learning from human feedback.

53. 11:40

    But sometimes the human in the loop, which is nice to hear, sometimes what actually hides behind are expensive annotation campaign. So in my experience, nobody wants to run a, an annotation campaign.

54. 11:52

    It is either ex-expensive or it is really useless because the reality is that people don't wanna do it. But you still want to keep a human in the loop.

55. 12:00

    So where does the human-in-the-loop come in the equation that I just showed you? When you train with RL, the most important thing you want to do is to build a reward signal.

56. 12:11

    A reward signal might come from different sources. Might be a systematic reward when it comes from, you know, does the code run? Is the syntax is correct? It can come from direct KPIs or bit- business outcomes.

57. 12:24

    One of our clients, CCS, the medical supply company I was mentioning before, as, as a customer support system that as any other customer support system, what is trying to maximize is containment rate.

58. 12:36

    Like how many calls are actually brought end-to-end by the model. And that reward, that percentage of calls that actually brought end-to-end is something you can directly maximize. Many other things like was the tone correct?

59. 12:48

    Were the business requirement followed? It's a bit of an open-ended question when it comes to systematic reward, but that issue can be solved with LLMs as judges. So the human-in-the-loop is helping just by defining the rubrics, defining the system prompt to these LLM as judges, and defining these scenarios, as in making sure that it's just aligned with

60. 13:09

    what they see. But this, let's say, activity that, that the human will do will take from few minutes to hours, but they will not take weeks, and you don't have to do it iteratively

61. 13:21

    dozens of times. For everything I mentioned before, RL not being just one algorithm, but the one algorithm that industrializes bringing model into production. In the last two years at Adaptive, we built Adaptive Engine that is an RL ops platform to evaluate, tune, and serve the best LLMs for your business.

62. 13:44

    The Adaptive Engine is an holistic platform where you can observe, train, and serve at once. When I mentioned at the very beginning that the goal is to accelerate the life cycles, that doesn't mean to accelerate training per se, right?

63. 14:00

    You also want to evaluate, be sure that the model is actually behaving, and you want a systematic way to find defect pre and post-production and to act accordingly. This is something you can do only if you have a systematic, holistic approach.

64. 14:17

    Our models are built on top of the best open source models. Any, let's say, m- most open source model you can think of that are available, like the latest Gemma, uh, Gemma Four that you heard a few days be- uh, ago.

65. 14:30

    The latest Minstrel, the latest Wen, they're all, you know, available in your company, and depending on, you know, the model of your preference, you can start building on, on top of it.

66. 14:40

    And finally, what, what's the catch with RL? The only catch with RL is that reinforcement learning is actually hard. Reinforcement learning is not as easy as changing a system prompt, and it's not as easy as just building a dataset for instruction fine-tuning.

67. 14:54

    Uh, one of the most famous RL algorithms, which is a PPO, uh, requires orchestrating not one but four large language models at the same time. That is where, you know, Adaptive Engine shines because we let you define the rubrics and the rest, but we take care, we take care of the complexity of reinforcement learning by exposing a

68. 15:15

    series of pre-built recipes for you. So you don't need to implement the latest algorithm, say, GSPO, and you don't need to build like, you know, the training recipe to run an actual training.

69. 15:28

    So once again, RL is the one algorithm that will let you bring model into production in a systematic and industrialized way, and all of that is possible with the Adaptive Engine.

70. 15:41

    Thank you very much for your attention. [audience clapping]

71. 15:50

    I have a question about, um, uh, some of the human feedback portion that can get incorporated into RL.

72. 15:56

    Yeah.

73. 15:56

    Uh, for example, um, last year Cursor had a blog post where they outlined how they take human feedback from production data-

74. 16:05

    Yeah

75. 16:05

    ... such as whether or not the tab completion is accepted or not.

76. 16:09

    Yeah.

77. 16:09

    And they constantly use it to update the model.

78. 16:12

    Yeah.

79. 16:12

    Um, in settings like this where an LLM is at play, um, like in a more traditional LLM RL style, you would do like several rollouts per prompt and pick the ones that, uh, work and train on just these samples.

80. 16:26

    Yeah.

81. 16:26

    When it's human feedback and there's a single signal-- uh, a single signal, um, do you do, uh, like replays to, um, have many variations of outputs for that problem and train on it?

82. 16:38

    Or is it effective to just have a single, uh, implicit feedback from production train on as a reward signal?

83. 16:45

    So I would say, you know, like what, what you ask is how do we, you know, uh, leverage a human feedback? Uh, I would say there's two, let's say, scenarios.

84. 16:55

    There's a scenario where, you know, sometimes the human feedback is like, uh, it doesn't come from production, right? It comes from, you know, uh, like ten to twenty feedbacks.

85. 17:04

    I- in that scenario, what we do is that we basically usually use it to improve the LLM as judges, as in that is good, that is bad. Like how does it fit into the current description of, you know, of what you are trying to do?

86. 17:18

    And the nice thing is that as you go to production, then you will have thousands of such feedbacks. So what we do is that we usually, rather than using-- for LLM as judges, at the very beginning, we just use prompted a really big large language model, say, uh, when, uh, twen- two hundred and thirty-five B.

87. 17:33

    As we move to production, we have so much data that what we do is that we use this data to train reward models so that we can basically scale that, uh, human feedback in, uh, like to actually train actively the, the LLM.

88. 17:46

    Got it.

89. 17:46

    Uh, and then, you know, with respect to, you know, to the question when these feedbacks are not as explicit, but more like implicit, I think it really depends on the specific use case, and then we can build a reward model, uh, accordingly.

90. 17:59

    You know, because we already have the data, we can have do two different scenarios to see which kind of training actually gives you the best, uh, output, give, uh, the best performance given a certain evaluation.

91. 18:10

    Got it. Thank you.

92. 18:11

    You're welcome. Okay. And, um, thank you very much.

93. 18:18

    Thank you. [audience clapping] [outro music]