Everything I Learned Training Frontier Small Models

Talk transcript

1. 0:00

   [upbeat music] Hi everyone.

2. 0:15

   My name is Maxime Labonne. Uh, in this presentation, I want to talk about the lessons I've learned, uh, pre-training small models. Um, so for context, I work at Liquid AI as head of pre-training.

3. 0:26

   At Liquid, we mostly focus on edge models for on-device deployment. And as you can see here, we have models from three fifty million parameters to, uh, twenty-four billion parameters.

4. 0:38

   So this is very, very small. And, um, yesterday we released our new vLLM, uh, four, uh, fifty M and, uh, the week before we released the new version of the, uh, three fifty M model for text.

5. 0:51

   Um, so this is what we do. We work across text, uh, vision and audio and, uh, yeah, the models are available on Hugging Face if you want to try them out.

6. 1:01

   Um, in this presentation I want to talk about what separates small models and big models, and there are three main characteristics I want to talk about. So first of all, uh, the small models, they are memory-bound, um, because the hardware is, is what it is, right?

7. 1:17

   On a phone, in a car, et cetera. Uh, we can't really use super big models, uh, which is why we try to keep the, the size quite small. And because of that, we have low knowledge capacity compared to bigger models.

8. 1:29

   Uh, then the models are task specific, which is great because if you have a small knowledge capacity, you can at least focus on one thing very well. And so that means that they are usually not general purpose chatbots like ChatGPT.

9. 1:42

   They are a lot more narrow in terms of focus, and they can do something like summarization and tool use very, very well. Uh, so that's the second aspect. Uh, and the final one is that it's very latency sensitive, and that means that you need to have very, very fast, uh, throughput.

10. 2:00

    So all these characteristics are very important and we'll see in this presentation how they play with each other and how we can do better. Um, but the main lesson I want you to retain from this presentation is that small models are not just scaled-down versions of bigger models.

11. 2:14

    They also have their unique challenges, and we will see about how we do it, um, in this presentation. The first thing I want to talk about is the model architecture because there's a lot of interesting things that we can do here for edge models.

12. 2:28

    Um, I want to first talk about, uh, Gemma 3 two hundred seventy M and Qwen 3.5 zero point 8B. So these models are the smallest version of their respective family.

13. 2:38

    And you can see that both of them, they adopt a hybrid architecture. Uh, Gemma 3, uh, has sliding-window attention and GQA hybrid. Um, Qwen 3.5 has a new architecture with, uh, gated, um, DeltaNet and gated attention.

14. 2:53

    This is great because this is a lot faster. But what I'm interested in here is actually the embedding layer because if you look at, uh, the size of the embedding layer compared to, uh, all the parameters of the model, you see that actually Gemma 3 two hundred seventy M is mostly an embedding layer.

15. 3:08

    Uh, it's sixty-three percent of the total parameters. And even Qwen 3.5 zero point 8B it's, it's still like twenty-nine percent of the parameters. Um, so that's not super efficient because, uh, the effective parameters, the parameters that are really used for reasoning, for knowledge capacity and all that stuff, um, are not the embedding parameters.

16. 3:29

    Um, they are-- it's the rest. So the effective size is actually a lot smaller and it means that you could squeeze more reasoning and, uh, more performance from the same memory footprint.

17. 3:39

    Um, and the reason why they do that is because they use distillation, uh, to train the models. Um, so they distill these models like those are the, uh, student models and they have teacher models with a huge vocabulary, uh, sizes and this is why, um, we have these super big, um, embedding layers.

18. 3:59

    All right. Uh, let's talk about the LFM2 architecture now. As you can see, the LFM2 architecture is actually not that different, uh, in terms of, um, just layers. We also have like a hybrid architecture and this time we have short convolutions and, uh, GQA.

19. 4:16

    Um, and I want to talk a bit about-- Well, first you can see that, um, the embedding layer is actually a lot smaller compared to the others. It's like nineteen percent of the parameters.

20. 4:25

    So we have more effective, uh, parameters which is great. And I want to talk about how we created this architecture and we did on-device profiling. So instead of doing more like theoretical work, uh, we decided to say like, "Okay, let's try to really implement it on the target hardware."

21. 4:42

    So we had two target hardware here and, uh, we wanted to see how it performs in real life to be able to optimize, um, the architecture, find the right operators here.

22. 4:53

    And the thing that we found is this gated short convolution block that you can see here. And why this is nice, it's because it's very, very fast. Uh, the short conv are a lot faster than all the alternatives that you can see here, uh, compared to sliding window attention from Gemma 3, the gated DeltaNet from Qwen 3.5,

23. 5:13

    uh, gated linear attention and, uh, grouped-query attention. You can see that, uh, the cost ratio is really in favor of short conv which is great because we said that, um, this is very latency sensitive.

24. 5:23

    So this is exactly what we want. So this is quite theoretical, but if we look in practice, um, and we profile, uh, the inference of these models, you can see here on the two CPU, um, the AMD Ryzen, um, Max plus three nine five and the, uh, Samsung Galaxy S25 Ultra.

25. 5:42

    Um, all these models don't have necessarily the same size, but it gives you a rough picture and you can see that the short conv really allow LFM2 architecture to be a lot faster and also use, uh, less memory.

26. 5:54

    Um, so that's great. And you can see also GPU. It's not really just for CPU, but also on GPU you can see that it has a lot of, uh, throughput, uh, even at very high, uh, concurrency levels.

27. 6:07

    All right. Let's talk a bit about training now. So, [clears throat]

28. 6:12

    the LFM2.5 training recipe is quite similar to what you can five-- find, uh, elsewhere in terms of stages. We have pre and mid training on twenty-eight trillion tokens. We have, uh, supervised fine-tuning, preference alignment, and reinforcement learning.

29. 6:28

    And here you can see I'm talking about twenty-eight trillion tokens, and I said that we released a model of, um, three fifty M parameters last week. Um, so yeah, we pre-trained a three fifty million parameter model on twenty-eight trillion tokens.

30. 6:43

    If you're familiar with, uh, Chinchilla scaling laws, that might sound a bit weird because we're supposed to be compute optimal at like, I, I don't know, like maybe one billion, not, not even one, uh, trillion parameter, I mean.

31. 6:56

    Um, but it's actually not the case and we see that the performance still, um, grows when you scale the number of pre-training tokens. And there was a super interesting paper by Roberts et al., uh, published last week about the test time scaling laws.

32. 7:11

    And you can see here how LFM2.5-350M compares to their new scaling laws. You can see Chinchilla scaling laws here and the new ones that they proposed, uh, here. And actually, we did not pre-train the models on N of tokens.

33. 7:25

    We should pre-train even more, uh, to be, uh, optimal according to their laws. But this is cool because more pre-training works, and it works even at the smallest scale, which is great because these models are a lot cheaper to train than, uh, much bigger models.

34. 7:40

    And here you can see a comparison. It's not just for pre-training, it's like post-training, uh, models. And, um, you can see that the LFM2.5 model is, uh, significantly better than the, the previous version, LFM2-350M on a lot of different benchmarks.

35. 7:57

    So you have, uh, knowledge with GPQ Diamond, you have instruction following with IF bench, you have case report bench, which is data extraction, and also a lot of tool used with PFCL and, and Tao two bench.

36. 8:09

    Um, with this model, it's only three fifty million parameters, so what we wanted to do is we wanted the model to be very, very good at data extraction and at tool use.

37. 8:19

    And the rest, if it's not the best model at-- in code, it doesn't matter. Like, people don't use it that way anyway. Same for math. I think it's really nice to try to target some capabilities and not try to be like average on everything.

38. 8:31

    All right. Let's talk exactly about that. Post-training, um, small and big models, what the difference. Um, we have pretty much the same stage, so this is not really in terms of stages that we see a difference.

39. 8:43

    It's more about how you do it. So for supervised fine-tuning, it's better if you're actually quite narrow and you focus on some task. Um, it's true for general purpose post-training, but it's also true if you do fine-tuning.

40. 8:55

    So you can take one of these models on Hugging Face and just fine-tune it for your use case. And for example, you have a use case where you have a, a particular function that you want to call.

41. 9:04

    This is great. This is a excellent use case. Like, the more narrow you can, um, you can find it or design it, uh, the better it is. Then we have preference alignment.

42. 9:14

    So during post-training, we have our, um, own, uh, on-policy length normalize direct preference optimization algorithm that we quite like. And preference alignment is very nice because it brings you general improvements.

43. 9:27

    It's not just about benchmarks. It's really like overall, after preference alignment, the model is better, it sounds better, and this is really nice to be able to, um, just improve it overall.

44. 9:39

    And finally, we have reinforcement learning. And reinforcement learning is extremely efficient, even at very small scale. It's a really, really important technique that we use everywhere. And the main thing is that it's very narrow in terms of focus, so you want to have like as many environments, as many tasks as possible and make sure that, um, you

45. 9:59

    generalize well thanks to this. Um, and then for small models in particular, they're quite sensitive to cold start SFT data. So if you have a particular task in reinforcement learning, it's always good to have similar samples and a similar task in your supervised fine-tuning mixture.

46. 10:18

    And this is good feedback that you can see during reinforcement learning. Something doesn't train very well, it's probably because you are missing some, uh, cold start SFT data. Maybe the task is too complex.

47. 10:28

    There are different reasons. But, um, you can try to start again, uh, from the supervised fine-tuning, uh, stage, add your data, and then see if it improves anything.

48. 10:41

    All right. But there's a new problem, uh, with small language models that you might have encountered even with bigger ones, and this is Doom looping. So the problem with Doom looping is, as you can see here, it's going to start repeating a sequence of words over and over and over and over again, and it just like never

49. 10:57

    stops. Um, so this is a problem all the time, but it's particularly a problem if you have small models, if you have reasoning models, and if you have complex tasks.

50. 11:07

    If the task is basically too complex for the model. Hopefully, this recipe is not too complex for this model, but this can happen anywhere. And you have the three of them at the same time.

51. 11:17

    So if you have a tiny reasoning models on like super difficult math task, this is the perfect recipe to have a lot of Doom loops. Um, so this is a unique challenge that you find with small models, uh, to give you a concrete example.

52. 11:33

    And here I can talk about like how we solve it. The first thing is that we solve it during the preference alignment stage, and in particular for the data generation part that we do.

53. 11:45

    So here you can see the pipeline that we use, uh, to do the data generation, the on-policy data generation for preference alignment. So we start with prompts for like one million samples to give you a rough idea, and then we use the policy model, the model that we want to train, with temperature sampling, and we just generate

54. 12:04

    five rollouts. Because we use temperature sampling, these rollouts tend to be a lot more diverse, and we expect that not all of them will, uh, have Doom loops. At least one should not Doom loop, right?

55. 12:17

    And on the other hand, we generate just one extra, uh, rollout with a policy model with temperature zero. And this one we think that it's going to Doom loop And then we give everything to LLM jury to score all the, um, rollouts.

56. 12:33

    We pick the best one. The one with the highest score is the chosen answer, the one with the worst score as the rejected answer. And the idea is that if we have some Doom loop here, the response for the Doom loop will be rejected.

57. 12:46

    So we will train the model during preference alignment to not Doom loop. And this is quite effective. So this is solution number one. And then we have solution number two, and this one is about using reinforcement learning with verifiable rewards, and we add a bit of n-gram repetition penalty.

58. 13:05

    Uh, but you can see that with reinforcement learning with, um, verifiable rewards, it's a very nice way to actually, um, solve this issue. Because if you have a question, like a math question like this one,

59. 13:20

    you are going to try to extract the final answer. If you do not have a final answer, uh, you won't get a positive reward. So this is already being taken care of during, um, reinforcement learning with verifiable rewards.

60. 13:35

    But on top of that, you can add a bit of repetition penalty to make sure that, um, you are going to generate more, like less Doom loops in general.

61. 13:45

    And same thing, we also use temperature sampling here, so the rollouts are also quite diverse, and it just is less likely that you are going to get like a lot of Doom loops, uh, all the time.

62. 13:57

    So this is the, the second, uh, solution, and that allowed us to, uh, really reduce the Doom loop ratio. So this is a real example with, uh, LFM 2.5 1.2B thinking, which is a small model.

63. 14:11

    It's a reasoning model. And on top of that, we threw really hard task at it. So you can see that after mid-training, the Doom loop ratio that we calculated across like a lot of benchmarks was about fifteen percent, um, or even sixteen percent.

64. 14:27

    And then after SFT, it, it barely moves. Like SFT is not the right stage to fix this. Um, we didn't have Doom loop, uh, examples during the SFT stage, but it's not enough, uh, to get rid of this issue.

65. 14:40

    Um, after DPO, so that was a first solution, um, it really reduces quite a lot, and you can see that after reinforcement learning the problem is almost non-existent. If today you try to do the same thing with, um, Qwen 3.5-0.8B in reasoning mode, you will see a lot, a lot, a lot of Doom loops, like over fifty

66. 14:59

    percent of Doom loops. Just something that people complained about online, and that also shows that the Qwen 3.5, like this tiny model, is just a scaled-down version of bigger models.

67. 15:09

    And this is not the approach that we're taking here at Liquid. We want to say, "Okay," like the edge models, they are their own thing. And, um, this is also a way to just optimize the entire architecture, the entire pre-training stack to make sure that, uh, we treat them as, um, as best as possible.

68. 15:28

    And finally, I want to talk about, uh, next stage, next steps for, uh, all these, uh, small models with agency reinforcement learning. The final characteristic I didn't mention here is about, uh, being memory bound.

69. 15:40

    If you're memory bound, it means that you have low knowledge capacity. If you have low know- low knowledge capacity, it means that you're going to hallucinate a lot. But a nice way to solve this issue is just providing like web search tools to the model.

70. 15:53

    If you have a tiny model that is able to Google everything that you, um, throw at it in terms of like knowledge questions, you're going to have like much, much better performance, uh, than if you just rely on the, uh, base models.

71. 16:07

    And same thing with, um, a, a lot of problems that you can, uh, throw at the model. I think that from experience, these tiny models are actually very good at agentic task, and this is how we should use them.

72. 16:19

    Um, it doesn't matter if, um, they don't have the knowledge capacity of big models. What they truly need is really good reasoning capabilities to make sure that they are able to use these tools in a reliable manner.

73. 16:34

    And another point that I haven't mentioned here is that small models are also not very good at long context capabilities. But it's okay, because if you have like a recursive, um, language model environment, then you can use Python and, and like basically take a shortcut, uh, to, to solve this issue.

74. 16:52

    So most of the issues that you find with small language models can actually be fixed in different ways. It just requires more creativity. It just requires thinking about this problem, not like you would think about it, uh, from a bigger model perspective.

75. 17:06

    Uh, but everything about this is fixable. All right, so in conclusion, some takeaways. Um, I hope I convinced you that edge models have unique challenges, and they are actually interesting from scientific point of view and also production point of view.

76. 17:21

    Um, if you combine them with agentic tools, they tend to perform really, really well, and this is something that is currently underexplored. We talk about agentic workloads with really big models, but it's not necessarily, um, the best use case.

77. 17:35

    It's not necessarily the best fit all the time. And, uh, yeah, finally, we're working on LFM3, and we have like a, a ton of crazy experiments and, uh, ideas to try.

78. 17:45

    So, um, come work with us if you're interested, uh, in this space. Thank you everyone. [audience applauding]

79. 17:58

    Yes.

80. 17:59

    Um, can you share a bit about how you use these models in your workflow and how, how you make the decision about when to use a big model or a small model?

81. 18:07

    Yeah, this is a, a good question. So the question is like how we use this model in the workflows and, uh, how we decide between small models and big models.

82. 18:15

    So the, the main idea here is that you will try to use the small models when you don't have a internet connection, for example. So in-car deployment is a good example of that because y- you can't have like a reliable internet connection, so it makes sense.

83. 18:31

    Uh, latency is also a big one. If you have a workload that is very latency sensitive, uh, small models running locally are always going to be better. And another one is, is privacy.

84. 18:42

    If you use a regulated environment, um, if you work in finance or healthcare, this is also a good one.

85. 18:47

    Yes. Like in, in your specific workflow is the question. Like do you use-

86. 18:50

    It-- I make the models, so I... Like, I make the models for other people.

87. 18:56

    Okay.

88. 18:56

    But, uh, like in my workflows, like not necessarily. Yes.

89. 19:00

    Have you tried for Doom looping, have you tried like... You, you, you talked about how Qwen 3.5 is a scaled down version of a bigger one.

90. 19:07

    Yep.

91. 19:07

    Have you seen, uh, the work that you do with RL and all this, uh, re- reduction of Doom looping on a bigger model, uh, be able to be distilled into a smaller model without having to re- redo the steps?

92. 19:19

    It's a good question. I think we need to do some experiments to see like if, uh, just distilling from a bigger model translates well in terms of Doom looping.

93. 19:27

    I would say no. I, I don't think so because I think it would be too close to SFT. It depends like how you do this distillation. If you stop K and you have like enough, uh, K, maybe this is, uh, good enough.

94. 19:40

    Um, but I, I think that it would not completely be solved, and you would still need like several batches to make sure that it doesn't happen again.

95. 19:48

    Okay.

96. 19:48

    Right. Yeah. All right. I, I think I should quit, but, um, I, I'll be around if you, you have oth- other questions. Thank you very much.

97. 19:55

    Thank you. [audience applauding] [outro music]