Everything you need to know about Finetuning and Merging LLMs

[upbeat music] Hi, everyone.

Uh, in this session, we're gonna talk about, uh, everything you need to know about fine-tuning LLMs and model merging. Uh, quick intro, my name is Maxime Labonne. Uh, I'm a Staff Machine Learning Scientist at Liquid AI.

I'm also a Google Developer Expert. I write blog posts on these topics. I created the LLM Course, which is, uh, super popular on GitHub. Um, I also contributed to the open source community through models, uh, through tools, and I'm the author of, uh, Hands-On Graph Neural Networks using Python, uh, with Packt.

Um, so first of all, let's talk about fine-tuning. Uh, we saw a bit of fine-tuning in the previous session, so I'll try to not repeat too much. Uh, but basically, here's the LLM training lifecycle.

You see three stages. Uh, first of all, you have the pretraining stage, uh, where you give a lot of raw text to the model, and the idea is that the model learns to do, uh, next-token prediction.

The result of that is called a base model. This base model, uh, is really nice, uh, but if you ask it questions or instructions, it's going to autocomplete your question instead of answering it, which is why we have the supervised fine-tuning stage, where this time we give pairs of questions and answers to the model.

And, uh, we have a similar, uh, training objective, but the idea is that, uh, at the end of it, it's going to actually answer your questions and follow your instructions.

Then we have a third and final stage, the preference alignment stage, where we give, uh, human preferences to align the model to how we want it to behave, and the result is commonly referred to as chat model.

So when to use fine-tuning. Uh, here you can see a little flowchart that I've made. It's very high level. Uh, but basically, uh, there's a conversation about when to use prompt engineering, when to use fine-tuning.

I think it's good, in general, to start with prompt engineering if you can, and, uh, the idea is to have a really robust evaluation, uh, structure, where you have a lot of different metrics that you're interested in.

It can be, uh, the accuracy of the model. Uh, does it answer my question well? You can create a custom benchmark, uh, if you have a very niche use case, or you can reuse, um, open source benchmarks.

Also, in terms of cost, latency, because the question is, is it good enough? If it's good enough with just prompt engineering, then probably you don't need, uh, fine-tuning. The problem is solved.

Uh, congrats. Otherwise, the question is, can you make an instruction dataset? So can you create pairs of questions and answers, uh, to fine-tune the model? If it's not the case, it can be for multiple reasons, but it's probably a good sign that, uh, you need to rescope the project.

Uh, otherwise, fine-tuning is an option, and you can reuse the evaluation framework that you created, uh, to evaluate the model. Uh, so, so that was the technical answer, but you also have a non-technical answer to that.

Uh, here is a report from a16z, and the question is, why do enterprises care about open source? You can see that the, the two main items are actually control and customizability, and customizability is mostly about fine-tuning models.

So even if there's, like, arguments about the technical side and cost and agency, there's also, like, a strong argument for customizability and control over these models. So in terms of fine-tuning libraries, I think that you know about Unsloth now, uh, but I'm gonna talk about the other ones.

Uh, so TRL from, uh, Hugging Face, a great library built on top of Transformers, very easy to use. You have Axolotl, excellent library, uh, very versatile. Uh, you, you have a lot of, um, YAML config files.

And then you have LlamaFactory, uh, where you have a really good graphical user interface, uh, that is built in. So to talk a bit more about supervised fine-tuning, um, here you see an example of a sample that we, we give to the model.

So we have the instruction, which is both the system prompt and the user prompt, and the answer, which is the output. So in this, uh, in this case, the system prompt is used to steer the behavior of the model.

Uh, think like you're answering to a [REDACTED:age]. And the user actually gives the task, "Remove the spaces from the following sentence." We train the model usually, like, generally, on the output only, so we mask the rest.

It's used as context, and what we want to, to do is train the model to output the correct answer. Uh, most SFT datasets, I want you to say that use synthetic data, and that's perfectly fine.

Usually, it's generated with, uh, frontier models, and that's a, a great way of, uh, building high-quality datasets.

Then you have the preference alignments. I'm just gonna mention it here. Uh, there are a lot of different methods, PPO, DPO, KTO, IPO. Uh, in practice, Direct Preference Optimization is probably the most popular one.

Um, so here you see that you have a different format, uh, with an instruction, and you have a chosen answer, and a rejected answer. So the idea here is that you're going to show, like, a positive example, a negative example to the model.

And with DPO, the goal is to make sure that the model that you're currently training outputs higher probabilities, uh, for the chosen answers than the untrained version of the same model.

Not gonna delve, uh, too much into the details here, but this is the general idea and can be used to either censor the model. How to make a bomb, uh, the chosen answer would be, "As an AI assistant, I cannot tell you that."

Or it can also be used to, um, boost the performance of the model in general. Uh, how to create SFT datasets. So this is a very fundamental question, uh, in the post-training world.

Uh, and the, the main question is, okay, what's a good sample? Human evaluation is quite bad at, um, actually reviewing the samples, but what I like to define is, like, three main features.

The first one is the accuracy. We want the samples, the outputs to be, uh, factually correct. Um, maybe no typos would be good too. Uh, we don't want to compromise, uh, the knowledge of the model by giving it fake information.

Uh, then you have diversity. And diversity, you want to cover, um, as many topics as you can. Of course, it depends on your use case because if you do summarization, uh, you won't be as general as if you do, uh, general purpose fine-tuning.

Um, but it's always a good idea to include a lot of different, uh, topics, different writing styles, um, in this, uh, dataset. And finally, you have complexity. I think this one is, is a bit less trivial.

Uh, and it's about giving complex tasks to the model, forcing reasoning. So, for example, the output will have chain-of-thought reasoning because you want to train the model to have this kind of, of reasoning.

Or it can be task, like summarization, explain me like I'm a [REDACTED:age]. Uh, this kind of task really, um, force the model not to only answer the question like a QA with, um, answers you could find on Wikipedia.

It also forces it to reason over the, the prompt and, uh, give a more complex answer. So as a, a little recipe you can see here, um, I would recommend in general, uh, starting with open source datasets if you can.

Uh, combine some of them. Then you can apply different filters. The first one is data de-duplication. It can be either exact because you want to remove, uh, duplicates. It can be fuzzy, uh, so same, um, same idea.

And, uh, then you have data quality filters. Here you have different techniques. Can be rule-based filtering. Uh, for example, you want to remove every single row where you have, "As an AI assistant, I cannot..."

because people hate it. Uh, but you can also use, uh, more clever techniques like reward models or LLM-as-a-judge to evaluate the quality of each sample and filter out the bad samples.

And then you can use data exploration with different tools like Lilac, Atomic Atlas, text clustering to have, um, topic clustering to visualize your dataset, uh, to get ideas on how to improve it.

And with these ideas, you can, uh, go back to data generation and start the process all over again. In terms of L- SFT techniques, we have three main techniques.

Uh, full fine-tuning, this is like the most basic one. You take the base model and you just, uh, train it on the instruction, uh, dataset. It has the best performance, but is also, like, very inefficient in general.

A more efficient, uh, way of seeing it is LoRA. Uh, with LoRA, you are going to freeze all the pre-trained weights and you add adapters to each targeted layer.

Uh, these matrices A and B are these adapters. So you have, um... You don't train on all the parameters of the base model, you only train a subset of them.

Uh, so this is a, a lot faster, uh, but it can still be costly because you're still loading the entire model in, uh, six-bit, sixteen-bit precision here. So a more efficient way is to quantize, uh, the pre-trained model here in four-bit precision.

This is QLoRA, and, uh, you apply the same idea that you had with LoRA, but this time, uh, the, the weights are heavily quantized, so you have a lower VRAM usage.

But the problem is that it also degrades performance, so there's a trade-off here. Um, I want to briefly mention some hyperparameters, but Daniel al- already, uh, talked about a lot of them, so I, I'm gonna be brief.

I think the most important one is the learning rate. Uh, the learning rate is model-dependent. Uh, it requires a few experiments to be able to really tweak it and find the, the best one.

Generally, I would recommend to go as high as you can, uh, until your loss explodes, like in this graph. Uh, then you can, uh, reduce the size of the learning rate.

Uh, other super important hyperparameters, number of epochs. Um, I would say that depending on the size of your dataset, you can have, like, more or less epochs. Sequence length is also good, uh, because, um, it, it's a trade-off with the batch size because the longer, uh, sequence length you have...

So the, the bigger the context window, the more VRAM, uh, you're gonna use. Uh, but you don't need to use, uh, a sequence length that's as big as the pre-trained model.

Then you have the batch size. You want to maximize it to maximize the utilization of your GPUs. Um, and then you have, uh, the LoRA, uh, with the rank.

Um, this is, like, quite easy to, to, to fine-tune, so I don't want to go into the details here. Um, let's talk about model merging now. So model merging is the idea that you can take the weights of different fine-tuned models, and you can combine them together.

So you, you just can leverage, uh, what the open source community, uh, has, uh, produced on the Hugging Face, uh, Hub, for example.

Uh, it doesn't require any GPU, so it's super efficient, and it provides excellent results. Uh, so the Open LLM Leaderboard was updated this morning, uh, so we have a version two now.

But this is the version one. Uh, I haven't had time to update it. Uh, but you can see that for 7B, uh, parameter models, uh, the entire top eight or top 10 is just merged models.

Uh, so it really shows that, uh, this approach is extremely effective at producing high-quality models. Uh, and you can find similar results on, like, a really a lot of different datasets.

I would recommend using MergeKit. Uh, this is like the leading library in this, uh, in this space, uh, with a lot of different techniques that are implemented there. Um, so here you can see the family tree of, uh, merged models.

Uh, so in... You don't really need to see the, the name of the model, but you see that every node is actually a model. And we actually merge different merges together until it becomes like a giant family tree.

This one is actually quite small. Like, it can get a lot crazier than that, but it didn't fit in, on one slide, so I, I chose this one instead.

Um, about the merge techniques themselves, I want to mention- Like a few of them. The first one is called SLERP. It stands for spherical linear interpolation. So the idea is really to apply spherical but, uh, linear interpolation, uh, with the weights of different models.

You can only merge two models at the same time with this technique, uh, but you can really tweak it, uh, with different, uh, interpolation factors for different layers. Uh, here's a model that I've made, you know, Beagle, uh, 14-70 B, uh, which was a really, um, efficient way of, uh, leveraging, uh, the, uh, different models that were

created by the open source community. And then you have DARE. Uh, so in DARE you want to reduce the redundancy of the model parameters. Uh, to do that you're gonna use pruning.

You're gonna select the most significant parameters in your model weights, and you're gonna rescale, um, the weights of these, uh, source models. Uh, the advantage that it has is that you can merge different models, not just two, but even more together.

And, uh, I would advise, uh, I would recommend this technique, and not with just two model, not with three, but like, with seven or eight models. It works really, really well, uh, so I strongly recommend that.

Uh, then you have a very funny, um, uh, technique called passthrough. And in passthrough you can concanate, concatenate, uh, layers from different LLMs. Um, it can also be the same one.

We call it self-merge. Um, and so here you have an example that I've made, uh, recently. It's called Meta Llama 3 120B Instruct because I took Llama 3 70B Instruct and I just, uh, repeated 10 layers 6 times.

Uh, so you could say, like, this shouldn't work at all. Like, come on, uh, you, you haven't even trained the model. This is ridiculous. Uh, but actually, yeah, this is ridiculous.

Uh, people loved it on Twitter and Reddit and online in general. Uh, so it, it shows that, uh, there's a lot of things that we can still discover, uh, with these m- merge techniques, with these models.

Uh, they do not, um... They can be counterintuitive sometimes, and you can see that this model in particular was particularly good at, uh, creative writing. Uh, it was also quite unhinged in general, but, uh, really good at creative writing and, uh, now it's being used by a lot of people, uh, even though it's, it's super big.

But no kind of fine-tuning at all?

No, no fine-tuning. Nothing. Um, and then I want to mention the last techniques, uh, which is called mixture of experts. Uh, so in traditional mixture of experts you are going to pre-train a model, uh, with a router you can see on the bottom here, and different feedforward network layers.

And you pre-training, pre-train it from scratch. But you can do something quite smart, uh, with merging, where you extract the feedforward network layers from different fine-tune models and you combine them together, uh, like this.

So we call this a FrankenMoE. You add a router. You combine the FFN layers from different, uh, models, and this is how you create, like, your kind of, uh, mixture of experts.

It's actually, uh, pretty cool. It, it works pretty well in practice. Uh, you can see on the left a MergeKit config, uh, for the Beyonder, uh, model. Uh, so for this model I selected four different, uh, fine-tune models.

One, uh, as a chat model, one as a code model, one as a role play model, and one as a math model. You can see that I'm using positive prompts here.

So actually i- it's, it's a way to initialize the router, because if you go back to the previous, uh, slide, we can see that the router is supposed to, uh, select for each token and each layer where, like, which feedforward network, uh, layer is going to be used.

We used to, uh, in general. Um, and so how do we initialize it if we do not fine-tune it? Once again, we don't want to fine-tune it. We can, but we, we don't necessarily want to.

In this case, we're just going to use these positive prompts, uh, calculate the embeddings, and use these embeddings to initialize, uh, the routers, and that works really, really well.

So those are two models that I've used. For Fixtrol I had to modify it to make it compatible with Phi-2, and, uh, that outperformed the base model, uh, on a lot of task.

Um, so it's really a good technique to, to use in general. But I would say that, um, if you compare it to merging, uh, as we saw with SLERP and with DARE, I would say that, um, if you, uh, want to increase the performance, it's better to use, uh, SLERP and DARE instead of mixture of experts because

this is a bit more experimental. Uh, this doesn't... This will not bring you the same level of performance. Um, and here you can see the results of, uh, the Beyonder, uh, model.

Uh, you can see that the other models I'm comparing to are the source models that I've used in this, in this merge. Uh, so it, it's quite remarkable to see that it's actually performing better than the source models, um, on the, on a lot of different benchmarks.

Um, so yep, that's it for me. Uh, thank you for your attention. Uh, if you are interested in, uh, knowing more, if you want notebooks, uh, to, to run some code, I created the Large Language Model Course.

All these notebooks are available on GitHub, uh, llm-course. And, uh, yeah, thank you. [audience applauds] [upbeat music]