# Everything you need to know about Finetuning and Merging LLMs

Maxime Labonne, Liquid AI

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/uLrOI65XbDw-everything-you-need-know-about-finetuning-merging) · [Video](https://www.youtube.com/watch?v=uLrOI65XbDw)

## Everything you need to know about Finetuning and Merging LLMs

A base model can continue a question without answering it. Fine-tuning changes that behavior; model merging offers another way to reuse capabilities already learned by other models.

Familiarity with language-model tokens, training loss and GPU memory will help; no prior fine-tuning or model-merging experience is required.

### From completing questions to answering them

[1:00](https://www.youtube.com/watch?v=uLrOI65XbDw&t=60s)

Why might a language model autocomplete your question instead of answering it? Pretraining teaches **next-token prediction**: given large amounts of raw text, the model learns to continue that text. The resulting base model has learned useful patterns, but an instruction is still text to continue. Supervised fine-tuning, or **SFT**, supplies question–answer pairs so that a similar prediction objective teaches the model to respond to instructions.

Preference alignment adds another stage. Instead of supplying only examples of answers, it supplies preferences about how the model should behave. In the lifecycle Maxime Labonne presents, raw-text pretraining produces a base model, SFT produces an instruction-following model, and preference alignment produces what is commonly called a chat model.

![Three connected stages map raw text to a base model, instructions and answers to an instruct model, and AI or human preferences to a chat model.](https://ai.engineer/api/talk-essay-media/fdce066b4cca0929a4b2771a23c35c3fecce7d2465873571acc23424068f7220)

The LLM training lifecycle: pre-training, supervised fine-tuning, and preference alignment.

### Establish whether prompting is enough

[1:54](https://www.youtube.com/watch?v=uLrOI65XbDw&t=114s)

**Fine-tuning starts with an evaluation problem.** Begin with prompt engineering and establish what success means. Accuracy may require a custom benchmark for a niche application, or an existing open benchmark where one fits. Cost and latency belong in the same evaluation: a correct answer that is too expensive or too slow may still miss the requirement. If prompting meets those requirements, the problem is already solved.

If prompting falls short, ask whether you can create an instruction dataset: pairs of questions and desired answers. If you cannot, Labonne treats that as a reason to reconsider the project’s scope. If you can, fine-tuning becomes an option, and the evaluation framework you already built lets you test whether it helped.

The motivation is not exclusively technical. Labonne cites [a16z’s 2024 enterprise report](https://a16z.com/generative-ai-enterprise-2024/), whose interviews and survey emphasize control and customization as reasons to use open models. Fine-tuning is one way to exercise that customization. Those findings describe the enterprises surveyed, rather than a universal ordering of priorities.

### Choose a training interface

[3:22](https://www.youtube.com/watch?v=uLrOI65XbDw&t=202s)

After acknowledging Unsloth, covered in the preceding session, Labonne distinguishes three other training libraries by how you work with them.

| Library | Interface emphasized in the talk |
| --- | --- |
| TRL | Hugging Face library built on Transformers |
| Axolotl | Versatile training through YAML configurations |
| LLaMA-Factory | Built-in graphical interface |

These offer different entry points into training: a library integrated with the Transformers ecosystem, a configuration-driven workflow, or a graphical interface.

![Four library logos and descriptions highlight Transformers integration, reusable YAML configurations, a graphical interface, and efficient single-GPU fine-tuning.](https://ai.engineer/api/talk-essay-media/b552c40958aaaf9d06ee16e72120eeb800bf0fe9b77ef7ef04d15e6b3c19a329)

Fine-tuning libraries: TRL, Axolotl, LLaMA-Factory, and Unsloth.

### Separate context, answers and preferences

[4:06](https://www.youtube.com/watch?v=uLrOI65XbDw&t=246s)

An SFT example separates the instruction from the answer. The system prompt steers behavior—for example, by asking for an explanation suitable for a five-year-old—while the user prompt supplies the task. Labonne’s task example asks the model to remove spaces from a sentence. In the training setup he describes, the system and user tokens remain available as context, but are **masked out of the loss**. The answer tokens receive the supervision. Masking therefore does not mean hiding the instruction from the model; it means not training the model to reproduce that part of the example.

Those answers need not all be written by people. Labonne describes synthetic data as common in SFT datasets and endorses generation with frontier models as a way to build high-quality examples. The generated answer is still the training target, so its quality matters just as much as a human-written answer’s quality.

Preference training changes the example format. Labonne names PPO, [Direct Preference Optimization](https://arxiv.org/abs/2305.18290), KTO and IPO, describing DPO as the most popular in practice at the time. A preference example contains an instruction, a **chosen answer** and a **rejected answer**. The pair gives the model a positive and a negative example for the same request.

Labonne’s high-level intuition for DPO is to increase the probability of chosen answers relative to a reference version of the model that has not undergone this preference-training step. That is an intuition rather than a complete specification of the objective. His behavioral example prefers a refusal to a request for bomb-making instructions; preference training can also target better general performance, rather than refusal behavior alone.

### Build data that teaches the intended behavior

[5:47](https://www.youtube.com/watch?v=uLrOI65XbDw&t=347s)

What makes an instruction example useful? Labonne questions the reliability of human review alone and proposes three dimensions for judging samples:

- **Accuracy:** Answers should be factually correct and cleanly written. Training on false information risks damaging knowledge the model already has.
- **Diversity:** Cover the topics and writing styles relevant to the application. A specialized summarizer does not need the breadth of a general-purpose assistant, but it still benefits from variety within its scope.
- **Complexity:** Include tasks that demand more than retrieving a fact. Labonne points to reasoning-bearing answers, summarization and explanations for a five-year-old as examples that require the model to work with the prompt and transform information.

The preparation process is iterative:

1. **Start with existing open datasets** where possible, combining sources that fit the intended application.
2. **Deduplicate** with exact matching and fuzzy matching to remove repeated or near-repeated material.
3. **Filter for quality.** Rules can remove unwanted response patterns; reward models or an LLM judge can score samples for filtering. Labonne’s rule-based example removes formulaic refusals, which is a choice about desired behavior rather than a universal quality rule.
4. **Inspect coverage.** Lilac, Nomic Atlas and topic clustering can reveal how the dataset is distributed and suggest improvements.
5. **Generate additional data** from those findings, then repeat the process.

Inspection feeds the next round of generation instead of serving only as a final check.

### Decide which weights to train

[8:19](https://www.youtube.com/watch?v=uLrOI65XbDw&t=499s)

Full fine-tuning takes the model and trains its weights on the instruction dataset. Labonne presents it as the strongest-performing but least efficient option in his comparison. **LoRA** reduces the trainable portion: it freezes the pretrained weights and adds trainable adapter matrices, `A` and `B`, at targeted layers. The adapters learn the update while the base weights remain fixed.

| Method | Trainable parameters | Base-weight storage described |
| --- | --- | --- |
| Full fine-tuning | Model weights | Unquantized model |
| LoRA | Adapter matrices `A` and `B` | Frozen 16-bit weights |
| QLoRA | Adapter matrices | Frozen four-bit weights |

LoRA still requires loading the base model. [QLoRA](https://arxiv.org/abs/2305.14314) reduces that memory burden by quantizing the frozen base weights while retaining trainable adapters. Labonne warns about a performance trade-off, but degradation is not inevitable: the original QLoRA paper reports preserving 16-bit fine-tuning performance in its experiments. Neither that result nor the talk’s ordering guarantees the outcome for a particular task; the evaluation framework must settle it.

### Find a stable learning rate and a workable memory budget

[9:30](https://www.youtube.com/watch?v=uLrOI65XbDw&t=570s)

**Learning rate** is Labonne’s first tuning priority. It depends on the model and requires experiments. His suggested exploration is to try higher rates, observe where the loss becomes unstable, and reduce the rate from there. The exploding-loss graph illustrates a boundary to avoid, not a desirable training outcome.

The remaining choices interact:

- **Epoch count:** Choose how many passes to make with the size of the dataset in mind.
- **Sequence length:** Longer sequences consume more VRAM and constrain the batch size. Training does not have to use the pretrained model’s full context window.
- **Batch size:** Within memory limits, larger batches can improve GPU utilization.
- **LoRA rank:** This controls another part of the adapter setup, although Labonne names it without developing a detailed tuning procedure.

Sequence length and batch size therefore need to be chosen together: allocating more memory to each sequence leaves less room for sequences in a batch.

![A Hyperparameters slide lists dataset-dependent epoch counts, sequence length and batch size trade-offs in VRAM usage, gradient accumulation, and LoRA ranks of 16–64.](https://ai.engineer/api/talk-essay-media/5d6d555a2017984eec7b1608b803fa6f85fa7d865ee4b9021e2ae495a0766a47)

Hyperparameter guidance for epochs, sequence length, batch size, and LoRA rank.

### Reuse training already embodied in model weights

[10:26](https://www.youtube.com/watch?v=uLrOI65XbDw&t=626s)

**Model merging combines weights from existing fine-tuned models.** Instead of creating every capability through your own training run, you can draw on models released by the open-source community, including those on Hugging Face Hub. The merging operation can run without a GPU; that does not eliminate the compute needed to run the resulting model.

Labonne reports that merged models occupy roughly the top eight to ten positions among 7B models in the Open LLM Leaderboard v1 snapshot shown in the talk. This is historical evidence: he says version two had launched that morning, while his slide still showed version one. The displayed ranking is a motivation to investigate merging, not a claim about current leaderboard standings.

His recommended tool is [MergeKit](https://github.com/arcee-ai/mergekit), which implements multiple merging techniques. A source model can itself be a merge, so the ancestry quickly branches: the family-tree illustration connects models that were combined into other models, which were then merged again. Even the tree that fits on the slide is only a small example of that reuse.

### Interpolate two models or combine sparse updates

[12:01](https://www.youtube.com/watch?v=uLrOI65XbDw&t=721s)

**SLERP**, spherical linear interpolation, interpolates between two models’ weights. It merges two models at a time, but the interpolation factor can vary by layer, allowing different parts of the result to favor different sources. Labonne’s example is [**NeuralBeagle14-7B**](https://huggingface.co/mlabonne/NeuralBeagle14-7B). Its ancestry matters: the model card identifies NeuralBeagle as a DPO fine-tune of **Beagle14-7B**, which was itself a merge. Its results therefore cannot be attributed to interpolation alone.

**DARE** takes a pruning-and-rescaling approach to reducing redundancy. Labonne describes this informally as retaining significant parameters. More precisely, the [original DARE method](https://arxiv.org/abs/2311.03099) randomly drops entries from the fine-tuning deltas—the changes relative to a base model—and rescales the retained deltas by `1 / (1 − p)`, where `p` is the drop probability. It does not select the largest raw model weights. This approach can be used to merge more than two source models.

For multi-model experiments, Labonne recommends trying seven or eight sources with DARE, based on his experience. That is practical guidance rather than a measured optimum: the useful distinction is that SLERP is pairwise, while the DARE-based approach he recommends can combine contributions from a larger group.

### Repeat layers without another training run

[13:17](https://www.youtube.com/watch?v=uLrOI65XbDw&t=797s)

**Passthrough** changes the construction more directly: concatenate layers from different LLMs, or reuse layers from the same model. The latter is a self-merge. Labonne describes creating [**Meta-Llama-3-120B-Instruct**](https://huggingface.co/mlabonne/Meta-Llama-3-120B-Instruct) from **Llama 3 70B Instruct** by repeating a group of ten layers six times. This is layer stacking rather than interpolation between corresponding weights.

The counterintuitive part is that he did not train the enlarged model afterward. Labonne reports enthusiastic online reception, particularly for creative writing, while also describing the model as erratic. That is a reported use-case assessment, not evidence of general superiority. When a questioner asks whether there was any fine-tuning at all, he explicitly confirms there was none.

### Assemble experts and initialize a router from prompts

[14:26](https://www.youtube.com/watch?v=uLrOI65XbDw&t=866s)

A conventional **mixture of experts**, or MoE, is pretrained with a router and multiple feedforward networks. A **FrankenMoE** instead extracts feedforward layers from existing fine-tuned models, combines those layers as experts, and adds a router. The construction reuses already-trained components rather than pretraining the whole expert system from scratch.

For **Beyonder**, Labonne selects four fine-tuned sources with different specialties: chat, code, role-play and mathematics. The MergeKit configuration shown associates positive prompts with the experts. These prompts are not additional SFT examples in this construction; they help initialize routing.

The router’s decision happens **for each token at each layer**: which expert feedforward networks should process this token here? The described setup generally selects two experts. To initialize that behavior without a new fine-tuning run, the construction calculates embeddings from the positive prompts and uses those embeddings to initialize the routers. Further fine-tuning is possible, but it is not required for the assembly process Labonne demonstrates.

### Evaluate the assembly against its sources

[16:11](https://www.youtube.com/watch?v=uLrOI65XbDw&t=971s)

Labonne also describes **Phixtral**, for which he had to modify the implementation for compatibility with **Phi-2**. He reports that Phixtral outperformed its base model on multiple tasks. Even so, when performance is the priority, he recommends SLERP or DARE ahead of FrankenMoE, which he considers more experimental and less effective in his experience.

The final Beyonder comparison uses its own source models as baselines. Labonne reports that Beyonder outperforms those source models on multiple benchmarks. That makes the comparison particularly useful: it asks whether assembling the experts improves on the components already available, rather than comparing only with an unrelated model. The reported gains coexist with his preference for simpler merging techniques when seeking performance.

For hands-on continuation, Labonne points to his [Large Language Model Course](https://github.com/mlabonne/llm-course) and its GitHub notebooks. The closing slide brings together entry points for merging, evaluation, SFT, DPO, Axolotl and quantization, so readers can move from these mechanisms to a concrete experiment.

![Six boxes list LazyMergekit, LLM AutoEval, Mistral-7b fine-tuning with SFT+TRL and DPO+TRL, LazyAxolotl, and AutoQuant.](https://ai.engineer/api/talk-essay-media/14f19a4cf1c77393898e9aa2a6689fbefd45bc0d85f0ea218c20efde7d72aa67)

Large Language Model Course resources with Open in Colab buttons.

## Resources

- [Large Language Model Course](https://github.com/mlabonne/llm-course) — Labonne's LLM learning roadmaps and practical Colab notebooks.
- [MergeKit](https://github.com/arcee-ai/mergekit) — Tools and configuration documentation for merging pretrained language models.
- [NeuralBeagle14-7B](https://huggingface.co/mlabonne/NeuralBeagle14-7B) — Model card documenting NeuralBeagle's merged ancestry and subsequent DPO training.
- [Meta-Llama-3-120B-Instruct](https://huggingface.co/mlabonne/Meta-Llama-3-120B-Instruct) — Labonne's Llama 3 self-merge model card, including creative-writing guidance and limitations.
- [Beyonder-4x7B-v2](https://huggingface.co/mlabonne/Beyonder-4x7B-v2) — A four-expert model assembled from chat, code, role-play and mathematics models.
- [DARE: Language Models are Super Mario](https://arxiv.org/abs/2311.03099) — Original research on randomly dropping and rescaling fine-tuning deltas before merging models.
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) — Original QLoRA research explaining four-bit frozen base weights, trainable adapters and memory savings.
- [How Enterprises Were Building and Buying Generative AI in 2024](https://a16z.com/generative-ai-enterprise-2024/) — a16z's enterprise interviews and survey covering model selection, customization and spending intentions.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=uLrOI65XbDw&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7ee053f08ecd437190ab7919f061f9b3aac80e9c81aeabd912d59a65315b893.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=uLrOI65XbDw&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/1fdb2fcc75f0b843b514ecf929f7eec844e36eaf6db9489e770dd38f586dea8d.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB
Google Cloud
neo4j

### About Maxime Labonne

Maxime Labonne introduces his work at Liquid AI, his LLM Course with over 33k GitHub stars, his open-source models and tools, and his book on graph neural networks.

[0:28](https://www.youtube.com/watch?v=uLrOI65XbDw&t=28s) [Maxime Labonne's photo and a summary of his work, models, tools and book.](https://assets.ai.engineer/sites/aiecode2025/sha256/8dc263ebe9504abc63d7af99f7b856f10292fb750c2bf3950583d572b6afbbc4.webp)

About Maxime Labonne

PhD, Staff Machine Learning Scientist @ Liquid AI, GDE
Writing: blog posts, LLM Course (>33k⭐ on GitHub), etc.
Models: NeuralDaredevil, AlphaMonarch, Phixtral, etc.
Tools: LLM AutoEval, LazyMergekit, AutoQuant, etc.
Author of Hands-On Graph Neural Networks using Python

Experts
2024
Machine Learning
Liquid

### LLM training lifecycle

Raw text pre-training produces a base model. Supervised fine-tuning on instructions and answers produces an instruct model, and preference alignment produces a chat model.

[0:52](https://www.youtube.com/watch?v=uLrOI65XbDw&t=52s) [A diagram progresses from pre-training on raw text to supervised fine-tuning on instructions and answers, then preference alignment, producing a chat model.](https://assets.ai.engineer/sites/aiecode2025/sha256/85f0a121ca165516a8c1e2604ef5e025b508e0db1db7d73187da9795e8b3ec75.webp)

Raw text
The Pile
RedPajama
Dolma
C4

Pre-training → Base model

Instructions & answers
SlimOrca
OpenHermes
UltraChat
Alpaca

Supervised fine-tuning → Instruct model

AI or human preferences
orca-dpo-pairs
Nectar
dpo-mix-7k
truthy-dpo

Preference alignment → Chat model

### When to Use Fine-Tuning?

Start with prompt engineering and evaluate the result. If it falls short, fine-tuning is an option when you can make an instruction dataset; otherwise, rescope the project. Evaluate again after fine-tuning.

[1:53](https://www.youtube.com/watch?v=uLrOI65XbDw&t=113s) [Guidance on fine-tuning with a flowchart for evaluating prompt engineering and deciding whether to fine-tune.](https://assets.ai.engineer/sites/aiecode2025/sha256/c558e4f75a91cf4306838cf0f936f3e774b9dddd525a3bf26f96f8ca6a9396ca.webp)

When to Use Fine-Tuning?

• Prompt engineering + fine-tuning is often the best option if you can afford it
• Data-driven approach: always evaluate your system using quantitative techniques
• Distilling a large model into a smaller one can be interesting even if cost and latency are ok

Start with prompt engineering
Evaluation
Is it good enough? (cost, latency, etc.)
Yes → Solved!
No → Can you make an instruction dataset?
No → Rescope the project
Yes → Fine-tuning is an option
Evaluation

### Fine-Tuning Libraries

TRL builds on Transformers, Axolotl uses reusable YAML configurations, LLaMA-Factory offers a GUI, and Unsloth targets efficient single GPU fine-tuning with utilities.

[3:30](https://www.youtube.com/watch?v=uLrOI65XbDw&t=210s) [Overview of TRL, Axolotl, LLaMA-Factory and Unsloth.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bde4e17e27f8232fac49361f5cb58f78a57d9e7ed18ca26516daa6728914102.webp)

Fine-Tuning Libraries

TRL
HF’s fine-tuning library built on top of transformers

Axolotl
Versatile fine-tuning library based on reusable YAML configurations

LLaMA-Factory
User-friendly fine-tuning library with a GUI

Unsloth
Efficient single GPU fine-tuning with useful utils

### Supervised Fine-Tuning (SFT)

SFT trains on instruction-and-answer pairs. In this example, the system and user prompts provide context; the speaker says training usually masks them and focuses on the output. Many SFT datasets use synthetic data.

[3:56](https://www.youtube.com/watch?v=uLrOI65XbDw&t=236s) [SFT uses instruction-and-answer pairs, illustrated by a system prompt, user task, and output; synthetic data is also discussed.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c8fc700e60a73d2c9f0da772a14d6cf1988dcf5e1f93d6af9aea79bf219217c.webp)

Supervised Fine-Tuning (SFT)

• Like pre-training, SFT use next-token prediction as training objective (see InstructGPT paper)
• Main difference = dataset used (raw text vs. pairs of instructions and answers). Example:

System
You are a helpful assistant, who always provide explanation. Think like you are answering to a five year old.

User
Remove the spaces from the following sentence: It prevents users to suspect that there are some hidden products installed on theirs device.

Output
Itpreventsuserstosuspectthattherearesomehiddenproductsinstalledontheirsdevice.

• Most SFT datasets use synthetic data generated by models like GPT-3.5 and GPT-4 (see Orca paper)

### Hyperparameters: Learning Rate

Training loss stays low before a sharp spike. The speaker recommends experimenting with model-dependent learning rates, then reducing the rate if loss explodes; their default is 1e-5.

[9:21](https://www.youtube.com/watch?v=uLrOI65XbDw&t=561s) [Learning-rate guidance above a training-loss graph that spikes sharply.](https://assets.ai.engineer/sites/aiecode2025/sha256/cace5609c8cbae604ee51951460f0a4cc7526f0b20d554ac46e7919d30e62172.webp)

Hyperparameters

• Learning rate: model-dependent, requires multiple experiments.
Generally, as high as you can as long as the loss doesn’t explode. My default value is 1e-5.

train/loss
Y-axis ticks: 0, 2, 4, 6, 8, 10, 12
X-axis ticks: 500, 1k, 1.5k, 2k
[illegible]

### Hyperparameters: Epochs, Sequence Length, Batch Size and LoRA

Choose epochs based on dataset size, then balance sequence length and batch size against VRAM. The slide suggests gradient accumulation for an effective batch size of 8 and a LoRA rank of 16–64.

[9:57](https://www.youtube.com/watch?v=uLrOI65XbDw&t=597s) [Guidance on epochs, sequence length, batch size, VRAM usage and LoRA rank.](https://assets.ai.engineer/sites/aiecode2025/sha256/afeb609b684b64a88766a7d4507cf4ce8620ecca7e54790bd979bc8c36d9a5a5.webp)

Hyperparameters

• Number of epochs: depends on the learning rate and the number of tokens.
  • Large datasets (>2M samples) = 1-2 epochs
  • Small datasets (<1M samples) = 3-4 epochs

• Sequence length: 25-50% of the context window of the base model is okay. Tradeoff with the batch size in terms of VRAM usage.

• Batch size: as big as you can. Use gradient accumulation steps to have an effective batch size of 8. Ideally, we want a VRAM usage >90% (but lower than ~98% or you’re going to OOM).

• LoRA: a rank of 16-64 is good in most cases

### Merge techniques: passthrough

Passthrough concatenates layers from one or more LLMs. In a self-merge, the speaker repeats 10 layers of Llama 3 70B six times to create Meta-Llama-3-120B-Instruct.

[13:18](https://www.youtube.com/watch?v=uLrOI65XbDw&t=798s) [Passthrough merging concatenates model layers; an example repeats layers to create a larger model.](https://assets.ai.engineer/sites/aiecode2025/sha256/6831c0c2e0c3c864b10095118ba836e70edfc3bab8ab378c6d76986a2876184c.webp)

Merge techniques

Passthrough
• Frankenmerges: concatenate layers from different LLMs (or the same one)
• More layers/parameters = better reasoning abilities (even with the same model).

Example: Meta-Llama-3-120B-Instruct → upscales L3 70B by repeating 10 layers 6 times

Example: mlabonne/Meta-Llama-3-120B-Instruct

Inspired by other large merges like goliath-120b, Venus-120b-v1.0, MegaDolphin-120b, miquliz-120b-v2.0.

### Large Language Model Course

The course offers six Colab notebooks, including Mistral-7b fine-tuning with SFT+TRL or DPO+TRL. The speaker says the notebooks are available on GitHub.

[17:12](https://www.youtube.com/watch?v=uLrOI65XbDw&t=1032s) [Closing slide listing six language-model course notebooks, including fine-tuning and evaluation tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/06c1da2b0352e765d24d25f20e201765a40e6cd88b5d392704b04c9f6a770d26.webp)

Large Language Model Course

LazyMergekit
Open in Colab

LLM AutoEval
Open in Colab

Fine-tune Mistral-7b with SFT+TRL
Open in Colab

Fine-tune Mistral-7b with DPO+TRL
Open in Colab

LazyAxolotl
Open in Colab

AutoQuant
Open in Colab

Thanks! :)
Liquid

## Full transcript

[0:00](https://www.youtube.com/watch?v=uLrOI65XbDw&t=0s) [upbeat music] Hi, everyone.

[0:15](https://www.youtube.com/watch?v=uLrOI65XbDw&t=15s) Uh, in this session, we're gonna talk about, uh, everything you need to know about fine-tuning LLMs and model merging. Uh, quick intro, my name is Maxime Labonne. Uh, I'm a Staff Machine Learning Scientist at Liquid AI.

[0:28](https://www.youtube.com/watch?v=uLrOI65XbDw&t=28s) I'm also a Google Developer Expert. I write blog posts on these topics. I created the LLM Course, which is, uh, super popular on GitHub. Um, I also contributed to the open source community through models, uh, through tools, and I'm the author of, uh, Hands-On Graph Neural Networks using Python, uh, with Packt.

[0:47](https://www.youtube.com/watch?v=uLrOI65XbDw&t=47s) Um, so first of all, let's talk about fine-tuning. Uh, we saw a bit of fine-tuning in the previous session, so I'll try to not repeat too much. Uh, but basically, here's the LLM training lifecycle.

[1:00](https://www.youtube.com/watch?v=uLrOI65XbDw&t=60s) You see three stages. Uh, first of all, you have the pretraining stage, uh, where you give a lot of raw text to the model, and the idea is that the model learns to do, uh, next-token prediction.

[1:12](https://www.youtube.com/watch?v=uLrOI65XbDw&t=72s) The result of that is called a base model. This base model, uh, is really nice, uh, but if you ask it questions or instructions, it's going to autocomplete your question instead of answering it, which is why we have the supervised fine-tuning stage, where this time we give pairs of questions and answers to the model.

[1:32](https://www.youtube.com/watch?v=uLrOI65XbDw&t=92s) And, uh, we have a similar, uh, training objective, but the idea is that, uh, at the end of it, it's going to actually answer your questions and follow your instructions.

[1:41](https://www.youtube.com/watch?v=uLrOI65XbDw&t=101s) Then we have a third and final stage, the preference alignment stage, where we give, uh, human preferences to align the model to how we want it to behave, and the result is commonly referred to as chat model.

[1:54](https://www.youtube.com/watch?v=uLrOI65XbDw&t=114s) So when to use fine-tuning. Uh, here you can see a little flowchart that I've made. It's very high level. Uh, but basically, uh, there's a conversation about when to use prompt engineering, when to use fine-tuning.

[2:07](https://www.youtube.com/watch?v=uLrOI65XbDw&t=127s) I think it's good, in general, to start with prompt engineering if you can, and, uh, the idea is to have a really robust evaluation, uh, structure, where you have a lot of different metrics that you're interested in.

[2:19](https://www.youtube.com/watch?v=uLrOI65XbDw&t=139s) It can be, uh, the accuracy of the model. Uh, does it answer my question well? You can create a custom benchmark, uh, if you have a very niche use case, or you can reuse, um, open source benchmarks.

[2:31](https://www.youtube.com/watch?v=uLrOI65XbDw&t=151s) Also, in terms of cost, latency, because the question is, is it good enough? If it's good enough with just prompt engineering, then probably you don't need, uh, fine-tuning. The problem is solved.

[2:40](https://www.youtube.com/watch?v=uLrOI65XbDw&t=160s) Uh, congrats. Otherwise, the question is, can you make an instruction dataset? So can you create pairs of questions and answers, uh, to fine-tune the model? If it's not the case, it can be for multiple reasons, but it's probably a good sign that, uh, you need to rescope the project.

[2:56](https://www.youtube.com/watch?v=uLrOI65XbDw&t=176s) Uh, otherwise, fine-tuning is an option, and you can reuse the evaluation framework that you created, uh, to evaluate the model. Uh, so, so that was the technical answer, but you also have a non-technical answer to that.

[3:08](https://www.youtube.com/watch?v=uLrOI65XbDw&t=188s) Uh, here is a report from a16z, and the question is, why do enterprises care about open source? You can see that the, the two main items are actually control and customizability, and customizability is mostly about fine-tuning models.

[3:22](https://www.youtube.com/watch?v=uLrOI65XbDw&t=202s) So even if there's, like, arguments about the technical side and cost and latency, there's also, like, a strong argument for customizability and control over these models. So in terms of fine-tuning libraries, I think that you know about Unsloth now, uh, but I'm gonna talk about the other ones.

[3:38](https://www.youtube.com/watch?v=uLrOI65XbDw&t=218s) Uh, so TRL from, uh, Hugging Face, a great library built on top of Transformers, very easy to use. You have Axolotl, excellent library, uh, very versatile. Uh, you, you have a lot of, um, YAML config files.

[3:51](https://www.youtube.com/watch?v=uLrOI65XbDw&t=231s) And then you have LlamaFactory, uh, where you have a really good graphical user interface, uh, that is built in. So to talk a bit more about supervised fine-tuning, um, here you see an example of a sample that we, we give to the model.

[4:06](https://www.youtube.com/watch?v=uLrOI65XbDw&t=246s) So we have the instruction, which is both the system prompt and the user prompt, and the answer, which is the output. So in this, uh, in this case, the system prompt is used to steer the behavior of the model.

[4:18](https://www.youtube.com/watch?v=uLrOI65XbDw&t=258s) Uh, think like you're answering to a [REDACTED:age]. And the user actually gives the task, "Remove the spaces from the following sentence." We train the model usually, like, generally, on the output only, so we mask the rest.

[4:32](https://www.youtube.com/watch?v=uLrOI65XbDw&t=272s) It's used as context, and what we want to, to do is train the model to output the correct answer. Uh, most SFT datasets, I want to say, use synthetic data, and that's perfectly fine.

[4:42](https://www.youtube.com/watch?v=uLrOI65XbDw&t=282s) Usually, it's generated with, uh, frontier models, and that's a, a great way of, uh, building high-quality datasets.

[4:50](https://www.youtube.com/watch?v=uLrOI65XbDw&t=290s) Then you have the preference alignments. I'm just gonna mention it here. Uh, there are a lot of different methods, PPO, DPO, KTO, IPO. Uh, in practice, Direct Preference Optimization is probably the most popular one.

[5:01](https://www.youtube.com/watch?v=uLrOI65XbDw&t=301s) Um, so here you see that you have a different format, uh, with an instruction, and you have a chosen answer, and a rejected answer. So the idea here is that you're going to show, like, a positive example, a negative example to the model.

[5:14](https://www.youtube.com/watch?v=uLrOI65XbDw&t=314s) And with DPO, the goal is to make sure that the model that you're currently training outputs higher probabilities, uh, for the chosen answers than the untrained version of the same model.

[5:24](https://www.youtube.com/watch?v=uLrOI65XbDw&t=324s) Not gonna delve, uh, too much into the details here, but this is the general idea and can be used to either censor the model. How to make a bomb, uh, the chosen answer would be, "As an AI assistant, I cannot tell you that."

[5:35](https://www.youtube.com/watch?v=uLrOI65XbDw&t=335s) Or it can also be used to, um, boost the performance of the model in general. Uh, how to create SFT datasets. So this is a very fundamental question, uh, in the post-training world.

[5:47](https://www.youtube.com/watch?v=uLrOI65XbDw&t=347s) Uh, and the, the main question is, okay, what's a good sample? Human evaluation is quite bad at, um, actually reviewing the samples, but what I like to define is, like, three main features.

[5:59](https://www.youtube.com/watch?v=uLrOI65XbDw&t=359s) The first one is the accuracy. We want the samples, the outputs to be, uh, factually correct. Um, maybe no typos would be good too. Uh, we don't want to compromise, uh, the knowledge of the model by giving it fake information.

[6:15](https://www.youtube.com/watch?v=uLrOI65XbDw&t=375s) Uh, then you have diversity. And diversity, you want to cover, um, as many topics as you can. Of course, it depends on your use case because if you do summarization, uh, you won't be as general as if you do, uh, general purpose fine-tuning.

[6:28](https://www.youtube.com/watch?v=uLrOI65XbDw&t=388s) Um, but it's always a good idea to include a lot of different, uh, topics, different writing styles, um, in this, uh, dataset. And finally, you have complexity. I think this one is, is a bit less trivial.

[6:42](https://www.youtube.com/watch?v=uLrOI65XbDw&t=402s) Uh, and it's about giving complex tasks to the model, forcing reasoning. So, for example, the output will have chain-of-thought reasoning because you want to train the model to have this kind of, of reasoning.

[6:54](https://www.youtube.com/watch?v=uLrOI65XbDw&t=414s) Or it can be task, like summarization, explain me like I'm a [REDACTED:age]. Uh, this kind of task really, um, force the model not to only answer the question like a QA with, um, answers you could find on Wikipedia.

[7:08](https://www.youtube.com/watch?v=uLrOI65XbDw&t=428s) It also forces it to reason over the, the prompt and, uh, give a more complex answer. So as a, a little recipe you can see here, um, I would recommend in general, uh, starting with open source datasets if you can.

[7:22](https://www.youtube.com/watch?v=uLrOI65XbDw&t=442s) Uh, combine some of them. Then you can apply different filters. The first one is data de-duplication. It can be either exact because you want to remove, uh, duplicates. It can be fuzzy, uh, so same, um, same idea.

[7:36](https://www.youtube.com/watch?v=uLrOI65XbDw&t=456s) And, uh, then you have data quality filters. Here you have different techniques. Can be rule-based filtering. Uh, for example, you want to remove every single row where you have, "As an AI assistant, I cannot..."

[7:47](https://www.youtube.com/watch?v=uLrOI65XbDw&t=467s) because people hate it. Uh, but you can also use, uh, more clever techniques like reward models or LLM-as-a-judge to evaluate the quality of each sample and filter out the bad samples.

[7:59](https://www.youtube.com/watch?v=uLrOI65XbDw&t=479s) And then you can use data exploration with different tools like Lilac, Nomic Atlas, text clustering to have, um, topic clustering to visualize your dataset, uh, to get ideas on how to improve it.

[8:10](https://www.youtube.com/watch?v=uLrOI65XbDw&t=490s) And with these ideas, you can, uh, go back to data generation and start the process all over again. In terms of L- SFT techniques, we have three main techniques.

[8:19](https://www.youtube.com/watch?v=uLrOI65XbDw&t=499s) Uh, full fine-tuning, this is like the most basic one. You take the base model and you just, uh, train it on the instruction, uh, dataset. It has the best performance, but is also, like, very inefficient in general.

[8:31](https://www.youtube.com/watch?v=uLrOI65XbDw&t=511s) A more efficient, uh, way of seeing it is LoRA. Uh, with LoRA, you are going to freeze all the pre-trained weights and you add adapters to each targeted layer.

[8:41](https://www.youtube.com/watch?v=uLrOI65XbDw&t=521s) Uh, these matrices A and B are these adapters. So you have, um... You don't train on all the parameters of the base model, you only train a subset of them.

[8:53](https://www.youtube.com/watch?v=uLrOI65XbDw&t=533s) Uh, so this is a, a lot faster, uh, but it can still be costly because you're still loading the entire model in, uh, six-bit, sixteen-bit precision here. So a more efficient way is to quantize, uh, the pre-trained model here in four-bit precision.

[9:09](https://www.youtube.com/watch?v=uLrOI65XbDw&t=549s) This is QLoRA, and, uh, you apply the same idea that you had with LoRA, but this time, uh, the, the weights are heavily quantized, so you have a lower VRAM usage.

[9:19](https://www.youtube.com/watch?v=uLrOI65XbDw&t=559s) But the problem is that it also degrades performance, so there's a trade-off here. Um, I want to briefly mention some hyperparameters, but Daniel al- already, uh, talked about a lot of them, so I, I'm gonna be brief.

[9:30](https://www.youtube.com/watch?v=uLrOI65XbDw&t=570s) I think the most important one is the learning rate. Uh, the learning rate is model-dependent. Uh, it requires a few experiments to be able to really tweak it and find the, the best one.

[9:40](https://www.youtube.com/watch?v=uLrOI65XbDw&t=580s) Generally, I would recommend to go as high as you can, uh, until your loss explodes, like in this graph. Uh, then you can, uh, reduce the size of the learning rate.

[9:51](https://www.youtube.com/watch?v=uLrOI65XbDw&t=591s) Uh, other super important hyperparameters, number of epochs. Um, I would say that depending on the size of your dataset, you can have, like, more or less epochs. Sequence length is also good, uh, because, um, it, it's a trade-off with the batch size because the longer, uh, sequence length you have...

[10:08](https://www.youtube.com/watch?v=uLrOI65XbDw&t=608s) So the, the bigger the context window, the more VRAM, uh, you're gonna use. Uh, but you don't need to use, uh, a sequence length that's as big as the pre-trained model.

[10:17](https://www.youtube.com/watch?v=uLrOI65XbDw&t=617s) Then you have the batch size. You want to maximize it to maximize the utilization of your GPUs. Um, and then you have, uh, the LoRA, uh, with the rank.

[10:26](https://www.youtube.com/watch?v=uLrOI65XbDw&t=626s) Um, this is, like, quite easy to, to, to fine-tune, so I don't want to go into the details here. Um, let's talk about model merging now. So model merging is the idea that you can take the weights of different fine-tuned models, and you can combine them together.

[10:42](https://www.youtube.com/watch?v=uLrOI65XbDw&t=642s) So you, you just can leverage, uh, what the open source community, uh, has, uh, produced on the Hugging Face, uh, Hub, for example.

[10:53](https://www.youtube.com/watch?v=uLrOI65XbDw&t=653s) Uh, it doesn't require any GPU, so it's super efficient, and it provides excellent results. Uh, so the Open LLM Leaderboard was updated this morning, uh, so we have a version two now.

[11:04](https://www.youtube.com/watch?v=uLrOI65XbDw&t=664s) But this is the version one. Uh, I haven't had time to update it. Uh, but you can see that for 7B, uh, parameter models, uh, the entire top eight or top 10 is just merged models.

[11:16](https://www.youtube.com/watch?v=uLrOI65XbDw&t=676s) Uh, so it really shows that, uh, this approach is extremely effective at producing high-quality models. Uh, and you can find similar results on, like, a really a lot of different datasets.

[11:27](https://www.youtube.com/watch?v=uLrOI65XbDw&t=687s) I would recommend using MergeKit. Uh, this is like the leading library in this, uh, in this space, uh, with a lot of different techniques that are implemented there. Um, so here you can see the family tree of, uh, merged models.

[11:40](https://www.youtube.com/watch?v=uLrOI65XbDw&t=700s) Uh, so in... You don't really need to see the, the name of the model, but you see that every node is actually a model. And we actually merge different merges together until it becomes like a giant family tree.

[11:53](https://www.youtube.com/watch?v=uLrOI65XbDw&t=713s) This one is actually quite small. Like, it can get a lot crazier than that, but it didn't fit in, on one slide, so I, I chose this one instead.

[12:01](https://www.youtube.com/watch?v=uLrOI65XbDw&t=721s) Um, about the merge techniques themselves, I want to mention- Like a few of them. The first one is called SLERP. It stands for spherical linear interpolation. So the idea is really to apply spherical but, uh, linear interpolation, uh, with the weights of different models.

[12:19](https://www.youtube.com/watch?v=uLrOI65XbDw&t=739s) You can only merge two models at the same time with this technique, uh, but you can really tweak it, uh, with different, uh, interpolation factors for different layers. Uh, here's a model that I've made, NeuralBeagle, uh, 14-7B, uh, which was a really, um, efficient way of, uh, leveraging, uh, the, uh, different models that were

[12:39](https://www.youtube.com/watch?v=uLrOI65XbDw&t=759s) created by the open source community. And then you have DARE. Uh, so in DARE you want to reduce the redundancy of the model parameters. Uh, to do that you're gonna use pruning.

[12:49](https://www.youtube.com/watch?v=uLrOI65XbDw&t=769s) You're gonna select the most significant parameters in your model weights, and you're gonna rescale, um, the weights of these, uh, source models. Uh, the advantage that it has is that you can merge different models, not just two, but even more together.

[13:04](https://www.youtube.com/watch?v=uLrOI65XbDw&t=784s) And, uh, I would advise, uh, I would recommend this technique, and not with just two model, not with three, but like, with seven or eight models. It works really, really well, uh, so I strongly recommend that.

[13:17](https://www.youtube.com/watch?v=uLrOI65XbDw&t=797s) Uh, then you have a very funny, um, uh, technique called passthrough. And in passthrough you can concanate, concatenate, uh, layers from different LLMs. Um, it can also be the same one.

[13:28](https://www.youtube.com/watch?v=uLrOI65XbDw&t=808s) We call it self-merge. Um, and so here you have an example that I've made, uh, recently. It's called Meta-Llama-3-120B-Instruct because I took Llama 3 70B Instruct and I just, uh, repeated 10 layers 6 times.

[13:43](https://www.youtube.com/watch?v=uLrOI65XbDw&t=823s) Uh, so you could say, like, this shouldn't work at all. Like, come on, uh, you, you haven't even trained the model. This is ridiculous. Uh, but actually, yeah, this is ridiculous.

[13:52](https://www.youtube.com/watch?v=uLrOI65XbDw&t=832s) Uh, people loved it on Twitter and Reddit and online in general. Uh, so it, it shows that, uh, there's a lot of things that we can still discover, uh, with these m- merge techniques, with these models.

[14:06](https://www.youtube.com/watch?v=uLrOI65XbDw&t=846s) Uh, they do not, um... They can be counterintuitive sometimes, and you can see that this model in particular was particularly good at, uh, creative writing. Uh, it was also quite unhinged in general, but, uh, really good at creative writing and, uh, now it's being used by a lot of people, uh, even though it's, it's super big.

[14:24](https://www.youtube.com/watch?v=uLrOI65XbDw&t=864s) But no kind of fine-tuning at all?

[14:26](https://www.youtube.com/watch?v=uLrOI65XbDw&t=866s) No, no fine-tuning. Nothing. Um, and then I want to mention the last techniques, uh, which is called mixture of experts. Uh, so in traditional mixture of experts you are going to pre-train a model, uh, with a router you can see on the bottom here, and different feedforward network layers.

[14:46](https://www.youtube.com/watch?v=uLrOI65XbDw&t=886s) And you pre-training, pre-train it from scratch. But you can do something quite smart, uh, with merging, where you extract the feedforward network layers from different fine-tune models and you combine them together, uh, like this.

[15:00](https://www.youtube.com/watch?v=uLrOI65XbDw&t=900s) So we call this a FrankenMoE. You add a router. You combine the FFN layers from different, uh, models, and this is how you create, like, your kind of, uh, mixture of experts.

[15:12](https://www.youtube.com/watch?v=uLrOI65XbDw&t=912s) It's actually, uh, pretty cool. It, it works pretty well in practice. Uh, you can see on the left a MergeKit config, uh, for the Beyonder, uh, model. Uh, so for this model I selected four different, uh, fine-tune models.

[15:27](https://www.youtube.com/watch?v=uLrOI65XbDw&t=927s) One, uh, as a chat model, one as a code model, one as a role play model, and one as a math model. You can see that I'm using positive prompts here.

[15:35](https://www.youtube.com/watch?v=uLrOI65XbDw&t=935s) So actually i- it's, it's a way to initialize the router, because if you go back to the previous, uh, slide, we can see that the router is supposed to, uh, select for each token and each layer where, like, which feedforward network, uh, layer is going to be used.

[15:52](https://www.youtube.com/watch?v=uLrOI65XbDw&t=952s) We use two, uh, in general. Um, and so how do we initialize it if we do not fine-tune it? Once again, we don't want to fine-tune it. We can, but we, we don't necessarily want to.

[16:02](https://www.youtube.com/watch?v=uLrOI65XbDw&t=962s) In this case, we're just going to use these positive prompts, uh, calculate the embeddings, and use these embeddings to initialize, uh, the routers, and that works really, really well.

[16:11](https://www.youtube.com/watch?v=uLrOI65XbDw&t=971s) So those are two models that I've used. For Phixtral I had to modify it to make it compatible with Phi-2, and, uh, that outperformed the base model, uh, on a lot of task.

[16:22](https://www.youtube.com/watch?v=uLrOI65XbDw&t=982s) Um, so it's really a good technique to, to use in general. But I would say that, um, if you compare it to merging, uh, as we saw with SLERP and with DARE, I would say that, um, if you, uh, want to increase the performance, it's better to use, uh, SLERP and DARE instead of mixture of experts because

[16:43](https://www.youtube.com/watch?v=uLrOI65XbDw&t=1003s) this is a bit more experimental. Uh, this doesn't... This will not bring you the same level of performance. Um, and here you can see the results of, uh, the Beyonder, uh, model.

[16:55](https://www.youtube.com/watch?v=uLrOI65XbDw&t=1015s) Uh, you can see that the other models I'm comparing to are the source models that I've used in this, in this merge. Uh, so it, it's quite remarkable to see that it's actually performing better than the source models, um, on the, on a lot of different benchmarks.

[17:11](https://www.youtube.com/watch?v=uLrOI65XbDw&t=1031s) Um, so yep, that's it for me. Uh, thank you for your attention. Uh, if you are interested in, uh, knowing more, if you want notebooks, uh, to, to run some code, I created the Large Language Model Course.

[17:25](https://www.youtube.com/watch?v=uLrOI65XbDw&t=1045s) All these notebooks are available on GitHub, uh, llm-course. And, uh, yeah, thank you. [audience applauds] [upbeat music]
