AI Engineer World's Fair 2024
Fixing bugs in Gemma, Llama & Phi-3
Read the talk
Fixing the small mismatches that break a fine-tune
From Gemma’s activation function to Llama 3’s special tokens, reliable fine-tuning depends on preserving the model’s assumptions through training, conversion and inference.
From a talk by Daniel Han
Before you start: Familiarity with tokenization, supervised fine-tuning and LoRA will help; the code examples use Python and PyTorch.
The activation function is part of the model
What happens when a model expects approximate GELU, but its implementation uses exact GELU? The mathematically more exact function is still the wrong function for those weights. Gemma provides Daniel Han’s opening example: one of Unsloth’s fixes was to use the approximate activation the model expected. The opening slides show GELU code comparisons alongside other implementation and precision differences.
Tokenization creates another class of mismatches. Han points to the Gemma bug report and its accompanying Colab, then briefly revisits Phi-3: he reports a sliding-window correction from 2,047 to 2,048 and recommends separating its fused QKV matrices for LoRA fine-tuning. These are the background to the main walkthrough, Llama 3, which he introduces as eight findings, including some not yet publicly announced.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Insert the beginning-of-sequence token once
A fine-tune can train successfully and still encounter a different input format at inference. Han’s first Llama 3 example is duplicate beginning-of-sequence tokens: training examples start with two BOS tokens, while inference prompts start with one. He reports lower inference accuracy from this mismatch. Unsloth checks for the extra token and removes it automatically; Han notes that the same issue also affects Mistral and Gemma.
Llama 3 needs a BOS token, but only one component should insert it. The distinction is between rendering a chat template and subsequently tokenizing that rendered text. A template can already contain the special tokens, so adding them again during tokenization duplicates them. Current Hugging Face chat-template guidance makes the two safe paths explicit: tokenize inside apply_chat_template, or render first and then tokenize with add_special_tokens=False.
python
def tokenize_conversation(tokenizer, messages):
return tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=False,
)
def tokenize_rendered_conversation(tokenizer, messages):
rendered = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=False,
)
return tokenizer(
rendered,
add_special_tokens=False,
)["input_ids"]
Both functions assume a model-appropriate template that supplies its required special tokens. Inspect the resulting token sequence before training; checking only the human-readable prompt can miss an extra insertion by the tokenizer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A chat template can reference untrained embeddings
The next mismatch is between the checkpoint and its template. Llama 3 Base and Llama 3 Instruct do not have the same training history for special tokens. Han warns that applying the instruct template to the base model can introduce untrained tokens and produce NaN gradients during fine-tuning. His list includes reserved special tokens numbered 0–250, the end-of-turn token, and the start-header and end-header tokens. The embedding-mean graph shows some entries at zero: tokens deliberately left unused in the base model.
The proposed repair initializes untrained rows from the mean of the trained rows. Unsloth performs this automatically. Exclude untrained rows from the mean itself, including its denominator. Otherwise, zero rows dilute the initialization. Han’s example of 10,000 untrained tokens illustrates this arithmetic; it is not a count of the affected Llama 3 tokens. The Llama 3 companion article describes the same masked-mean repair.
python
import torch
@torch.no_grad()
def initialize_untrained_rows(weight, untrained_mask):
# weight: [vocabulary_size, embedding_dimension]
# untrained_mask: one Boolean per vocabulary row
untrained_mask = untrained_mask.to(
device=weight.device, dtype=torch.bool
)
trained_mask = ~untrained_mask
if not trained_mask.any():
raise ValueError("At least one trained row is required")
if not untrained_mask.any():
return
mean_row = weight[trained_mask].float().mean(dim=0)
weight[untrained_mask] = mean_row.to(weight.dtype)
The mask must identify the untrained rows for the checkpoint being repaired. The function leaves trained rows unchanged and computes the average only over them.
For the ordinary fine-tuning path, Han’s recommendation is to use the instruct template with the instruct checkpoint. If the task requires training the base model with these tokens, another remedy is to train the token embeddings and language-model head rather than leaving those parameters frozen. That lets the newly used token representations learn alongside the rest of the adaptation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Do not mask away the instruction to stop
Padding creates a subtler failure. In a training pipeline that masks labels by the padding token’s identity, assigning PAD and EOS the same token ID removes both from the cross-entropy loss. The real end-of-sequence token then provides no stopping supervision. Han describes the resulting symptom as infinite generation and points to Phi-3’s matching PAD and EOS IDs. The mechanism depends on how labels are masked: sharing an ID is dangerous here because the masking rule cannot distinguish genuine endings from padding.
Consider a teaching sequence with a content token 42, an EOS token 2, and two padding positions. If PAD is also 2, masking every occurrence of the pad ID turns [42, 2, 2, 2] into labels [42, -100, -100, -100]. Assigning padding a distinct ID, here 0, lets the same rule preserve EOS: [42, 2, 0, 0] becomes [42, 2, -100, -100]. The -100 labels are ignored by the loss.
Han describes Unsloth’s repair as selecting an available untrained token for padding, or adding a new token when necessary. Adding one requires checking the vocabulary for collisions; his implementation appends extra hash characters until the proposed token string is unique.
A distinct padding ID preserves EOS supervision
Constructed example: Token IDs 42, 2 and 0, the four-position sequence, and the use of ignore label -100 are teaching values, not token IDs or a sequence shown in the recording. The comparison illustrates the described repair, not an executed training result.
Content token 42 → EOS token 2 → padding → padding
Operation: Assign padding the distinct token ID 0 instead of EOS ID 2, then rebuild labels by masking positions whose token ID equals PAD.
Position 1: content
Token 42; label 42
Token 42; label 42
Position 2: actual ending
Token 2; label -100 (ignored)
Token 2; label 2 (supervised)
Position 3: padding
Token 2; label -100 (ignored)
Token 0; label -100 (ignored)
Position 4: padding
Token 2; label -100 (ignored)
Token 0; label -100 (ignored)
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the training template into deployment
The same formatting contract must survive export. For Ollama, the chat template must match the one used during fine-tuning. Han introduces automatic generation of the Ollama Modelfile to avoid reconstructing that template by hand. His two Colab workflows cover an Alpaca dataset and a user-uploaded CSV, both ending in an Ollama deployment.
Three community findings extend the checks into conversion and inference:
- GGUF conversion: Han reports that the affected conversion workflow required CPU conversion rather than GPU conversion. He tentatively attributes the difference to float16 conversion precision. This is a historical workaround for the workflow he describes, not a general prohibition on GPU conversion.
- Duplicate-BOS warnings: A community contribution added a warning in
llama.cppwhen BOS tokens are duplicated. The warning identifies the same training-and-serving mismatch discussed earlier. - System prompts: Some users found that including a system prompt improved results with Llama 3 Instruct. Han suggests trying it, especially when a prompt may have been omitted; he gives no measured comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train sensitive parameters carefully, and offload asynchronously
Han directs readers to the open-source Unsloth repository, where the Start Free Fine-Tune entry opens a corrected Colab notebook. By this point in the presentation, he says the Llama 3 fixes have been pushed to the repository. He also points to Discord support and blog posts covering Gemma, Llama 3, Phi-3 and continued pretraining.
Continued pretraining brings the embedding and output layers back into the optimization problem. Han recommends training the token embeddings and language-model head with learning rates roughly 5–10 times lower than the other trained parameters. The intent is to adapt these layers without updating them as aggressively as the rest of the fine-tune.
Han reports four times longer context with approximately 1–2% slower training completion using Unsloth’s offloaded gradient checkpointing. The talk does not specify the benchmark configuration. He describes moving gradients into system RAM; the accompanying engineering explanation clarifies that the checkpointing mechanism offloads activations. RAM is preferred before disk because disk transfers can make the fine-tune much slower. Even RAM offloading can lose its benefit if transfers block computation: the crucial implementation detail is non-blocking transfers, allowing data movement and GPU work to overlap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose sequence length and memory settings from the data
The live walkthrough opens the Ollama Colab notebook, installs Unsloth and starts with max_seq_length. This setting should reflect the long sequences actually available for training. Setting it to one million or ten million does not create long-context training data; Han’s deliberately oversized example shows why the configuration must match the dataset’s content.
Next comes load_in_4bit. Han describes four-bit loading as reducing memory use by four times and warns that disabling it may push usage to 16 GB in the demonstrated setup. He recommends leaving it enabled on the free Colab Tesla T4 and reserving higher-precision loading for larger GPUs. The talk does not separate weight storage from total training memory, so the fourfold figure should not be used as a total-memory sizing rule.
The model-name field takes a Hugging Face model identifier. Han names Llama, Mistral, Gemma and Phi-3, then encourages trying models beyond the examples listed in the notebook. Compatibility still depends on the architecture and the installed Unsloth version; an identifier by itself does not establish support. The notebook frame shows model-loading settings alongside completed download bars before the walkthrough moves to adapter configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give LoRA enough capacity without exhausting memory
get_peft_model adds the PEFT LoRA adapters. The notebook starts with rank r=16; Han suggests 16, 32, 64 or 128 as practical choices. Powers of two are a preference, not a requirement. Increasing rank gives the adaptation more capacity, but also increases memory use and the risk of overfitting. He gives the model dimension as an upper-bound intuition, using 4,096 as his example, rather than recommending that rank as a starting point.
Han recommends targeting all seven named attention and feed-forward projections rather than adapting only a subset:
| Part of the model | LoRA target modules |
|---|---|
| Attention | q_proj, k_proj, v_proj, o_proj |
| Feed-forward network | gate_proj, up_proj, down_proj |
For lora_alpha, he recommends at least the rank and reports better results with twice the rank, illustrated by r=16 and lora_alpha=32. These settings control different things: rank sets the adapter’s capacity, while alpha participates in scaling its contribution.
One API distinction matters when following the recording. Han describes use_rslora=True as choosing alpha automatically. In current PEFT LoRA documentation, it instead changes the scaling rule; rank and alpha remain configured values.
Finally, the notebook offers Unsloth’s gradient-checkpointing mode for long-context fine-tuning. Han contrasts it with setting the option to ordinary True, which he describes as using more memory in this workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn a CSV row into a repeatable conversation
For data preparation, Han uploads a Titanic CSV. The task is to predict whether a passenger survived from attributes such as age, fare and embarkation location. Those feature columns must become the model’s input, with survival as the output.
The demonstrated notebook expects an instruction/output pair. When a CSV has many feature columns, merge them into one instruction instead of discarding information. For example, this Python function formats a row while keeping the target out of the prompt:
python
def passenger_example(row):
instruction = (
"Predict whether this Titanic passenger survived.\n"
f"Age: {row['Age']}\n"
f"Fare: {row['Fare']}\n"
f"Embarked: {row['Embarked']}"
)
output = "survived" if int(row["Survived"]) == 1 else "died"
return {"instruction": instruction, "output": output}
Han contrasts this with Alpaca-style data, which separates instruction, input and response. Combining the instruction and auxiliary input gives the notebook a single user-side field and one response field. This is the interface of the export workflow he demonstrates, not a general two-column restriction imposed by Ollama or GGUF.
The custom-template step then requires two input/output repetitions. One exchange does not fully specify how the next exchange joins it: trailing newlines and separators can become ambiguous. Han says the second repetition resolves these dangling-newline boundaries. His Llama 3 template example also contains two iterations, and he warns that omitting the second causes this notebook’s template processing to error. The repeated example defines the conversation boundary that must later be reproduced during inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Accumulate gradients, then check the inference prompt
Han’s suggested training configuration uses a per-device batch size of 2 and gradient_accumulation_steps=4. For the single-device notebook, these give an effective batch size of eight:
A larger microbatch requires more batch-related memory at once. Increasing accumulation instead lets multiple forward/backward passes contribute to one optimizer update while retaining the smaller microbatch. Other choices, including sequence length and model configuration, still affect overall memory use.
For the learning rate, Han suggests 2e-4 or smaller, with 2e-5 as another starting value. After training, the notebook runs inference using apply_chat_template. This returns to the first failure mode: the inference prompt must preserve the training format, including exactly one BOS token where required. Han notes that Unsloth handles the duplicate-BOS correction in this path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Export the model and the configuration that makes it usable
The final notebook steps carry the fine-tune into Ollama:
- Install Ollama for the deployment environment.
- Save the model as GGUF, specifying multiple output formats if needed. The notebook can generate multiple GGUF outputs automatically instead of requiring a separate save workflow for each one.
- Inspect the generated
Modelfileand copy it into a custom Ollama setup when appropriate.
Han says the earlier workflow for producing multiple GGUF outputs required an extra ten minutes; he does not specify the model, hardware or formats for that timing. The generated Modelfile is the more consequential artifact: it carries the prompt template and stop parameters into the serving environment. Han identifies its automatic generation as the complicated part of the integration.
The walkthrough ends with Ollama inference. The exported weights now have a corresponding serving configuration, so the model can receive the same conversation structure it learned during fine-tuning. Han points back to the Ollama chat-template notebook in the talk slides as the end-to-end path from data preparation to that final inference step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Unsloth's explanation of eight Gemma issues involving token formatting, numerical precision, normalization and GELU.
Historical Llama 3 fine-tuning results and an explanation of untrained special tokens and their initialization.
The public Unsloth project, including installation guidance and training examples.
Further reading
The April 2024 explanation of activation offloading, with maximum-context estimates and benchmark conditions.
Updates since the talk
Current guidance for formatting conversations and avoiding duplicate special tokens during tokenization.
Configuration details for LoRA, including rank, alpha, trainable modules and rank-stabilized scaling.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hello, everyone. Um, so today I'm gonna talk about how to fix b- bugs in open source models.
- 0:18
Um, thanks for coming again. Um, we had a talk yesterday, the three-hour workshop, um, and thanks for coming again. Um, so we have slides. It's at tinyurl.com/unsloth2. Um, for the workshop slides, which we did yesterday, um, you can also access that now, um, tinyurl.com/unsloth as well.
- 0:38
So you might know me from, like, the Gemma bug fixes that we did. So Gemma was an open source model by Google, and there are, like, a few bugs in there, and we fixed a few of them.
- 0:47
Um, and we just did some tweets about this, and, you know, like, there's many bugs in there, like, you know, the activation function had to be exa- um, approximate GELU, not exact GELU, and there are some other issues that we talked about for Gemma.
- 1:02
We also have, like, some, a few stickers which you can, you know, get them from, like, when we're outside. Um, but this is... Yeah, we won't be handing them during the talk.
- 1:11
Um, but yeah, they're very cool and cute. Um, and also there's, like, tokenization problems as well in language models, which we also helped to fix.
- 1:19
Today, I'm just gonna be talking about Llama 3 bugs. So yesterday I talked about Gemma and Phi-3, and today we're just sh- sharing all the stuff that we found with Llama 3.
- 1:29
For Gemma, you can access, like, all the bug fixes that we did, um, in our blog post, and we have a Colab notebook for all of the Gemma bug fixes as well.
- 1:38
For Phi-3, for example, um, we talked about this yesterday, um, and I just pasted some slides again if you wanna, like, review this in your own time. Um, for example, like, the sliding window should be two thousand and forty-eight, not two thousand and forty-seven.
- 1:52
Um, you can also, like, you should unfuse all of the QKV matrices, otherwise LoRA fine-tuning will not work that well.
- 2:00
But we'll be talking mainly about Llama 3. Um, so there is actually eight bugs in Llama 3. We... Some of them are not announced yet. Um, we will be announcing these later.
- 2:09
Um, so this is, like, a prerelease. Um, and we'll be going through each of them separately.
- 2:15
The first one is you must not use double BOS tokens. Um, so this is actually a very common theme in fine-tuning Llama 3. Some people don't actually know that you're adding two beginning of sentence tokens to the fine-tune, and this will actually ruin your fine-tune, um, by making the accuracy of your inference time lower.
- 2:32
Um, so please, like, check before you fine-tune if you're using double BOS tokens. Um, in Unsloth, we do this, we check this automatically, and we'll remove the extra BOS token automatically for you.
- 2:42
So this will actually cause your model to lose accuracy, because if you trained on two BOS tokens and you do inference on one, then your model template will be incorrect.
- 2:51
Um, so please check this. It's not just a Llama 3 problem. Other models like Mistral and Gemma also have problems like this, so just be careful of this issue.
- 3:01
So a very easy way to check if you have double BOS tokens, if you use the apply chat template, um, uh, from Hugging Face, if you do the first one, your chat template must have a BOS token, otherwise it won't add it.
- 3:13
Um, Llama 3 does require a BOS token. If you do the second one, you're actually having two BOS tokens if you do this, so please do not add a BOS token to the chat template.
- 3:24
The second issue we found was you must not use the Llama 3 base model if you're using the Llama 3 template. Um, there are some untrained tokens in Llama 3 instruc-- um, in Llama 3 base.
- 3:35
The instruct version actually has these tokens trained. So please be careful when you wanna use the Llama 3 base model when you wanna do your fine-tuning, because some of these tokens will cause NaNs for your gradients.
- 3:46
These tokens include the reserve special tokens from zero to two hundred and fifty, the, um, end of, the end of term token, the start header, and the end of header.
- 3:55
Um, and the graph I showed shows you the, uh, mean of the embeddings versus the other tokens, and some of them actually are zero. So the Llama 3 team made some of these to- tokens go to zero purposefully because these tokens are not actually used for the model.
- 4:09
Um, so just please don't use these, some of these tokens when you do fine-tuning as well. Um, if you wanna fix them, set them to the mean of the entire tokens.
- 4:19
Um, in Unsloth, we do this automatically as well for you. Um, so we showed some code where you can take the mean of the un-- um, trained tokens and set them for the untrained tokens.
- 4:29
Just be careful. Don't do this, like, incorrectly as well. If you wanna take the average of all the tokens, don't just take the average. You must remove the untrained tokens from the average.
- 4:38
If you do not do that, you might actually have an incorrect average, right? If there's, like, ten thousand tokens which are untrained, if you divide it by ten thousand plus the number of trained tokens, your average will be incorrect.
- 4:49
So, um, you have to do this more complicated method of masking out the untrained tokens and then take the average. Also, reminder, because of this issue, the Llama 3 chat template will not work for the base model.
- 5:02
I have known, like, many fine-tuning people have used the Llama 3 instruct, um, chat template for the base model, and your fine-tune will actually be incorrect. Um, you will get NaNs in your gradients, and your whole fine-tune will be broken.
- 5:14
So please do not use the Llama 3 instruct chat template for the Llama 3 base model. Only use this for the instruct model itself. Another way to fix this is to actually train the LLM head and the embed tokens, which will actually learn and remove the NaNs in your models.
- 5:31
Another interesting fact, and not just a Llama 3 problem for, but for other models, is the pad token and the EOS token, uh, must not be the same. If you do this the same, your model will have infinite generations.
- 5:44
Um, the reason is because the pad token gets masked out during the era, um, during the cross-entropy loss, and if you use the same t- pad token as the EOS token, then your EOS token, the end of sentence token, will be masked out.
- 5:55
So just be very, very careful when you do fine-tuning to check what is the pad token ID and the EOS token ID. For example, if you look at Phi-3-
- 6:04
They're the same. So technically, Phi-3, when you do fine-tuning, it will be infinite generations. So just be careful and look, you know, before you do have to fine-tune, check what is the EOS token and what is the pad token.
- 6:16
They must be different. For Unsloth, we also do this automatically. We fix this for you, um, and we essentially check if there is any unreserved to-- uh, if there is any unreserved tokens, and we just select one which is untrained.
- 6:30
If there is no untrained tokens, then we will add an extra pad token ourselves. Um, be careful. Do not add a pad token which has the same, like, vocabulary as your current vocabulary.
- 6:40
So what we do is we actually check the tokens inside the vocabulary and add, like, extra hashes to see, you know, to make a new pad token.
- 6:50
Another issue we found for fine-tuning, people, is, like, when you finish a fine-tune, you don't actually know how to export it to Ollama, and that is because the chat template for Ollama must be exactly the same as your fine-tune.
- 7:01
Um, and this was actually very complicated to do before, and now we can actually automatically generate the model file for you during the fine-tune. So we have, like, two Colab notebooks for you to use for Ollama.
- 7:12
Um, one of them's the Alpaca dataset, and one of them's a-- You can upload a CSV file to make, um, Ollama work after you finish fine-tuning.
- 7:22
Now, there are some community contributions for Llama 3, uh, bugs. Um, there is like thr- uh, three of them. The first one is someone noticed that you can only use CPU conversion and not GPU conversion when you convert to GGUF or llama.cpp.
- 7:35
Um, so be, you know, be careful when you convert to llama.cpp that you must use the CPU version. Um, I think the main reason is because the precision is different in a GPU than a CPU.
- 7:45
The CPU, when you do float16, it's different from when the GPU does float16 conversion. So just be careful on that as well.
- 7:54
Another issue is, remember we talked about the double BOS tokens? Um, through a community contribution, the, um, llama.cpp now has a warning for you to tell you that you're using w-- BOS tokens.
- 8:05
So please, you know, take heed of the warning and do not add double BOS tokens to your chat template and when you do inference.
- 8:13
Another point someone found was adding a system prompt could make fine-tuning much better. Um, and so, like, sometimes when you in-- do inference on Llama 3 Instruct, if you add an actual system prompt, this could make your whole fine-tuning better.
- 8:27
Um, I think for some people, when they add the system prompt, you actually miss the system prompt, like you don't actually add one. And so maybe try your fine-tune with the system prompt, and you never know, this could work.
- 8:38
So we have like a GitHub package, um, which is open source, um, and you can click the button Start Free Fine-Tune to start your first free fine-tune using Unsloth.
- 8:46
We already pushed all the Llama 3 bug fixes to our GitHub repo, and so the Start Free Fine-Tune button will redirect you to a fixed Colab notebook for all of these issues.
- 8:56
Um, feel free to star us as well. We also, like, have a Discord channel, so if you have any questions, um, you can ask, you know, any question that you like about our, you know, how to do fine-tuning, talk about AI, and talk about our bugs as well.
- 9:10
We also have like a blog post, so blog posts about all our fixes, about Gemma, b-- um, Llama 3, Phi-3, and more. Um, for example, we talked about continued pre-training.
- 9:19
You can do continued pre-training using Unsloth now. We-- You can train on the LLM head and the embed tokens, and we show that instead of just training like that, you need to reduce the learning rate of the LLM head and the embed tokens by ten or, you know, maybe five to ten, and this will make your training
- 9:34
much better. We also support four times longer context using Unsloth, um, and this also does not increase the time c-- uh, the time of completion. So we make it, uh, one to two percent slower, but you get four times longer context using Unsloth.
- 9:49
Um, and this, uh, this was because, like, we use something called, um, offloading gradient checkpointing, where we offload the gradients to system RAM. There are some other systems which offload the gradients to the disk.
- 10:00
Please do not do that. If you offload to disk, then your sys-- your time of completion of your fine-tune will be extremely slow. Um, so try to offload to system RAM first, and then offload to disk.
- 10:10
Although if you don't... If you offload incorrectly, you might actually make this slower as well. So your offloading must be non-blocking calls, and do not do blocking calls to the system RAM.
- 10:21
Yeah, so I will show you, um... Okay, let's see if I can open up, um... Let me go to...
- 10:34
Does this work? Okay, I'm gonna open up a Colab notebook, um, for the Ollama one. Um, yes.
- 10:48
Okay. So for the Ollama Colab notebook, you can simply just install Unsloth over here. Um, this is already for free for everyone to use. Um, and essentially, you don't forget when you do the Colab notebook, you have to select a max sequence length.
- 11:02
Um, this de- this determines how long your model wants to do long context fine-tuning. Um, you can set this to any number that you like, but remember, your dataset must match the max sequence length.
- 11:12
So for example, if you have-- If you wanna set the max sequence length to be like ten million or one million, um, but your dataset is only like one million tokens or like less, try to like not set that max sequence length to be that large.
- 11:23
Otherwise, your model cannot do fine-tuning on long sequence. Um, load in four bit does four bit training, so this actually reduces memory usage by four times. Um, if you do, if you do it to false, your memory usage will explode, so please do not try false, um, especially on a free Colab Tesla T4.
- 11:40
Um, if you do false, your memory usage might skyrocket to sixteen GB, so do not do that. Um, you only should do this if you use more stronger GPUs.
- 11:50
We support, like Unsloth supports fine-tuning all models, including like Llama, Mistral, Gemma, Phi-3, and more. Um, so this area, like the model name over here, you can actually s-- try to select any model name that you like.
- 12:02
Um, I don't think that people know that Unsloth can support other models other than the ones we listed. So please try to put any model, like a Hugging Face model name, in there, um, and it should work.
- 12:15
So for the get_peft model, this is where you add the Peft LoRA adapters.
- 12:20
The R is the rank, so we set it to be 16, but you can select any number that you like for fine-tuning. Um, so we suggest you normally to use powers of two, but you can use any number like one, two, three, like any number that you like.
- 12:31
The larger the rank you select, you can make the model learn more about your data set. So... But if you add too large of a rank, you might actually overfit your data set, and also your memory usage might skyrocket again.
- 12:41
So we normally suggest people to select sixteen, thirty-two, sixty-four, or one hundred and twenty-eight. Try not to select too large ranks. Um, the maximum rank you should select is the size of the dimension of the, uh, model itself.
- 12:52
So if it's four thousand and ninety-six, set this to be four thousand and ninety-six.
- 12:57
For the target modules, be careful. Um, you must do fine-tuning on all linear layers, right? So QKVO down, up, and, uh, gate. Some people have done fine-tuning without doing some of these layers.
- 13:09
Please do not do that because this will cause your fine-tune to be not optimal. And the LoRA alpha, there is actually a trick for this. Um, normally speaking, select the alpha to be the same as the rank or larger.
- 13:20
We found that if you do sixteen times two, so the rank times two, this can, this can make your fine-tuning much better. You can also use use_rs_lora to be True to, to set the rank automatically for you, uh, to set the alpha automatically for you.
- 13:33
For the gradient checkpointing, Unsloth is the method which we showed that you can do long context fine-tuning. You can also set this to be True, um, but your memory usage will increase again.
- 13:44
We also show you how to do data preparation in a co- Ollama Colab notebook. So this one is, we upload a Titanic CSV. So the Titanic data set, the goal was, can you predict if someone died or survived, um, if you're on the Titanic?
- 13:56
And you get details about the person, for example, their age, their, like, fare, where did they embark from, and so on.
- 14:05
With our new Colab notebooks, you have to be very careful when you do, um, Ollama chat templates, um, because when you do fine-tuning, you can only have two columns, the instruction and the output.
- 14:14
But what happens if your CSV has more than one-- like more than two columns, the instruction and output? What we can do is you can merge the columns into one column, and with Unsloth now, you can actually do that.
- 14:23
You can merge the columns into one. And also, we show you that you can do customizable chat templates now. So previously, if you want to do an Alpaca-style fine-tune, you have to use instruction, input, and response for the Alpaca-style fine-tuning.
- 14:39
But remember, the problem is if you wanna output to Ollama or GGUF, you can only have two columns, the instruction and output, right? If you do ChatGPT, you c- you have to type something, and then the output comes along.
- 14:50
You can't have, like, three inputs, right? So, so what we do is you can actually customize your tap- chat template, and you must include the input and the output, and you must do this repetition twice.
- 15:01
Um, some people have asked me, like, "Why do you have to do, uh, two repetitions of this chat template?" It's because there is dangling new lines, um, and we found this-- we found a solution to this, is you have to specify two iterations of your ta- chat template.
- 15:16
We also show examples of how to do the Llama 3 chat template using our methodology. Um, so you can see there is two iterations of the chat template.
- 15:24
Reminder, if you don't use the two iterations, you actually-- it will error out.
- 15:29
And this is the training methodologies. We normally suggest people to use a batch size of two, gradient accumulation of four. Remember, the memory usage is only relevant to the batch size, so try not to set the batch size to be very large, otherwise your memory usage will explode.
- 15:42
Instead, set your gradient accumulation steps to be larger. Um, so the formula for the effective batch size is batch size times the gradient accumulation. So in this case, it's two times four, which is eight.
- 15:53
Set your learning rate to be two e minus four or smaller, maybe two e minus five.
- 16:00
And after that, you can also do inference on the model. Um, so now you have to use the apply chat template. Remember, be careful of double BOS tokens, but we in Unsloth fixed this.
- 16:11
And finally, you have to save this to Ollama, um, and, you know, you have to install Ollama first. Um, saving now, we now support saving multiple GGUF files, so you don't actually have to save it to one GGUF file.
- 16:22
You can save it to multiple, and we actually allow you to do this now. Um, before, if you want to save to multiple GGUF files, you have to wait ten minutes extra.
- 16:29
You can now do this automatically by, you know, specifying more than one format.
- 16:36
We also can show you the model file which we created, so you can actually copy-paste the model file and put this to custom, like a custom Ollama as well.
- 16:43
Um, so the model file was the complicated part when we had to automatically generate this. So we have, like, internal code to generate the model file automatically.
- 16:52
And finally, when you wanna do inference, you can do Lo- Llama to do inference. Um, and you know, it works, um, in general. So try that out. Uh, the Ollama chat template notebook is in the slides, so tinyurl.com/unsloth2.
- 17:05
Um, and remember the workshop slides which we did yesterday, um, is tinyurl.com/unsloth. Um, and don't forget to join our Discord channel. Um, if you have any questions, I'm outside.
- 17:15
You can ask questions and stuff like that. Um, and yes, like, thanks for the, uh, thanks for coming and much appreciated. Thanks a lot. [outro music]