AI Engineer Summit 2023
Domain adaptation and fine-tuning for domain-specific LLMs
Read the talk
Domain adaptation: what to change, what to freeze, and what to measure
Adapting a language model requires more than choosing a fine-tuning method: the domain, training data, memory budget, evaluation, and operating pipeline all shape the result.
From a talk by Abi Aryan
Before you start: Familiarity with neural-network weights, attention, and the distinction between training and inference will help with the adaptation mechanisms.
When a general model does not know your domain
How do you make a general-purpose language model useful in a domain that its training data barely represents? Some specialist knowledge is scarce; some cannot enter public training corpora because of compliance restrictions. Abi Aryan begins with this practical gap: a broadly capable model does not automatically cover every enterprise or hobby use case. Fine-tuning is one response, but knowledge bases, retrieval augmented generation (RAG), and prompting also belong in the decision.
A second motivation is avoiding a fresh data-collection effort for every domain. Aryan uses language transfer to illustrate the attraction of reusable representations: embeddings can capture structure that helps a model transfer between related settings. She tentatively offers English and Latin as an example of similarity. That analogy motivates transfer, but shared structure alone does not establish that a model can reliably handle an entirely unseen language without examples. The broader goal is access: adaptation can also personalize a model for users whose needs the general model does not adequately serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Locate the parameters before choosing an update
Fine-tuning changes learned parameters to improve a pretrained model for a new requirement. The initial neural-network picture is familiar: inputs pass through hidden layers, whose weights and biases determine the outputs. Aryan then moves to an encoder–decoder transformer diagram, with attention and feed-forward networks on both sides. This is the architectural reference for the methods that follow, rather than a claim that every language model has both an encoder and a decoder. The useful question is where adaptation enters that computation and which parameters remain fixed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full updates versus selective adaptation
The first choice is whether to update all model weights or a much smaller set. Aryan connects full updates to earlier transfer-learning and teacher–student distillation work from the 2016–2018 period: a student learns from a teacher, with training changing the student's parameters. Distillation describes the source of supervision, however, rather than requiring a particular parameter-update scope. The practical pressure is compute and storage. As models grow, repeatedly training and storing complete task-specific copies becomes expensive. Selective adaptation aims to retain useful performance with less trainable state; matching ChatGPT is not a guaranteed consequence.
The talk organizes the next methods as adapter tuning, prefix tuning, and parameter-efficient tuning. More precisely, parameter-efficient fine-tuning (PEFT) is the umbrella that includes adapters and learned prefixes as well as low-rank updates. Aryan also mentions instruction tuning through examples and sets aside reinforcement learning from human feedback because collecting human judgments is expensive. Examples used for instruction tuning become training data; examples placed only in a prompt do not update weights. That distinction separates training from in-context learning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add small adapter modules
In adapter tuning, small trainable components are inserted into the existing network. Aryan's illustration adds two components containing extra weights while leaving the original transformer intact. These modules provide a place to learn domain-specific changes without updating the entire pretrained model. This is the central mechanism of adapter-based parameter-efficient transfer learning.
Aryan reports matching model performance with only 0.15% of the parameters, but does not specify the task, benchmark, or parameter denominator. Treat that number as an attributed result, not a capacity-planning target. Her application example is biochemical engineering: a substantially different domain whose terminology and patterns may require adaptation. She uses “adapter” and “adaptive” fine-tuning interchangeably here; in implementation, keep the distinction between an adapter module, which is a mechanism, and domain adaptation, which is a goal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Steer attention with a learned prefix
Prefix tuning keeps the base model frozen and learns continuous prefix vectors that subsequent tokens can attend to. Aryan describes this as adding an embedding layer in front of attention. Unlike an ordinary text instruction, the prefix consists of trainable values: optimization changes those values while preserving the pretrained parameters.
The analogy is a water tank and a tap. The tank supplies the water; the tap shapes how it comes out. Similarly, a learned prefix steers how the model uses its existing capabilities. Aryan also calls this a masking layer, but the literal mechanism is conditioning attention on learned vectors, not masking or replacing the base weights. The model's output behavior changes even though its pretrained parameters do not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn a low-rank update, not a smaller base model
LoRA, or Low-Rank Adaptation, and QLoRA enter the discussion through resource constraints. Aryan points to laptops and even Arduino-class devices as motivations for efficiency. Those examples express the desire to work with limited resources; reducing fine-tuning memory does not by itself make a large language model deployable on a microcontroller. Training memory, adapter storage, and inference requirements are separate budgets.
To explain rank, Aryan asks the reader to imagine matrix rows or columns that repeat information. If one is a scalar multiple of another, it does not add an independent direction. That intuition helps explain why a change to a large matrix might be represented with fewer degrees of freedom. LoRA applies that low-rank constraint to the learned update. It does not inspect the pretrained matrix and delete redundant rows or columns. The base weight matrix stays frozen, while two smaller matrices are trained.
For a base matrix W₀ with output dimension d and input dimension k, the forward computation is:
Here, r is the chosen rank and α controls the update's scale. The following PyTorch module makes the frozen base and trainable update explicit:
python
import torch
from torch import nn
class LoRALinear(nn.Module):
def __init__(self, base: nn.Linear, rank: int, alpha: float):
super().__init__()
if rank <= 0:
raise ValueError("rank must be positive")
self.base = base.requires_grad_(False)
self.scale = alpha / rank
self.A = nn.Parameter(
base.weight.new_empty(rank, base.in_features)
)
self.B = nn.Parameter(
base.weight.new_zeros(base.out_features, rank)
)
nn.init.normal_(self.A, std=0.02)
def forward(self, x: torch.Tensor) -> torch.Tensor:
update = (x @ self.A.T) @ self.B.T
return self.base(x) + self.scale * update
Initializing B to zero makes the initial update zero, so the layer initially preserves the base output. Training changes A and B, not base.weight. This reduces trainable parameters and their associated gradient and optimizer state; it does not automatically shrink the frozen base model. That is the precise memory benefit behind the talk's broader model-size discussion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quantize the frozen weights with QLoRA
QLoRA adds another memory-saving choice: the frozen base weights use four-bit storage, and gradients pass through that base into trainable LoRA adapters. Aryan describes a workflow of starting with a pretrained model, collecting labeled target data, and training adaptation matrices. The low-rank matrices multiply each other to form an update whose contribution is added to the base computation; they are not simply multiplied into the main weight matrix. Likewise, reducing a distance between source-domain and target-domain predictions is her description of the adaptation goal, not the defining QLoRA objective. The training loss still depends on the task and dataset.
The implementation tradeoff is support. Aryan flags dependencies such as bitsandbytes, warns that device availability and cross-model efficiency testing are limited, and favors optimizing LoRA at the time of this October 2023 talk. That is historical implementation advice: current library support covers additional hardware configurations, so today's choice requires checking the actual backend and device rather than carrying forward a blanket preference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompting, retrieval, and training can work together
The discussion now returns to the larger choice among prompting, RAG, and fine-tuning. Prompting can provide no demonstrations, one demonstration, or several: zero-shot, one-shot, and few-shot use. Aryan suggests about ten few-shot examples, citing ChatGPT rather than a task-specific experiment. She presents additional examples as helpful and prompting as cheaper and less data-intensive, while characterizing it as more suited to general demonstrations than demanding applications. Neither a fixed example count nor a blanket performance ranking against fine-tuning follows from that discussion; the examples and task determine whether the extra context helps.
These are composable choices. Fine-tuning can shape a model's behavior, prompting can specify the current request, and retrieval can supply relevant information at inference time. Aryan explicitly allows prompt engineering and RAG alongside fine-tuning, as well as combining domain-focused and behavior-focused training. Choosing one does not require abandoning the others.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the domain, the task, and the update budget
Consider Aryan's legal-company example. A company may want a model that handles five or ten tasks across legal work, or it may want to optimize one specific task. She calls the first goal adaptive fine-tuning and the second behavioral fine-tuning. For the latter, the focus narrows to the task's label space and prior probability distribution: what outputs are possible, and how frequently they occur. She suggests inference and reasoning as potential uses and offers LangChain functions as an analogy for task-focused behavior, rather than as a training mechanism.
| Scope | Question it answers | Example or mechanism |
|---|---|---|
| Domain adaptation | What subject area should improve? | Several tasks within legal work |
| Behavioral tuning | What specific behavior should improve? | One task and its output space |
| Parameter efficiency | How much trainable state can we afford? | Freeze the base; train a small update |
The first two rows describe the goal; the third describes how to pursue it. LoRA or QLoRA can therefore support either domain-focused or task-focused training. They are not competing objectives on the same axis.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The adaptation method depends on the data
Once the method is chosen, data quality becomes the limiting factor. Aryan's preparation sequence covers collection, tokenization, cleaning, normalization, and noise removal. These choices determine what information the model actually receives—not merely whether the training job can load the dataset.
Deduplication is especially consequential. Repeated records expose the model to the same material again and again, increasing its influence on training and encouraging memorized outputs. Aryan attributes much memorization to duplication; the narrower supported conclusion is that duplication contributes to it, not that removing duplicates eliminates memorization. After deduplication, she adds data augmentation to the preparation toolkit. Augmentation should expand useful variation rather than recreate the repeated exposure that cleaning was meant to reduce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialization does not guarantee broad generalization
Even with a suitable base model and carefully prepared data, specialization has limits. Aryan warns against expecting a fine-tuned model to match GPT-4 on broad, complex tasks merely because it learned the nuances of one dataset. Her reference to GPT-5 also occurs in this 2023 discussion, not as a measured comparison with that later model. The actionable distinction is between performance inside the target domain and generalization to new data or entirely different domains.
For changing inputs, Aryan proposes in-context learning with dynamically loaded examples. The examples can change with the request or current data, providing an inference-time way to address drift while managing costs. She then recommends decomposing broad objectives into smaller tasks: instead of attempting to train for an entire language at once, identify the particular operations the application needs. A narrower task makes both adaptation and evaluation more concrete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trade recomputation for activation memory
Gradient checkpointing addresses a different part of the training-memory budget. Normally, training retains intermediate activations from the forward pass so the backward pass can calculate gradients. Checkpointing retains fewer intermediates and recomputes the missing ones when needed. Aryan describes this as recomputing weights or retraining; the objects being recomputed are intermediate activations, not a second set of learned weights. The result is additional forward computation in exchange for less activation storage. Whether that trade is worthwhile depends on whether memory or computation is the tighter constraint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Treat training settings as hypotheses
Aryan suggests batch sizes of 32 or 64 without specifying a model, dataset, or memory budget. She suggests one epoch for a casual Google Colab test and 100 epochs as a starting point for domain optimization. These are the talk's proposed settings, not validated defaults: a useful training budget depends on dataset size, repeated exposure, and held-out performance. A smoke test and a domain-training run answer different questions, so success at the former does not establish the duration of the latter.
Adam is her general-purpose optimizer recommendation. She then turns to regularization and early stopping, arguing that increasing parameter counts can distract from extracting better performance from existing models. Early stopping means monitoring a validation criterion and stopping when further training no longer improves it. If performance declines, Aryan recommends inspecting the batches and embeddings instead of assuming that more training or a larger model is the remedy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate from four perspectives
After fine-tuning comes the harder question: did the application improve? Aryan mentions Aviary by Ray and NVIDIA libraries among the available tooling, but emphasizes that loss, accuracy, and perplexity do not provide a complete picture. Evaluation must reflect the business and use case. The historical Aviary repository now leads to the archived RayLLM repository, so that reference should not be treated as a recommendation for a currently maintained evaluation service.
- Metric-based evaluation: BLEU and ROUGE provide quantitative comparisons against reference outputs. They offer one perspective on quality rather than a complete judgment of usefulness.
- Tool-based evaluation: Aryan mentions Weights & Biases debugging tools and an Auto Evaluator, associating automated checks with catching compilation errors. The precise products and integrations are not specified, so the actionable category is automated detection of concrete failures.
- Model-based evaluation: Another model, potentially a smaller one, judges the model being tested. Aryan sees potential for automation and standardization but reports limited success in her own experience.
- Human-in-the-loop evaluation: Human review remains a common option, although Aryan considers it inefficient and does not explore it further in this session.
These perspectives expose different kinds of failure. A strong aggregate metric cannot substitute for checking whether the system produces a valid, useful result for its intended users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The unit of reliability is the whole pipeline
Fine-tuning is only one part of the operating system around a model. Data collection, storage management, and base-model selection constrain what training can achieve. The closing pipeline diagram extends that scope through pretraining, fine-tuning and prompt engineering, evaluation, orchestration, deployment and monitoring, and security and reliability engineering. A one-off demonstration can succeed while leaving several of those responsibilities unresolved.
A durable application must keep working as its data and prompts change. Aryan does not prescribe a fixed testing duration; she focuses instead on detecting problems early enough to avoid exposing an enterprise to reputational or compliance risk. The adaptation decision therefore includes how the system will be monitored, how drift will be recognized, and how failures will be caught—not just which parameters the next training run will update.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Learned continuous prefixes steer frozen language models for generation and summarization.
The original method for adapting frozen models with trainable low-rank matrices.
Four-bit frozen model weights combined with LoRA adapters to reduce fine-tuning memory.
Further reading
Original BERT adapter research, with task-specific parameter and GLUE performance results.
Experiments connecting training-data duplication, memorized output and evaluation contamination.
Foundational research on trading extra forward computation for lower training memory.
Updates since the talk
Quantization library with installation guidance and a current hardware support matrix.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] Hi, everyone.
- 0:16
Welcome to my presentation. So I don't know what's the best way to start about it. I would probably say something along the lines of, well, you've heard a lot of really good presentation that have focused on one very specific thing.
- 0:32
In this session itself, we'll more so focus on an overview of domain adaptation and fine-tuning for large language models. Because there's so much information out there which is like, "Oh, take this, use that."
- 0:47
So my goal is to sum up all the literature for you to be able to make an informed decision on how to be able to do domain adaptation for your particular enterprise use case or for your hobby use case, however you're using it for.
- 1:02
So about me, uh, this has already [chuckles] been spoken, so let's, let's skip this. Um, why do we care? I think, I think the answer to this is pretty obvious, which is, I mean, there are-- ChatGPT as a model, or even if you're looking at open source large language models, they're not trained for every single use case out
- 1:23
there. There are some domains that are underrepresented. There are some domains for which there is not enough data because of compliance or for whatever reasons. And for that, we need to be able to have some sort of method to be able to fine-tune the models or use some other strategy and alternative of fine-tuning, whether that's knowledge bases,
- 1:44
whether that's RAGs, or whether that's prompting. The second is basically you don't want to collect new data for every single domain. One of the best things that has happened with large language models, I would say, is the ability of these models to be able to transition to a new domain.
- 2:02
So there's one paper that I would reference. Uh, so one quick example I would say is before, let's say, transformer models, uh, or even while we were having transformer models, um, to be able to train a model to learn a new language, we needed to collect the data for that particular language, and then
- 2:25
to be able to, um, do whatever task that we want to do in that language. One of the best things that has happened is now, because the models are learning via embeddings, they're able to learn on a new language that they've previously not seen as well, because they're essentially learning the structure of the languages instead of like
- 2:46
what is the taxonomy of the language. Which means there are some languages which are semantically similar. So for example, English is very semantically similar to Latin. I'm not entirely sure there are a couple of languages that do fall into like that one domain, which is, oh, these languages are similar.
- 3:05
They have semantic similarities. There are other set of languages that have semantic similarities. So it's very easy to be able to transition between those languages without ever having seen any data or any examples in those languages.
- 3:18
The third is basically you want the models to be able to be accessible to a wide range of users. And what I mean by that is more so like all the work that was happening along in personalization.
- 3:32
So simple reasons. This is something alr- almost everybody is aware of. What is fine-tuning? Fine-tuning is almost a way of us,
- 3:43
um, teaching the model to be able to learn something for which it hasn't already been trained before, so improving the performance of a pre-trained model. Um, one of the ways we're doing that is by updating the parameters, right?
- 4:02
You take some input, you have a hidden layer, um, in which you're calculating the weights, you're calculating the biases, and then you have an output layer. That all stuff I think is obvious to almost everybody.
- 4:14
You've seen what a transformer model. For people who don't know what the structure of transformer model is, there's an encoder, there's a decoder. The reason I'm referencing this is we'll go a little bit more into details of these while we are talking about different fine-tuning methods itself.
- 4:30
So there's an encoder, there's a decoder. It has, uh, s- a feed forward network. It has an attention network. Same for the decoder one. Now, this is, this is how we were looking at transformer models they are-- the way they are, and this is storing the weights and the biases right now.
- 4:51
But now let's talk about making these models better. So there are a couple of ways that we can fine-tune our methods. We can update all the model weights, or we can update some of the weights.
- 5:05
If we update all the model weights, that falls into the category of some of the models that you've seen earlier, which is the all the research work that was between two thousand and eighteen, two thousand sixteen, all those, all those years, which is more around transfer learning cross distillation models in which you have a teacher model and
- 5:23
a student model. The student model is learning from the teacher model, and that's, that's the way you're sort of updating all the weights. But it is very expensive to do that, and it is computationally--
- 5:36
It takes more storage as well. The second option that we've-- we're now looking at, the reason we are having this discussion today is f- how can we update our models?
- 5:48
Because the parameters have gone s- so big, we cannot keep updating [chuckles] all the weights. So how about we update just some of the weights without making sure, uh, while making sure that we're able to get equivalent performance and-
- 6:05
I would, I would put an asterisk on, you know, like equivalent performance because we may not be able to get the ChatGPT performance, and that is something we'll talk about eventually.
- 6:16
So in terms of if we update some of the weights, you can break it down into three categories. To be honest, more like five categories, but there are three main ones, which is adapter tuning, uh, there's prefix tuning, and there's parameter-efficient tuning.
- 6:30
There's instruction tuning, which is basically giving a couple of examples. Um, this is something you've seen a lot at couple of-- so many examples throughout this conference, and the one that was prior to my talk as well, uh, where we were doing instruction tuning.
- 6:45
RLHF, obviously not super relevant to most of us, which is we would-- it's too expensive to have real human beings to be able to fine-tune your parameters for you, um, or to be able to provide your examples and say, "This is wrong.
- 7:01
This is right." So we are only left with three techniques, which is adapter tuning, prefix tuning, and fine-- uh, parameter-efficient fine-tuning. We'll go a little bit more into detail of what these are, why are we using these ones,
- 7:17
and when do they do well. So the first one, this is basically adapter-based tuning. The, the thing that really happens in adapter-based tuning is it's really good. Uh, what it does is it adds a small number of parameters to the existing model.
- 7:34
Uh, those parameters are basically stored in the adapter components that you're seeing over there. This is the entire model of the transformer remains the same, but we are adding two new components to it that contains the extra weights.
- 7:49
So what this does is this exposes the model to the new information, and according to, according to the original paper that came out, you know, it is able to improve the performance of the model.
- 8:02
Or you could say it matches the performance of the model with only zero point fifteen percent of the parameters. Where is it good? Where-- In which cases would we use something like this?
- 8:14
So adapter fine-tuning or adaptive fine-tuning, both are the same things. Um, ideally, you use it when you're trying to learn a new domain itself, which is if you're trying to fine-tune your model for like a very different domain, let's say biochemical engineering, that's, that's more so where you would use, uh, adapter-based fine-tuning.
- 8:36
The second is prefix-based fine-tuning. So prefix-based fine-tuning, what it does is it introduces some prefixes where we are storing the model weights, and what they're able to do is they're able to mimic the behavior of the prefix that we are giving it, which is the couple of weights that we are adding in front of that tension model.
- 8:58
Um, so in a very simple-- in very simple words, what it does, it, it, um, it adds an embedding layer at the front of the attention layer, uh, to mimic that behavior.
- 9:11
One very simple example to understand this a little bit better is, you know, all of the water that we get comes out of a tank, right? But the way we are able to access it is using a tap, and water takes the form of a tap, which is it comes out in, in this quantity.
- 9:30
So that's, that's very much like how prefix tuning works, which is it's not changing the behavior of the model, but it's just mimicking or adding a masking layer on top of, uh, the existing weights or on top of the existing model that there is.
- 9:46
The third and the final one, which is the prefix ba-- uh, which is the parameter-efficient fine-tuning method. So this one, the one example that you're seeing is basically the LoRA one.
- 9:57
There are two commonly known parameter-efficient fine-tuning methods that are out there, LoRA and QLoRA. The way LoRA really works, uh, what it really is, is basically Low-Rank Adaptation method.
- 10:11
Where you use it, any sort of parameter-efficient fine-tuning method is sort of used where you want to compress the model sizes, or you want to run it on low-resource devices.
- 10:22
So very ideal for large language models. Biggest reason is because we have massive parameters that we are trying to run on very small devices, which could be our laptops and even smaller devices, which is basically the HTML devices, the Arduino, and all of that stuff.
- 10:39
So that's one reason the entire community has been talking more so about LoRA and QLoRA because, again, we are looking for efficiency. The way it works under the hood is, um, all of the weights are usually stored as what is basically a matrix, right?
- 10:56
So most of these weights, um, there, there are a lot of layers in these weights that aren't, uh, unique. And what I would-- what LoRA usually does is it identifies the linearly independent layers, um, in, in terms of the weight, uh, matrix itself.
- 11:18
So in the matrix, you're looking at all the linearly independent lines or the columns, and you're picking and choosing only those ones. So what it does is if two things are very similar or two things are almost like you could transform one easily through a mathematical function as like a multiplier of the other one, then storing that
- 11:40
one extra layer, which is a copy of the original one, doesn't really make sense, right? So that's, that's how LoRA works under the hood, which is we are reducing the size of the matrix, which is the size of the weight matrix, essentially.
- 11:54
Practical benefits, obviously, um, you know, you're, you're able to decrease the size of the model, and you're also-- you can also, um- You can also-- You're, you're also using less memory right now.
- 12:11
Um, the second method that we are looking at is basically called QLoRA, which is quantized, uh, LoRA method. The way it works is it changes the model weights to four-bit precision.
- 12:25
Um, the way it usually works is you start with the pre-trained models. Uh, you collect a dataset with labeled, um, with labeled data, and you train adaptation matrix, uh, and multiply it with the main weight matrix.
- 12:42
And what you're essentially trying to do is you're trying to decrease the distance between the predicted outputs of the source domain and the target domain. That's, that's what's essentially going on in QLoRA.
- 12:56
One quick comparison, uh, obviously, I mean, w- w- in, in terms of like the people who are saying, "Okay, QLoRA is great. Should we use LoRA, QLoRA?" Um,
- 13:08
one quick thing I'll say on that one is while QLoRA works really good on the original dataset that it was trained on, but to be able to get it to perform really well requires a library, bitsandbytes libraries, and some other things which are not available on all the devices.
- 13:26
Not a lot of testing has really happened for QLoRA's efficiency on all of the models. So I would probably say maybe still sticking with LoRA and being able to optimize the performance with the LoRA model is ideally like the better way to go, at least at, uh, this current point in time.
- 13:43
So to, to very quickly summarize, um, which is, again, we have three different methods to be able to do d- domain adaptation. We have prompting, we have RAGs, we have fine-tuning.
- 13:57
For prompting, you can sort of prompt your models, again, with no examples, with one example, with a couple of examples. When it comes to a couple of examples, I think a good answer would be about ten, which is what ChatGPT says, where obviously the performance is better the more examples you're able to give it.
- 14:15
Where it works is, um, in, in the domains that you're looking for more generalizable models, but usually that's just demos, not real-world examples. Requires less training data. It's cheaper obviously, but is not as performant as fine-tuning.
- 14:30
On fine-tuning, you're looking at three different methods, which is like adaptive fine-tuning, you're looking at behavioral fine-tuning, and parameter-efficient fine-tuning. Um, on each of these ones, um, you don't need to pick one of these three techniques.
- 14:46
You can also combine them with prompt engineering. You can combine them with RAGs as well, or you can do both of those things, which is you can do adaptive as well as behavioral fine-tuning.
- 14:56
The key difference between all those three methods is adaptive fine-tuning really works well on when you have a target domain that you're trying to optimize for. So for example, like if you have multiple tasks within a single domain, let's say you are, um, you have a legal company, and you're trying to build a model that works really
- 15:16
well on five different or ten different tasks within just the legal domain itself, adaptive fine-tuning works great. Behavioral fine-tuning is basically where you're trying to optimize the model performance on a target task only.
- 15:29
So you're not really optimizing for the entire domain, you're optimizing for just one particular task. The way it really works is you're optimizing for the label space and the prior probability distribution.
- 15:43
So very helpful when you're trying to, um, get to, uh, show some sort of like inference and reasoning capabilities. You could also-- A good analogy on behavioral fine-tuning is it's very similar to LangChain functions, um, if, if you've used LangChain functions.
- 16:02
And parameter-efficient fine-tuning is like the standard fine-tuning where we are freezing some of the parameters, and we're only updating a very small amount of parameters using the techniques LoRA, QLoRA, and so on.
- 16:17
But coming to, you know, are these techniques going to really work? Sure, we have all of this available. It would only work depending on how good your data is, which is how-- It depends on how you're collecting your data, how you're tokenizing your data, how you're cleaning and normalizing your data.
- 16:37
Are you removing the noise and sort of sanitizing your models? Are you doing data duplication as well to be able to, um, remove the duplicate entries? So there was another research that was published, which was basically like the memorization which happens in models is mainly because of data deduplication.
- 16:56
Um, if, if we're removing the duplicate entries, that, that reduces the probability of a model to be able to memorize certain tasks, because again, it's seen those, uh, datasets over and over again in some form or the other.
- 17:09
So it's, it's naturally collecting-- creating sort of bias towards those things, and it's naturally outputting those very quickly. And the last one being data augmentation.
- 17:21
Now, let's say you've done all of this. Let's say you've picked the right model. Let's say you've done your data collection thing perfectly. You've got the best data out there.
- 17:30
What are still the things that you can think of while optimizing the performance of your model? So the first thing is do not try to compare with a GPT-4 or GPT-5.
- 17:40
It's not going to work comparatively, especially for more complex tasks. It's not a generalized model. Um, while it may be able to capture the nuances of your actual data, but it may not be able to capture the nuances of the new data that it hasn't seen or in newer domains that it hasn't seen before.
- 18:01
Um, so that's, that's one thing which I've seen a lot of companies are trying to-- are sort of in a dilemma with, uh, which is, "Oh, we've fine-tuned our model, but it's not working as good as a GPT-4."
- 18:14
The second one is basically using- In-context learning with dynamic examples. And one of the big reasons for that is, um, the, the big problem that we see with the drift in the model-- with the data drift in the models.
- 18:30
So using in-context learning with dynamic example loading allows you to be able to deal with that par-particular problem while also making sure that you are able to do, uh, cost management as well.
- 18:46
Um, the third thing that also one needs to think of is breaking down this task into smaller tasks. So for example, like if we are working with any sort of language, then instead of trying to train the model for like the entire language, can we break it down into like very specific tasks?
- 19:02
Um, so that's, that's another thing which people need to think of. Um, the final thing I would say is, uh, implementing some sort of gradient checkpointing. So what gradient checkpointing essentially does, it, um, it reduces the memory usage.
- 19:19
Um, what it, what it essentially does is it retrains the model, uh, and recomputes the weights during the backward pass. While it may look like, you know, it's, it's not the, it's not the smartest choice to make, um, but, you know, while the computation is higher, which is, yes, the, the weights will need to be
- 19:44
recomputed, um, but the downsides are easily weighed by the memory consumption. So the memory consumption is very, very less if we are implementing some sort of gradient checkpointing. So another cost effect, uh, cost management thing.
- 20:00
Um, now few more considerations and limitations, which is let's talk about the hyperparameters now. Choosing a batch size. Ideally, we go with a batch size of thirty-two or sixty-four.
- 20:12
Uh, choosing the number of training epochs. Again, one of the questions I often get is what's the right number of epochs that we should be training with? Um, if you're, if you're doing a simple test, which is if you're running something in a Google Colab, uh, for fun thing, maybe having epoch one is nice.
- 20:33
Uh, but if you're, if you are working with a good model and if you're trying to optimize for like a particular domain, then choosing to go with hundred epochs as like the starting point is, is probably like the ideal choice.
- 20:47
Choosing an optimizer. There are different optimizers that are out there. Uh, Adam optimizer is the standard choice because it's general purpose, and it works really well with different domains as well.
- 20:59
Um, implementing some sort of regularization, early stopping. Again, uh, one of the things is basically like in, in terms of if, if you're looking at the models that have been trained till now, they're, they're not a lot of-- there's not a lot of implementation on optimizing those performances.
- 21:17
While there are bigger models that we are seeing every single day with more and more parameters, it-- they're not essentially squeezing all the performance out of those models. So one of the easy ways to be able to do that is using some sort of early stopping, which is making sure that you're only working with the data that
- 21:37
is most efficient. If the model performance is declining, then you need to reconsider your batch and look into that batch, consider your embeddings.
- 21:48
Now, let's say if you fine-train, uh, fine-tune the model, the next part, which is the hardest part of the process, is, um, you know, how do we evaluate our models?
- 21:57
There are so many, um, benchmarks out there. There are so many libraries out there. Um, so there's every by Ray, there's, um, libraries j-- by NVIDIA. Um, but what you're essentially looking for mostly is the loss accuracy and perplexity, but that doesn't really paint the full picture.
- 22:17
So while I say, you know, it is the hardest part, which is there needs to be some sort of adaptation for every single business and every single use case, which is we need to be looking at evaluation from four different perspectives or four different components.
- 22:33
The first is doing some sort of metric-based evaluation, which is something like BLEU score, ROUGE score that we were considering before. Doing some sort of tool-based evaluation. So I think Weights & Biases does have a library for doing that particularly, which is their auto evaluate-- the debugger one.
- 22:50
And then there's another one, Auto Evaluator. Um, so that is able to catch the compilation errors very quickly. The, the third one is using some sort of model-based evaluation, which is using a smaller model to be able, able to evaluate the other model.
- 23:05
So while this is something which is, um, I've, I've not seen a lot of performance with this one because, again, it's hard to do, but it has a lot of potential, which is it does standardize the process eventually, and it automates the process.
- 23:22
And the final one is basically human-in-the-loop, which is something I feel like, you know, this is something that we are-- everybody is doing, um, but not the most efficient.
- 23:31
So let's, let's just ignore human-in-the-loop. Maybe let's let OpenAI talk about this. Um,
- 23:40
the final thing that I wanted to say on this one, uh, for this particular presentation is, um, while fine-tuning is great, yes, you-- but you also need to think about the entire pipeline, which is how you're thinking about the data collection, how you're thinking about the storage management, how you're choosing a base model.
- 23:59
So optimizing the performance of your model doesn't really depend on just one feature. While it may work perfectly for like a single one-off demo, but to be able to put a robust application that does sustain the test of time, and obviously, I, I'm, I'm not saying, you know, what would be an ideal time that you should be
- 24:20
testing on. Um, but in, in the case, the goal is to be able to get the optimal performance of the model and to be able to deal with all the data drift and the prompt drift and all of those things, while also making sure that we're catching a few things early and we're not exposing the enterprise to
- 24:37
like reputational risk, compliance risk, and all of those things. The entire thing has to be thought of. Um, so it is a big picture decision that I would say, um, that needs to be taken.
- 24:48
So that's, that's all my presentation for today. Um, I, I hope everybody learned something new. If there is, uh, something you would like to go with me in detail, then we can do that after the presentation.
- 24:59
But thank you so much. [audience applauding] [upbeat music]