AI Engineer World's Fair 2024
LLM Quality Optimization Bootcamp
Read the talk
Fine-tuning a small LLM for structured PII redaction
Follow a PII-redaction workflow from training data to LoRA deployment and evaluation, with the model choices, serving economics and live-demo failure that shape the result.
From a talk by Thierry Moreau and Pedro Torruella
Before you start: Familiarity with Python, chat-completion messages and JSON tool definitions will help you follow the notebook workflow.
When the prototype costs too much and still falls short
What do you do when a promising GenAI prototype still misses the production quality bar—or its inference bill has already become painful? Thierry Moreau opens with those two problems. He describes customers spending tens or hundreds of thousands of dollars per month while sometimes still struggling to deliver acceptable output.
The exercise follows a full deployment cycle: collect data, fine-tune a model, deploy it and evaluate its output. Its worked task is personally identifiable information (PII) redaction, a narrow application where a compact model can learn a specific behavior. GPU scarcity makes resource efficiency consequential: the goal is to move from an expensive prototype below the quality threshold to a system whose quality and operating costs support production margins.
OpenPipe handles managed fine-tuning, while OctoAI supplies the serving infrastructure. Moreau previews 47% better accuracy on the demonstrated task and a 99.5% inference-cost reduction, described as a 200× difference between GPT-4 Turbo and Llama 3. These are the opening claims; the later live evaluation provides the concrete scores and the generated-token prices behind the comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the intervention that matches the failure
Start by improving what you ask the existing model to do. Moreau's crawl–walk–run progression begins with prompt specificity, few-shot examples and chain-of-thought prompting. These techniques leave the model weights unchanged and try to elicit better behavior from the model already available.
Next, determine whether the model lacks information. Its parametric memory—information encoded in its weights—reflects its training data and training cutoff. Retrieval-augmented generation supplies additional context through vector similarity search, API calls, search engines or database queries. That external information lets the model answer questions about material absent from its training.
Fine-tuning becomes attractive when the remaining problem is a focused behavior that high-quality examples can teach. Moreau presents the progression as a simplification, not a requirement to build retrieval for every task. Classification, extraction, formatting and function calling are good candidates when prompting and relevant context improvements have plateaued. The prerequisites are a sufficiently narrow task and good training data; high inference bills add an economic reason to try a smaller specialized model.
The resulting workflow is a loop:
- Build a representative dataset.
- Fine-tune the model.
- Deploy it at the required volume.
- Evaluate quality and feed new failures back into the dataset.
Production inputs change. Monitoring, dataset updates and retraining keep the model aligned with those changes, so completing one pass through the notebook is only the first iteration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build data that resembles the work
Training examples should resemble the inputs and desired outputs the system will encounter in production. Moreau describes three practical sources:
| Prompts | Responses | Practical tradeoff |
|---|---|---|
| Real human interactions | Human responses | Close to the task, but potentially difficult to acquire |
| Real user inputs | Strong-model responses | Realistic inputs with cheaper response generation |
| Synthetic inputs | Synthetic labels or responses | Accessible starting point without existing logs |
Customer-service conversations illustrate the first option. For the middle option, GPT-4 or Claude can generate responses to real prompts, and OpenPipe can log the input-output pairs for training. Fully synthetic datasets from Hugging Face or Kaggle provide the starting point used in this exercise.
Keep the roles of the data partitions separate. Training data updates model parameters. Validation data supports development decisions such as hyperparameter selection. A third test set evaluates the resulting model outside those decisions. Evaluating examples the fine-tune has already seen cannot establish how well it handles new inputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the base model and the artifact you will serve
The first model decision concerns control over the trained result. Moreau favors open models because exported weights let the operator choose where and how to serve them. A proprietary fine-tuning service may keep the result inside its own hosting ecosystem. Access to the resulting weights therefore affects both deployment flexibility and dependence on a vendor.
The next decision is size. Llama 3 8B is the small-model example; Mixtral 8x22B is the much larger mixture-of-experts alternative. Moreau associates larger models with stronger reasoning and broader context options, but also higher training and serving costs, potentially requiring hardware such as H100s. His recommendation is to start with Llama 3 8B or Mistral 7B, measure the specialized task, and increase model size only if necessary. This exercise chooses Llama 3 8B.
Training methodology determines the artifact you deploy. LoRA, a parameter-efficient fine-tuning method, adds a compact learned adaptation to a base model. Full-parameter fine-tuning instead updates the model more broadly and produces a full checkpoint.
| Choice | Artifact | Serving implication |
|---|---|---|
| Parameter-efficient tuning | LoRA adapter | Multiple adapters can share a resident base model |
| Full-parameter tuning | Full checkpoint | Each customized model carries its own full weights |
Moreau contrasts an approximately 50 MB adapter with a 15 GB checkpoint. The adapter size is additional model state, not total inference memory: the base model must also be available. In the demonstrated serving design, several users' adapters share a GPU and endpoint, reducing the resources allocated to each customization. Moreau contrasts that with separately provisioned checkpoints. Full-parameter tuning provides more adjustable parameters for demanding tasks such as reasoning; he considers LoRA sufficient for the specialized classification, labeling and function-calling task here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide which infrastructure to operate
You can operate the training stack yourself or use a managed service. Axolotl is the named DIY option: it gives you training tooling, while GPU acquisition, library setup and hyperparameter choices remain your responsibility. OpenPipe packages data collection and fine-tuning into a managed workflow, making the transition from a general model to a specialized one more approachable. Starting with managed training does not prevent a later move to operating the stack directly.
Serving presents a similar choice. vLLM, MLC LLM, TensorRT-LLM and Hugging Face TGI can run models on infrastructure you control. They do not remove the need to provision GPUs, scale with demand and maintain reliability under load. OctoAI is the workshop's managed alternative, exposing familiar hosted-model endpoints while handling those operational concerns.
Moreau reports approximately 150 tokens per second for Llama 3 8B on OctoAI, attributing the speed to in-house serving optimizations. He quotes $0.15 per million tokens for that model versus $30 for GPT-4, with no additional serving charge for a customized adapter. He also reports platform volume above 20 billion tokens per day. These are historical vendor figures; the speed claim does not specify workload or concurrency conditions. For organizations requiring their own environment, he introduces OctoStack for on-premises or VPC deployment, including healthcare, banking and government use cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define what a good answer means
Evaluation depends on the output contract. Some properties can be checked directly: whether generated code or SQL executes, whether extracted labels match expected values, or whether an output satisfies a JSON format. Other properties, such as politeness and professionalism, require a judgment about which answer is better.
- Direct checks: use executable tests, labels and schema validation for properties with a precise expected result.
- Preference judgments: use human comparisons or an LLM judge for qualities that lack a simple pass/fail rule.
An LLM judge introduces its own behavior into the measurement. Moreau warns that GPT-4 judging a comparison involving GPT-4 may prefer its own answer. The evaluation method therefore needs scrutiny alongside the model being evaluated.
The evaluation also determines when to repeat the deployment cycle. There is no universal retraining interval: Moreau suggests that the right cadence could be weekly or yearly, depending on the application and its constraints.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract sensitive values before deciding how to redact them
The practical exercise moves into a Google Colab notebook, shared with attendees through a QR code, with questions directed to the event's bootcamp Slack channel. Its source is AI4Privacy's PII Masking 200K, a synthetic dataset introduced as covering 54 PII classes across 229 discussion subjects. Examples span customer support, banking and conversations between individuals.
The slide's email-like example contains a credit-card number, an IP address and a job title. The desired redacted version preserves the surrounding text while replacing the sensitive values with category labels such as CREDITCARDNUMBER, IPV4 and JOBTITLE. This retains information about what was removed without retaining the original value.
The LLM does not directly write that redacted text. It receives a system prompt describing the redaction task, the supported PII categories, a tool specification and the user's message. Its output is a call to redact containing the sensitive strings and their categories. A large tool specification makes this a useful test of function-calling behavior: the model must select both the right spans and the right labels.
Separate detection from replacement policy. Once the model has produced structured arguments, application code can replace each value with its category, obscure it completely, or consult a database for a stable fictional substitute. The last option preserves consistency across records: repeated references to one person can receive the same fake name or address, keeping the text useful for downstream processing. A returned tool call supplies proposed arguments; the application still performs the replacement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn labeled examples into tool-call conversations
The recorded setup requires OpenPipe and OctoAI accounts, plus an OpenAI API key for the GPT-4 comparison. The Python dependencies are openai and Hugging Face datasets; the notebook accepts the OctoAI token and OpenAI key. Moreau proceeds past a displayed Colab dependency error without diagnosing it. Treat the OctoAI setup and deployment steps as historical: its former homepage now redirects to NVIDIA, so the recording is not a current account-creation or endpoint guide.
The dataset provides source text, a redacted target and a privacy mask pairing sensitive values with their classes. The system prompt explains the categories in natural language, and the tool specification constrains each extracted value to a PII type. Moreau contrasts that interface with hand-written regular expressions. Here he describes 56 categories in the prompt and enum, whereas the dataset introduction specified 54; the recording does not explain the difference.
After loading the source dataset, the notebook converts labeled examples into records that look like logged LLM conversations. Each record contains the system prompt, user text, tool specification and expected assistant tool call. The following compact Python example shows that record structure with an email address as the teaching input; a full dataset uses the complete category enum and all labeled values for each message.
python
import json
from pathlib import Path
text = "Contact Alex at alex@example.com about the invoice."
fields = [{"string": "alex@example.com", "pii_type": "EMAIL"}]
tool = {
"type": "function",
"function": {
"name": "redact",
"description": "Identify sensitive values for redaction.",
"parameters": {
"type": "object",
"properties": {
"fields_to_redact": {
"type": "array",
"items": {
"type": "object",
"properties": {
"string": {"type": "string"},
"pii_type": {
"type": "string",
"enum": ["EMAIL"]
}
},
"required": ["string", "pii_type"]
}
}
},
"required": ["fields_to_redact"]
}
}
}
record = {
"messages": [
{
"role": "system",
"content": "Identify email addresses and call redact."
},
{"role": "user", "content": text},
{
"role": "assistant",
"tool_calls": [{
"id": "call_redact_1",
"type": "function",
"function": {
"name": "redact",
"arguments": json.dumps({"fields_to_redact": fields})
}
}]
}
],
"tools": [tool],
"tool_choice": {"type": "function", "function": {"name": "redact"}}
}
Path("openpipe_dataset.jsonl").write_text(
json.dumps(record, ensure_ascii=False) + "\n",
encoding="utf-8"
)
The nested arguments value is a JSON string inside the outer JSON record. The training target is the assistant's structured extraction, not a rewritten email and not the result of executing redact.
The demonstrated export contains 10,000 records in openpipe_dataset.jsonl. The upload procedure is straightforward:
- Create a dataset inside an OpenPipe project.
- Name it under Settings; the demonstration uses the June 26 Lunch and Learn.
- Upload the downloaded JSONL file under General.
- Wait for processing to format inspectable examples and create the training/held-out split.
At this point Moreau calls the held-out partition validation data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Configure training, then inspect a prepared run
Once processing finishes, the dataset view exposes individual examples and a roughly 90/10 split, now described as training and test. The selected model is a Llama 3 8B variant with a 32K context window, associated with Nous Research's Theta model; the corresponding public model family is OpenPipe Hermes-2-Theta-Llama-3-8B-32k. The displayed training estimate is $40 for the demonstrated dataset. Moreau explains that cost scales with sample count and suggests that additional examples can improve task performance. He also mentions a possible $100 introductory OpenPipe credit, qualifying that offer as his understanding at the time.
The upload is slow, so the demonstration moves to prepared artifacts rather than waiting through a fresh training job. Moreau uses a cooking-show analogy: one model goes into the oven while a previously prepared model comes out. Training completion would trigger an email, and the result of OpenPipe's parameter-efficient training is the LoRA intended for deployment.
With the current upload still unfinished, he opens a dataset uploaded days earlier. The input column contains the long system prompt, user message, tool choice and PII tool specification. The output column contains the assistant's redact call, with a list of sensitive strings and PII types. This inspection checks the actual learning contract: what the model will receive and exactly what response it should learn to produce.
The Fine-tune dialog exposes the model selector and training controls, but Moreau deliberately does not click Start Training. He instead opens an existing trained model showing 9,000 training examples and 1,000 examples labeled test. OpenPipe manages settings such as learning rate and epoch count, reducing the manual tuning needed to get started. The UI's test label should not obscure the earlier distinction between validation used during development and an independent final test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Export the adapter and register a serving asset
OpenPipe exposes both LoRA exports and merged checkpoints. A merged checkpoint incorporates the adaptation into full model weights; exporting one does not mean the original training updated every parameter. The demonstration chooses an FP16 LoRA, available as a small ZIP or a download link. Copying the link provides the handoff to OctoAI without manually moving the archive between interfaces.
The intended deployment sequence is:
- Install the OctoAI CLI in the notebook.
- Authenticate and confirm the token is valid.
- Replace the LoRA URL placeholder with the OpenPipe export link.
- Choose an asset name.
- Register the LoRA as a
safetensorsasset associated with Llama 3 8B.
OctoAI's asset library stores custom adapters and checkpoints; Moreau notes that it also supports image-generation assets such as those for Stable Diffusion. Training, export preparation and availability of the download link can introduce separate waits before registration begins.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover from the upload failure and call the model
The fresh LoRA upload fails, and retrying does not resolve it. Moreau asks Pedro Torruella to check the capability with the product team, saying that it had worked earlier that day. To continue, he retrieves previously uploaded assets, suggesting octoai asset list during the lookup, then selects an existing adapter and assigns its name in the notebook. The successful continuation therefore uses an earlier registered asset; the failed upload is not repaired during the recording.
Inference uses OpenAI-style chat completions against the OctoAI endpoint. The request supplies the system prompt and example email, selects the OpenPipe Llama 3 8B-32K model, and includes the parameter-efficient fine-tune setting identifying the chosen LoRA asset. The returned response contains a tool call to the redaction function. That demonstrates structured inference with the existing adapter, leaving application code to execute the chosen redaction policy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure extraction quality against the privacy labels
The dataset's privacy masks provide ground truth, so this evaluation does not require an LLM judge. The custom score penalizes missed sensitive information—false negatives—and incorrectly added PII—false positives. A similarity-distance comparison matches predictions to labels and allows partial credit.
In the scoring illustration, a perfect match receives 1.0. Labeling Billy as a first name rather than a middle name still receives a high score because the prediction is close to the expected extraction. An illustrated GPT-4 response misses two of three sensitive items and receives roughly one-third. These examples show why the reported score is not exact-match accuracy or a direct measurement of PII recall: detecting the right sensitive value and assigning precisely the right category contribute differently to the result.
For the live comparison, Moreau reduces the evaluation to 100 examples and calls the OctoAI fine-tune and OpenAI's GPT-4 back-to-back. He encourages larger runs, but notes the expense of the baseline: approximately $1 for 100 examples or $10 for 1,000 examples in this exercise.
The displayed results include both individual examples and the aggregate:
| Custom redaction score | GPT-4 | Fine-tuned Llama 3 8B |
|---|---|---|
| Shown example A | 0.49 | 0.85 |
| Shown example B | 0.30 | 1.00 |
| Mean over the 100-example run | 0.68 | 0.97 |
The fine-tune scores substantially higher on this comparison. The displayed means imply about a 43% relative increase in the custom score, so they do not directly reproduce the opening 47% claim. The exact scoring formula and the evaluation sample's independence from training and model selection are not established in the recording. Moreau connects the result to a production quality threshold, but a mean partial-credit score of 0.97 does not establish that 97% of messages have every sensitive value safely removed.
The closing price comparison is $30 per million generated tokens for GPT-4 versus $0.15 for Llama 3 8B on OctoAI, a quoted 200× difference. Those prices imply a 99.5% reduction in generated-token charges. They describe the workshop's historical output-token comparison, not a measured reduction in total application cost including input tokens, training and operations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the quality loop running
The durable workflow begins with prompting and relevant retrieval, then uses high-quality task data to teach the behavior that remains missing. Once deployed, the fine-tune needs continued quality monitoring and new training examples as inputs change. Managed tools reduce the effort of repeating that cycle, but the dataset remains the foundation of the result.
The serving economics also depend on work below the model API. Moreau points attendees toward Pedro's related discussion of compiler, systems and infrastructure optimization, and says OctoAI's inference prices support positive margins rather than subsidized serving. He directs SaaS and enterprise-deployment questions to the team's conference booth.
The event's LLM Quality Optimization Bootcamp Slack channel remains the offered route for questions and follow-up, alongside a historical survey promotion for additional credits. Moreau ends by inviting attendees to try the notebook, explicitly contingent on fixing the LoRA upload issue. The recording closes with that deployment problem unresolved: structured inference and evaluation worked with a prepared adapter, while the fresh export-to-serving handoff still needed repair.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Synthetic multilingual PII examples with source text, masked targets and labeled sensitive spans.
OpenPipe's 32K Theta model repository, including model lineage and inference examples.
Open-source framework for configuring datasets and running full or parameter-efficient model fine-tuning.
Further reading
Original explanation of adapting frozen model weights with trainable low-rank matrices.
Local PII processing with Python and CLI interfaces for tagging, redaction and consistent fake-data replacement.
Read the complete timestamped transcript
- 0:00
[upbeat music] So welcome everyone. Thanks for making it to this Lunch and Learn.
- 0:17
Uh, my goal today is to make sure that I get to, uh, share my knowledge and experience on LLM fine-tuning. Uh, just to get a quick sort of poll from the audience here, how many of you are...
- 0:32
have heard of the concept of fine-tuning here? Okay, so quite a few people. How many of you have actually had hands-on experience in fine-tuning LLMs? Okay. All right. That's pretty good.
- 0:44
Uh, that's more than I'm usually used to. I mean, this is quite, uh, fantastic that in this conference, the makeup of AI engineer is close to a hundred percent.
- 0:53
That's not, uh, something I'm, I'm generally used to when presenting at other, uh, you know, hackathons and, and conferences. So I feel like I'm, I'm speaking to the right crowd.
- 1:02
So just to kinda contextualize this talk, really, I'm trying to address two pains that a lot of GenAI engineers face. And to get a sense of where you are in your journey, how many really identify and can relate to the first one, which is, "My GenAI spend has gone through the roof"?
- 1:24
Okay. Yeah? All right. And how many of you are in this other segment of this journey, which is, you know, you've built POCs, it's showing promise, but you haven't yet quite met this quality bar to go to production?
- 1:39
Can I get a sense of... All right. So, so I think, you know, we have a good amount of, uh, a good fraction of the audience that can relate to one of these two problems.
- 1:48
Uh, myself, I'm a co-founder at OctoAI, and I'm gonna talk a little bit more about what we do. But the customers I've been working with, they feel those pains in very real way.
- 1:58
We're talking about tens of thousands, if not hundreds of thousands of dollars in monthly bills, and perhaps even having issues trying to go to production because the quality bar hasn't yet been met.
- 2:12
So the overview of this, uh, fifty-minute talk, uh, is going to be spent on understanding the why of fine-tuning. Really try to understand when to use fine-tuning. It's not really a silver bullet for all the problems you're gonna face, but when used right, in the right context for the right problem, it can really deliver results.
- 2:31
I'm also gonna try to contextualize this notion of fine-tuning within the crawl, walk and run of LLM quality optimization because there's different techniques that you should attempt before trying to do fine-tuning.
- 2:43
But finally, when you're convinced that this is the right thing for you, I'm gonna talk about this continuous deployment cycle of fine-tuned LLMs. So we're going to go through today over a whole crank of that wheel of this deployment cycle composed of, you know, model, uh, you know, dataset collection, model fine-tuning, deployment and evaluation.
- 3:05
And really, I'm trying to demystify this whole journey to you all, 'cause in the next fifteen minutes, we're actually gonna go through this whole process, and hopefully that's something that you're gonna feel comfortable going through and, you know, applying to your own dataset, to your own problems.
- 3:18
And so for illustrating today's use case, we're gonna use this, uh, personally identifiable information redaction use case. Now, that's a pretty traditional sort of data scrubbing type of, uh, application, but we're gonna use LLMs, and we're gonna see that we can essentially achieve state-of-the-art accuracy, uh, while keeping efficiency at, at the highest, uh, using essentially very compact,
- 3:42
very lightweight models that have been fine-tuned for that very task. So again, trying to motivate this talk, what limits GenAI adoption in most businesses today based on the conversations that I've had in the field, discussions I've had with customers and developers?
- 3:57
The first one is there's a limited availability of GPUs. I think we're all familiar with this problem. It's one of the reasons why NVIDIA is so successful lately. I mean, everyone wants to have access to those precious resources that allow us to, to run GenAI at scale, and that can also drive costs up, right?
- 4:16
So we have to be smart about how to use those GPU resources. And also, uh, when people build POCs, it displays and shows promise, but sometimes you don't reach the expected quality bar to go to production.
- 4:30
And so on this X, Y axis, you know, on this chart where Y-axis is cost and the X-axis symbolizes quality, maybe p- many people start, uh, on that green cross here, right?
- 4:44
On this upper quadrant of very high cost, maybe not having met the quality bar. That's your first POC. But really, to go to production, you need to end on the opposite quadrant, right?
- 4:54
Lower cost, higher quality, where you've met the bar, you're able to run this in a way that essentially is margin positive. And many of us are on this journey to reach that point of, uh, you know, profitability.
- 5:08
So we're gonna learn today how to use and how to fine-tune an LLM. Now, fine-tuning is a method that we're gonna use to improve the LLM quality. But as a bonus, we're gonna be also showing how to improve quality significantly.
- 5:24
And I use quality as the title of this talk because really, I think many of us AI engineers really care about reaching the high quality bar when we're using LLMs, and hopefully, um, you know, the goal of today's talk is to instill you with some knowledge on how to tackle this journey.
- 5:40
And so in terms of tools that we're gonna use today, we're gonna use OpenPipe, which is a SaaS solution for fine-tuning that really lowers the barrier of entry for people to run their own fine-tunes.
- 5:52
Uh, you don't need hardware or cloud instances to get started. And we're gonna use this to deliver quality improvements over state-of-the-art LLMs.
- 6:00
And of course, since I work at OctoAI, I'm gonna also be using OctoAI here for the LLM deployment, and that's gonna be the solution that we're gonna use to achieve cost efficiency at scale.
- 6:13
And really the, the key here is to be able to build on a solution that is designed to serve models at production scale volumes.
- 6:23
And just to give you a little bit of a sneak peek in terms of the results that we're gonna showcase today after you go through this whole tutorial, and this is something that you're gonna be able to reproduce independently, so, you know, all the code is there for you to go through.
- 6:36
We're gonna be able to show that we can achieve forty-seven percent better accuracy at the task that I'm gonna showcase today using this OpenPipe fine-tuning. And by deploying the model on OctoAI, we're gonna achieve this, I mean, it seems kinda ridiculous, ninety-nine point five percent reduction in cost.
- 6:55
This is really a two hundred x reduction in cost here from a GPT-4 Turbo to Llama 3. And mostly because this is a much smaller model, it's open source, and we've optimized the hell of this model to serve it cheaply on OctoAI.
- 7:07
So I'm gonna explain how this is achieved, but I hope your interest at least has been piqued on those results that you yourself can reproduce.
- 7:16
So when to use fine-tuning. Uh, again, it's not really a silver, uh, bullet for all your quality problems. It has its right place and time. So I like to contextualize it within the crawl, walk, run of quality optimization, right?
- 7:30
And as GenAI engineers, uh, many of us have embarked on this journey. We're at different stages of this journey. And really, it should always start with prompt engineering, right?
- 7:40
And many of you are familiar with this concept. You start with a model, you're trying to have it accomplish a task, and sometimes you don't really manage to see the result you expect to see.
- 7:50
So you're gonna try prompt engineering, and there's different techniques of varying levels of so-sophistication. Uh, this talk is not about prompt engineering. So, you know, you can improve prompt specificity.
- 8:00
There's few shots at prompting, where you can provide examples, uh, to improve essentially the, the quality of your output. There's also chain of thought prompting. I mean, some of you probably have heard these concepts.
- 8:10
But this is where you should get started, right? Make sure that given the model, given those weights, you just try to improve the prompt to get the right results.
- 8:17
Sometimes that's not enough. And, uh, there's a second class of solutions which I like to map to the walk stage, um, Retrieval Augmented Generation, right? We've probably seen a lot of talks on RAG today and throughout this conference.
- 8:31
So, you know, there's hallucinated results. Sometimes the answer is not truthful. Well, why is that? It's because the, um, the weights of the model that is really the parametric memory of your model is limited to, you know, the point in time at which the model was trained.
- 8:48
So when you try to ask questions on data it hasn't seen or information that's more recent than when the model was trained, it's not gonna know how to respond, right?
- 8:56
So the key here is to provide the right amount of context. And so this is achieved through, uh, similar research, for instance, in a vector database through function calling to bring the right context by invoking an API, through search, uh, through querying a database.
- 9:11
And so this is something that I think many of us AI engineers have been dabbing in in order to provide the right context to generate truthful answer, right? Complement the parametric memory of your model with non-parametric information, and that's RAG in a nutshell, right?
- 9:25
So you've tried prompt engineering, you've tried RAG, you've eliminated quality problems and hallucinations, but that's still not enough, right? So what do you try next? Well, fine-tuning, I think, is, uh, the next stage.
- 9:37
And again, I'm generalizing a very complicated and complex journey. But in spite of your best efforts, you've tried these techniques for maybe days, weeks, or even months, and you still won't get to where you need to be to hit production.
- 9:50
And we're gonna talk about this journey today, right? Fine-tuning. So when should you fine-tune a model? Uh, again, after you spend a lot of time in the first two phases of this, uh, journey, so spending time on prompt engineering, spending time on retrieval augmented generation, and you don't see the results improve.
- 10:09
And generally, what helps is whenever you use an LLM for a very specific task, something that's very focused, for instance, classification, information extraction, uh, trying to format a prompt, using it for function calling, if you can narrow the use case, uh, to something that is highly specific, then you have an interesting use case for, um, for applying
- 10:31
fine-tuning here. And another requirement is to have a lot of your own high-quality data to work with, because that's gonna be your fine-tuning dataset. That goes without saying. But a model is only as good as the data that it, that the model was trained on, and we're gonna apply this principle here in this tutorial.
- 10:48
And finally, I think as an added incentive, oftentimes we're all driven by economic incentive in the work we do. Uh, for those of you who are feeling the pains of, uh, high GenAI bills, whether it is with OpenAI or with a cloud vendor or a third party, well, this is generally a good reason to explore fine-tuning.
- 11:08
So we're gonna go over all the steps now that we've kinda contextualized why fine-tuning and when to consider fine-tuning. We're gonna consider all the steps here in this, uh, continuous deployment cycle.
- 11:20
It starts with building your dataset, then running the fine-tuning of the model, deploying that fine-tuned LLM into production, so you can achieve scale and serve your, your, you know, your customer needs or internal needs at high volumes, and also evaluate quality.
- 11:37
And this is an iterative process. There's not a single crank of the wheel. This is not a fire-and-forget situation because data that your model sees in production is gonna drift and evolve.
- 11:48
And so this is something that you're gonna have to monitor. You're gonna have to update your dataset. You're gonna have to fine-tune your model. And I don't wanna scare you away from doing this because it sounds fairly daunting.
- 11:57
And so by the end of this talk, we'll have gone through a full crank of that, uh, wheel, and hopefully, you know, it... Through these, uh, SaaS toolings that I'm gonna introduce you to is gonna feel a lot more approachable, and hopefully I'll demystify the whole process of fine-tuning models.
- 12:14
So let's start with step one, which is to build a fine-tuning data set. Now, the data of the model has to be trained on ideally real-world, uh, data, right?
- 12:25
It has to be as close as possible to what you're gonna see in production. So there's kind of a spectrum of ways to build and generate a data set.
- 12:33
Ideally, you build a data set out of real-world prompts and real-world-generated res-- uh, real-world human responses. So for instance, you have customer service, you've logged calls with a customer agent, you have an interaction between two humans.
- 12:47
That's a very good data set to work with, right, because it's human-generated on both ends. This is very high quality, but not everyone has the ability to acquire this data set.
- 12:56
Sometimes you're starting from scratch, so not everyone has the luxury to start there. There's also kind of an intermediary between real-world and synthetic, where you have real-world prompts but AI-generated responses.
- 13:08
And so this is kind of a good middle ground between cost and quality because you're starting from actual, uh, ground truth information that is derived from real data, but the responses are generated by a high-quality LLM, say, GPT-4 or Claude.
- 13:24
And actually, OpenPipe is a solution that allows you to log the inputs and outputs of an LLM like GPT-4 to build your data set for fine-tuning an LLM. So this is something that, uh, you know, a lot of practitioners use.
- 13:38
And finally, there's the fully synthetic data set using fully AI-generated labels. And oftentimes, when you go on Hugging Face or Kaggle, you'll encounter data sets that have been built entirely synthetically, and that's a great way to kinda get started on this journey.
- 13:54
And actually, one of the data sets we're gonna use today is, uh, from that latter category.
- 14:00
And of course, I mean, it probably goes without saying, but in case people are not fully, uh, [chuckles] familiar with this notion, you wanna split your data set into a training and validation, um, set because you don't wanna evaluate your model on data that your, um, fine-tune has seen, right?
- 14:16
And so, uh, many of you who are ML and AI engineers are already familiar with this, but I just wanna reiterate that this is important. And finally, you know, this is used for hyperparameter tuning, and when you're deploying it and actually testing it on real-world examples, you wanna have a third set outside of training and validation, which
- 14:33
is your test set. Now you've built your data set, you're ready to fine-tune your model, and there's a lot of decisions that we need to make at this point.
- 14:42
And the first one is gonna be open source versus closed source, right? And so who here, just like raise of hands, is using proprietary LLMs or GenAI models today from OpenAI, Anthropic, uh, Mistral AI?
- 14:56
Okay, good amount of crowd. Who here has been using open-source LLMs like Llama, some of the free Mistral AI models? Okay, so maybe a smaller crowd, right? And maybe that's because these models are not as capable and sophisticated, and...
- 15:12
But I'm gonna walk you through how you can achieve better results if you do the fine-tuning right. So of course, the benefit of open source, and this is why, you know, I'm, I'm obviously biased, but I'm a open-source advocate, is that you have to-- you, you get to have ownership over your model weights.
- 15:30
So one-- once you've done the fine-tuning, you are the proprietor of the weights that are the result of this fine-tuning process, which means that you can choose how you deploy it, how you serve it.
- 15:40
This is part of your IP. And I find that this is a great thing for anyone who wants to embark on this fine-tuning journey. With proprietary solutions, you're not quite the owner, or you don't have the flexibility to decide to go with another vendor to host the, the, the models yourself.
- 15:56
And so you're kinda locked into an ecosystem. Some people are comfortable with that, others are less comfortable with it. And many of the customers that we talk to, they're very eager to jump on the open source train, but they don't really know how to get started or, uh, you know, where to start on this journey.
- 16:12
So hopefully this can, this can help inform you how to take your first steps here into the world of open source.
- 16:18
Then there's a question of like, do I use a small model or a large model? Because, for instance, even in the world of open source, you have models that are in the order of eight billion parameters, like Llama 3 8B, and then you have the large models with, uh, Mixtral 8x22B.
- 16:32
So this is a mixture of expert model with over a hundred billion parameters. Uh, very different beasts, and we're gonna see even larger models from Meta. And generally, my recommendations here is, well, look, the large models are amazing.
- 16:45
They have broader context windows, they have higher capabilities at reasoning, but they're also more expensive to fine-tune and more expensive to serve. And typically, when you have to do a deployment, you're gonna have to acquire resources like H100s to run these models.
- 17:01
So generally, start with a smaller model like a Llama 3B. And sometimes you'll be surprised by its ability to learn, uh, specific problems. So that's my recommendation. Start with a smaller Llama 3 8B or Mistral 7B, and if that doesn't work out for you, then move towards, uh, larger and larger models.
- 17:21
And today we're gonna be using this Llama 3 eight billion parameter model.
- 17:25
There's also different techniques for fine-tuning, and I'm gonna go over this one fairly quickly, but there's two classes of, of fine-tuning techniques. One which is parameter efficient fine-tuning. It produces a LoRA.
- 17:39
And the other one is a full parameter fine-tuning, which produces a checkpoint. A LoRA is much more s-- much smaller and efficient in terms of memory footprint. We're talking about fifty megabytes versus a checkpoint that is fifteen gigabytes.
- 17:53
And so you can guess that because of its more compact representation, you're able to serve it on a GPU That doesn't require as much onboard memory. And you can even serve multiple LoRAs at the same time, so multiple fine-tunes on a GPU for inference, as opposed to the checkpoints which require dedicated GPU for every single fine-tune.
- 18:18
So there's more flexibility in deployment, and we're gonna use that today. We're actually gonna serve these LoRAs, which are the result of parameter efficient fine-tuning, on a shared tenancy endpoint with other users who have their own LoRAs all running on the same server.
- 18:33
And that allows us to really reduce the cost of inference. And there is a benefit to checkpoints though, and full parameter fine-tuning, which is that there are more parameters to tune, so it's a more flexible, uh, fine-tuning technique.
- 18:48
It allows the model to have... essentially achieve, uh, better results at more expensive tasks like logical reasoning. But for very specialized tasks, which is what we're gonna look at today, like classification or labeling or function calling, a LoRA is just fine.
- 19:04
So we're gonna use parameter efficient fine-tuning. And also when you're doing fine-tuning, you have to decide, am I gonna DIY it or am I gonna use SaaS? So I'm sure some of you only like to DIY things, others like the convenience of SaaS.
- 19:18
And here I'm not gonna take a side. I think there's some great tools right now to DIY your own fine-tuning, uh, for instance, the open source, uh, project Axolotl.
- 19:28
And actually at the conference there's the, the, the creator behind Axolotl who you might be able to catch. Um, and, you know, the challenge here is that you have to find your own GPU resources, you have to understand how to use these libraries even though they're, they're, they're easier than ever, uh, to, to adopt, and you have
- 19:45
to tune and tinker, uh, you know, settings and hyperparameters. Then there's SaaS, which really aim to make it easy to embark on this journey. Companies like OpenPipe, and there's, uh, many folks from the OpenPipe at this conference today, so if you can catch them, please do talk to them.
- 20:02
And they're trying to lower the barrier of entry to fine-tuning, right? To make it easy, and they bring all this tooling, all these libraries to make it as seamless as possible to, for instance, move from a GPT-4 model to a fine-tune with, uh, the least amount of steps in, in collecting your data, fine-tuning, et cetera.
- 20:17
And so we're gonna use SaaS today, but if you feel more comfortable in this journey, uh, you might s- wanna start with SaaS and then evolve into DIY-ing it.
- 20:27
When it comes to deployment, you have to navigate the same options, right? Once you have a fine-tuned model, now you need to decide, well, how am I gonna serve it, right?
- 20:33
Because I need to generate maybe thousands, millions, or billions of tokens a day. And so you need infrastructure, you need GPUs, you need inference libraries. Some people like to DIY it using libraries like vLLM, MLCLLM, TensorRT-LLM, Hugging Face TGI.
- 20:51
If these are all things that you might have heard of, uh, these are all solutions to run models on your own, on your own infrastructure.
- 21:01
But you need to provision the resources, you need to build the infrastructure to scale with demand, and that can get tricky, especially achieving high reliability under load. That's a challenge that many people face as they scale their business up.
- 21:15
With SaaS, you can essentially work with a third party like OctoAI, and obviously I'm a bit biased again, I work there, so I'm gonna insert a shameless plug for OctoAI, which allows users to get these fine-tunes deployed on SaaS-based endpoints.
- 21:32
So endpoints very similar to the ones from OpenAI, for instance, if you're familiar with that or Claude.
- 21:38
And, uh, it offers the ability to serve different kinds of customizations as well. And so very quickly I wanna go over the advantages of OctoAI here. First of all, you get speed.
- 21:49
So Llama 3 8B parameter model, you get achieve around a hundred fifty tokens per second, and we keep on improving that number because we've been applying our own in-house optimizations to the model serving layer.
- 22:01
It also has a significant cost advantage because it costs about fifteen cents per million tokens compared to, say, GPT-4, which costs thirty dollars per million tokens. So that's where the two hundred x comes from.
- 22:13
And we don't charge a tax for customization, so whether you're serving the base model or a fine-tune, it's the same cost. There's customization as I mentioned. You can load your own LoRA and serve it.
- 22:25
And finally, scale. Our customers-- some of our customers generate up to, uh, billions of tokens per day on our endpoints. I think we're serving around over twenty billion tokens per day.
- 22:35
And so we've focused and spent a lot of time on improving robustness. And also worth mentioning, if SaaS doesn't cut it for you, you are working for a Fortune five hundred company or, you know, a healthcare company, uh, banking sector, government, you need to deploy your LLMs inside of your environment, either on-prem or in VPC, we also
- 22:58
have a solution called OctoStack. Come talk to us at the booth. So that's it for the shameless, uh, plug section. Let's go over to section four, which is evaluating quality, right?
- 23:08
We've talked about dataset collection, fine-tuning, deployment, now quality evaluation. And we could have an entire conference just dedicated on that. I'm gonna try to summarize it into kinda two classes of eva-evaluation techniques that I've seen.
- 23:23
First of all, you know, can your quality be evaluated in a precise way that can be automated? For instance, you generated a program or a SQL command that can run.
- 23:33
Uh, can you, for instance, label or extract information or classify information in an accurate way? That's a kind of pass or fail scenario, right? Or formatting the output into a specific JSON formatting.
- 23:46
This is something that you can easily test as a pass or fail test. And then there's more of the soft evaluation. For instance, if I were to take an answer and say, "Well, which output is written in a more polite or professional way?"
- 23:58
You can't really write a program to evaluate this unless you're using an LLM of course, right? But you have to put yourself into maybe two thousand, two thousand... Sorry, two thousand and twenty, two thousand and twenty-one mindset before, uh, GPT was around.
- 24:11
Well, it'd be hard to build a program that can assess this, right? So generally you'd need a human in the loop to say which out of, out of A or B is a better answer.
- 24:21
Thankfully, today we can use LLMs to automate that evaluation. But keep in mind that, for instance, if you're using GPT-4 to evaluate two answers, well, if you're comparing against GPT-4, it might favor its own answer, and people have seen that in these kind of evaluations.
- 24:36
So this is a whole science. I mean, we could have a whole conference just on this. I just wanted to present the high-level, uh, guidelines of this whole cycle of deploying, uh, fine-tune LLMs.
- 24:48
And so really there is no finish line. That's what I wanna convey to you all, that going through a single iteration is something that you might have to do on a regular basis, maybe once a week, maybe once a year.
- 25:00
It all depends on your, uh, use case and constraints.
- 25:05
Now, let's get a bit more practical. Let's switch over to our demo. And so for those of you who came, uh, a little bit late, there's a QR code here that you can scan, and that will point you to our Google Colab.
- 25:21
And we also have under Slack... Uh, let me see if I can pull it. If you're in the Slack channel for AI Engineers World Fair, there is this, uh, quality optimization boot camp where you can ask questions here, if you wanna follow along.
- 25:36
And so we're gonna go-- we're gonna try to go over the, uh, practical component in the next, uh, twenty-five minutes. I just wanna
- 25:43
provide some context here. The use case is, uh, personally identifiable information redaction. We've taken this from a dataset composed by AI for Privacy called PII Masking 200K. It's one of the largest datasets of its kind.
- 26:00
It has fifty-four different PII classes or different kinds of sensitive data, like the name, the, uh, email address, a, you know, address of-- physical address of someone, uh, their credit card information, et cetera, et cetera, across two hundred and twenty-nine discussion subjects.
- 26:17
So that includes conversations from a customer ticket resolution, uh, conversations with a banker, conversations between individuals, et cetera. What this dataset looks like is as follows. You're gonna have a message, an email.
- 26:32
Uh, here we have, you know, something that looks like it came out of an email. That contains credit card information, IP address, maybe even a mention of a role or, or, uh, anything that is essentially personal i-- per-personally identifiable.
- 26:47
And I've highlighted those in red because they will need to be redacted. And after redaction, we should get the following text that shows, "Look, here is this information that is now redacted, anonymized."
- 26:59
But instead of just masking it, we're actually telling it what kind of category this information belongs to, right? A credit card number, an IP address, or job title. And this is how we're gonna redact this text.
- 27:11
So where do LLMs come in? The way we would use it is through function calling. Who here has used LLMs with tool calls or function calls? Okay. So quite a few people, you know, and, um, as, as many of us are aware, this kind of what powers a lot of the agentic applications.
- 27:31
So this is a great use case for people who wanna do function calling and are not seeing the results, you know, out of the box from, say, GPT-4, uh, that they would like to, to, to see.
- 27:41
And in this case, we're actually gonna see that, that these kind of state-of-the-art models aren't doing quite well at fairly large, uh, and complex, uh, function call use cases.
- 27:50
Uh, so to achieve this, uh, redaction use case, we're gonna pass in a system prompt. We're gonna also pass in a tool specification. The system prompt says, "Look, you're an expert model trained to do redaction, and you can call this function.
- 28:03
Here are all the sensitive, uh, PII categories for you to redact." And then as a user prompt, we're gonna pass in that email or that message, and then the output is a tools call.
- 28:15
So it's not the redacted text, it's actually a tools call to that redact function that's gonna contain all the arguments for us to perform the redaction. Why am I doing this as opposed to spitting out the redacted text?
- 28:28
Well, that gives us flexibility in terms of how we wanna redact this text. We could choose to just replace that information with the PII class. We could also completely obfuscate it.
- 28:40
Or we could choose to use, for instance, a database that maps each PII entry to a fake substitute so that we have an email that kinda reads normally, except the credit card, the, the, the names, the addresses are all made up, but they will always map to the same individual.
- 28:59
And so that allows us to do then more interesting processing on our dataset, right? So that's why we're gonna use function calling here, and let's start to build the dataset.
- 29:08
So I'm gonna switch over to our notebook here. Uh, this notebook is meant to be sort of, uh, self-explainable, so there's a bit of redundance-- redundant context. As part of the prerequisites, you're gonna have to get an account on OctoAI and OpenPipe, um, and, and these are the tools that we're gonna use.
- 29:26
And if you wanna run the evaluation function, also provide your OpenAI key because we're gonna compare against GPT-4. So we're gonna install the Python packages initially, only OpenAI and datasets from Hugging Face.
- 29:39
You can ignore this, uh, pip dependency error here, uh, which happens when you, uh, pip install datasets in a Colab notebook. But that's okay, we can get past that.
- 29:49
You can enter your OctoAI token and OpenAI API key at the beginning, and I've already done this. So we're gonna start with the first phase, which is to build a fine-tuning dataset.
- 29:59
So we have this PII masking dataset, and I'm gonna show it from Hugging Face, so PII, uh, masking. And you can see what the dataset looks like. It has the source text information.
- 30:12
As you can see, these are, you know, excha-- you know, snippets from emails, for instance. You have the target text that is redacted and the privacy mask that contains each one of the PII and the classes associated to it.
- 30:25
So this contains all the data, all the information input and labels that we need to build our Uh, our data set for fine-tuning. And so really what we're gonna do
- 30:37
is that we're gonna use the system prompt.
- 30:41
Here, we're gonna define our system prompt here, which is again, telling the model, "You're an ex-expert model trained to redact information," and here are the fifty-six categories explaining next to each category what that corresponds to.
- 30:55
And this is really the beauty of LLM and sort of natural language entry, is that in the old world, when we were doing PII redaction, we had to write complex regular expressions, and here this is all done through just providing a category and a bit of a description here, and the LLM will naturally infer how to do
- 31:13
the re-redaction. We're also going to, uh, define the tool to call, right? And so this is done as a, uh, essentially a dictionary, a JSON object, and as you can see, there is an array that contains, uh, dictionaries containing a string and a PII type, and the string is the PII information.
- 31:34
The type is essentially one of fifty-six categories that we provide as an enum. So right off the bat, you can see that this tool call is, um, you know, a bit of a large function, uh, specification.
- 31:46
And so let's load our data set from Hugging Face. In this case, it's gonna take maybe a few seconds to load in that data set of two hundred thousand, uh, entries.
- 31:56
And then what I have in the next cell when I'm downloading this, uh, data set is what I'm gonna use to build my fine-tuning training data set. And here's the thing about fine-tuning, is that to build your data set, you need to make it seem like you've essentially logged conversations with an LLM, right?
- 32:15
You're logging the prompts and the responses, 'cause that's how you're gonna fine-tune it. You need to tell it, "This is the input. We have system prompt, uh, tools, specification, user prompt, and here's the, uh, tools call response that I expect to see."
- 32:32
And so this cell here just sets it up so that we essentially have each training sample as a message from an LLM that's been logged. We're gonna see what that looks like in a second.
- 32:44
So we're gonna build a ten thousand entry training data set for OpenPipe,
- 32:51
and that's gonna be downloaded as this openpipe_dataset.jsonl. And so as I run this cell, it's gonna download this, uh, from Colab.
- 33:02
And now when you switch over to OpenPipe, we're gonna create a new data set. So once you're on OpenPipe console, you have a project here. I've, uh, generically named it Project One.
- 33:14
You can access data sets, and already, as you can see, I already have built a, a few data sets, uh, before, but if you're a first-time user, you're not gonna see anything under data sets.
- 33:24
So you can create a new data set here by clicking on this button. And if you go under Settings, we can name our data set, so I'm gonna call it,
- 33:34
uh, Lunch and Learn, and today is, uh, June two six. All right, so this is today's Lunch and Learn. I'm gonna, I'm gonna call this my dataset. And under General, I can upload the data that I just downloaded from my notebook, openpipe_dataset.jsonl.
- 33:54
So this upload operation is gonna take a few seconds or maybe a couple of minutes, because what's gonna happen o-on OpenPipe is not only we're uploading this data set, but it's gonna do some, uh, pre-processing here to split it into a training and validation set.
- 34:11
It's also going to get it all formatted in a nice way so we can essentially look into the data set.
- 34:18
Uh, so you can see there's this little window here that shows that you're uploading the data set and that it is essentially being processed. So while this is happening,
- 34:28
right, we've prepared our data set, and we're gonna take a look at it in a second while it's being processed on OpenPipe, but let's see how we're gonna do the fine-tuning in the next stage, right?
- 34:38
So once we have our data set uploaded, we're gonna have this view on the data set that shows every single entry that we can peek into and how it's split into training and test set, generally a ninety/ten percent split.
- 34:52
And from that UI, we can launch a fine-tune, and this is where we get to choose our base model, and what we're gonna choose is a Llama 3 eight billion parameter model with 32K,
- 35:03
uh, context width, which is a, uh, fine-tune from News Research called a Theta model. And you can see that there's essentially a pricing here that is being estimated for this fine-tune.
- 35:17
We have a substantial training set because it can range from, say, hundreds of samples to thousands to hundreds of thousands, and the cost can scale up as you, um, as you feed in more training samples, but it will improve the accuracy, and it also provides an estimated training price of forty dollars.
- 35:35
Now, that might seem like a lot, especially when you're tinkering with fine-tuning. But keep in mind, some of the people that we work with, they tend to spend tens of thousands or maybe hundreds of thousands of dollars a month on GenAI spend, so this is absolutely something that you can do up front that will pay off.
- 35:50
And I believe that on OpenPipe, if you get started, you get a hundred dollars credit, uh, so that allows you to, to run some fine-tunes off the bat, uh, without having to necessarily, uh, have to, to pay.
- 36:03
So, um, let's go over to OpenPipe. And it is still uploading. I think maybe the network is, uh, [chuckles] is a bit slow. But we're going to essentially start training at this point, and once the training is happening, we're gonna then deploy the fine-tune LLM Uh, when training is done.
- 36:24
And what happens on OpenPipe is when you're done with training, you're gonna get an email when that training job is done. It can take a few minutes, so I'm gonna pull a Julia Child here.
- 36:33
I'm gonna stick the, you know, the turkey in the oven and, and the second oven, I'm gonna have a, a pre-baked turkey, just so that we don't lose time.
- 36:40
But as you're going through this on your own, keep in mind it's gonna take a little bit of time to just kick off that whole fine-tuning process. But it's not that long because, um, you know, you're training a fairly small model here.
- 36:54
All right, so this is still, uh, saving.
- 36:57
But let's kinda take a look at what we've done so far, right? So we've built our data set using a synthetic data set from Hugging Face. Uh, we format each input/output pair from the data set as logged LLM messages, and this is essentially stored as a JSONL file that we upload to OpenPipe, and we produce ten thousand
- 37:15
training samples. We're fine-tuning a model from OpenPipe, and we're-- OpenPipe uses a parameter-efficient fine-tuning, which produces a LoRA, and we choose Llama 3 eight billion parameter model as the base.
- 37:27
And when we deploy, what we're gonna use here is OctoAI.
- 37:32
So let's see. This didn't finish uploading, so I'm gonna go into the one that I uploaded just a couple days ago, just to essentially show you what you should see on the, uh, user interface.
- 37:44
So as you peruse through the training samples, what you're gonna see is an input column and output column. And so on the left, you have the input with the system prompt.
- 37:54
As you can see, it's a, it's a big boy, uh, because it has all these different categories, right, that it needs to classify. It also has the user prompt, which is the message that we need to redact, the tool choice, and the tool specification here with all the different categories of PII types.
- 38:11
And then the output will be, will be this tools call from the assistance response, and that will have this redact call, along with these arguments, field to redact as a list of dictionary entries containing string and PII-type information, right?
- 38:27
And so this is what we've passed into our fine-tuning, uh, data set into OpenPipe.
- 38:35
And, uh, this is still saving, so I'm just going to go ahead and go to the model. So
- 38:42
once you have the data set uploaded, again, you hit this Fine-tune button, and this is what's gonna allow you to launch a fine-tuning job, right? I can call this blah.
- 38:51
And this is where you select under this dropdown the model that you wanna fine-tune. Uh, this is again what we saw before. Training size is substantial. I'm not gonna hit Start training because I already have a trained model.
- 39:02
But when you do that, it's gonna kick off the training, and when it's done, you'll get notified by email.
- 39:07
Now let's fast-forward. Let's assume I've already trained my model. So I'm gonna have this model here that's been fine-tuned from this data set. I'm gonna click on it. As we can see, it's an Llama 3 8B model.
- 39:20
It's been fine-tuned over these ten thousand data sets split into nine thousand, uh, training samples and a thousand test samples. Uh, we can even look at the evaluation.
- 39:33
But going back to the model. And the nice thing is that it's taking care of the hyperparameter, like learning rate, number of epochs. It kinda figures it out for you, so you don't really have to tweak those settings.
- 39:46
And I find that to be very convenient, especially for people who haven't yet built an understanding of how to tweak those values.
- 39:52
And the beauty of using OpenPipe is that you can now export the weights and be the owner of those weights, right? Remember when we talked about open source, it's really important to own the result of the fine-tuning.
- 40:03
So you can download the weights in any format you want. You have LoRAs, but also merged checkpoints, so you can have a, a, a parameter-efficient representation as well as a checkpoint.
- 40:13
And so we've selected to export our model as a FP16 LoRA, which is what we're gonna use to upload our model on OctoAI, which is what we're gonna use to deploy the model.
- 40:23
So now I can download the weights as a zip file, and it's fa-fairly small, only fifty megabytes. But I can also copy the link, copy the URL, and this is what we're gonna need to do in this tutorial.
- 40:36
So to deploy the model, what we need to do is copy this URL. I'm gonna download in the cell the OctoAI CLI. This is a command line interface for users to upload their own fine-tunes to what we call our asset library.
- 40:53
So this is a place where you can store your own checkpoints, your own LoRAs, for not just LLMs, but also models like Stable Diffusion if some of you are developers who also work in the image gen space.
- 41:03
And so we can serve these customized models, uh, on our platform. And so we're gonna upload this LoRA from OpenPipe to OctoAI. So we're gonna log in just to make sure credentials are good, and here we have a confirmation that our token is valid.
- 41:24
And in this cell, we have to replace the LoRA URL from set me to that URL that I just copied here from Download Weights.
- 41:32
And keep in mind, this might take a couple minutes to get the link to appear. But once you have that link... And again, I'm kinda skipping ahead because when you're gonna run this at your own time, it might take a, you know, a few minutes to run the fine-tune, it might take a few minutes to download the
- 41:47
weights. But everything that I'm running here is essentially the steps that you'll take yourself. And what I'm doing here is, uh, passing in this URL here and setting a LoRA asset name in my OctoAI asset library.
- 42:02
So I can then create this asset from this LoRA as a SafeTensor file and, uh, based on the Llama 3 8B model. I'm gonna name it, uh, let's see.
- 42:19
Seems like something has, uh, failed here. So let's try to run it again.
- 42:38
And so what this is doing is, uh, let's see.
- 42:47
Usually that, that, uh, that should have worked. [chuckles] So what's, uh, what should happen here is, at this point, once you've taken the, uh, URL of your fine-tune asset,
- 43:01
should be able to host it on our asset library, and then from there, serve it to start running some inferences. So this, uh, [laughs]
- 43:12
this, this LoRA upload step didn't quite work here. So Pedro, are you able to maybe double-check with product whether this capability is working?
- 43:23
Uh, this isn't a good demo unless something fails and... [laughs] So, uh, yeah, you know, I just tested it earlier today and it was working flawlessly. So, uh, let's see.
- 43:36
I might have to list my assets so I can pull an old one.
- 43:41
Um, actually, one second. Pedro, can you, can you tell me what the, uh, command is to, to list the assets that are on... I think it might be OctoA- OctoAI asset list.
- 43:55
All right, let's... Okay, there we go. So I'm gonna pull from an asset that I uploaded earlier.
- 44:10
It could be the third one. The third one?
- 44:16
All right. Yeah, because... All right, so I'm gonna take an asset that I uploaded earlier. I'm gonna have to look into why, uh, that step failed, but, uh, let's,
- 44:34
let's try this. Okay, so I'm gonna use an asset that I uploaded earlier. I'm not sure why this didn't work, but I'll make sure that this is working for you all to reproduce this step.
- 44:45
And I'm gonna set LoRA asset name equals this. All right, so these other LoRAs I uploaded using the exact same steps as I used for this, uh, tutorial.
- 45:00
So we'll make sure to get to the bottom of this and, uh, we'll use the Slack channel here for folks who want to run through this, uh, step. But I'm just gonna run an example inference here on this asset that I pulled from OpenPipe.
- 45:13
And so again, we have our, uh, system prompt. We're gonna pass in this, uh, ex... you know, this message, this email as our test prompt. And then when we're invoking this OctoAI endpoint, we're using the standard chat completions, uh, from OpenAI.
- 45:30
And what we're passing here is this OpenPipe Llama 3 8B-32K model, and we pass in this argument for parameter efficient fine-tune, and pass in the LoRA asset name that we just uploaded to the asset library.
- 45:47
And as we can see, the response here contains the tool calls and the call to the function that will do the redaction. So this is behaving exactly as we intend it to.
- 45:57
So now we can move on to the quality evaluation. For quality evaluation, what we've done is use essentially an accuracy metric. Thankfully, we have a ground truth, right, from our dataset.
- 46:09
All the exchanges have been labeled with privacy mask information that we can use as ground truth. So that makes evaluating scoring or results fairly easy. We don't have to use an LLM, for instance, for that.
- 46:21
We can actually use more traditional techniques of accuracy evaluation. And so we have a metric that we've built. It assigns a score that can be penalized when PII information was missed or mistakenly added, i.e., false negative or false positive.
- 46:37
And then we use a similarity distance metric to kinda match the responses from the LLMs compared to our ground truth. So for illustration purposes, we have, for instance, this PII information that's been redacted.
- 46:50
That's a score of one point oh because it's the perfect match. Our fine-tune might, for instance, miss the fact that Billy was the middle name and might interpret it as first name.
- 47:00
In that case, we're still attributing a high score because it's close enough, and probably for a practical use case, that would be good enough. But for instance, upon calling GPT-4, it fails to identify two out of the three information that we had to redact, and so the score is about a third here, right?
- 47:17
So in this case, what we're gonna do here, I'm just gonna reduce the test size to a hundred samples, and I am going to run this evaluation inside of, uh, this cell.
- 47:30
It's gonna bring us a hundred test samples that we can then run our evaluation metric and get our overall scoring out of. Uh, so if we look at, you know, the, uh, output from the cell, essentially, we're just evoke, invoking back-to-back the fine-tune running OctoAI, and we're invoking GPT-4 on
- 47:55
OpenAI to do the results collection. So we're gonna collect some results here.
- 48:02
And, uh, once we've collected the results, once we get to a hundred, I think we're getting pretty close here, we can run the quality evaluation metric. And of course, I invite you to run it on more samples, maybe a thousand or ten thousand.
- 48:14
Uh, it just gets more expensive as you're using GPT-4. You know, to run a hundred samples, it costs about a dollar in inference. So, uh, then a thousand samples costs, uh, ten dollars.
- 48:27
And now we're gonna score it. All right, so we're gonna go through every single entry. We have our ground truth information. We have our...
- 48:35
Eval and labels from GPT-4 and our eval and labels from our fine-t-tune. And we can see that right off the bat, the fine-tune is actually better at finding the PII to redact.
- 48:47
Here, GPT-4 scored only a score of, uh, zero point four nine, whereas our fine-tune achieved zero point eight five, and here zero point three for GPT-4, one point zero for the fine-tune.
- 48:58
So the fine-tune overall is performing better, and once we aggregate and average the score, GPT-4 achieved zero point six eight out of one, whereas our fine-tune achieved zero point nine seven.
- 49:12
And so that's the difference between prototype and production, right? You're expected to achieve somewhere in the single nine or two nines of accuracy, and this is what this technique shows, uh, allows you to achieve.
- 49:23
And again, I want to reiterate that in terms of cost, GPT-4 costs upwards to thirty dollars per million tokens generated, whereas Llama 3 8B on OctoAI costs just fifteen cents.
- 49:34
That's a two hundred x difference, right? So with that, I just want to conclude with some takeaways on fine j- j- uh, on, on fine-tuning, right? Fine-tuning is a journey, but a very rewarding journey.
- 49:48
There's truly no finish line here. You need to attempt fine-tuning after you already tried other techniques like prompt engineering, retrieval augmented generation. But once you en- decide to embark, data is very important, collecting your data set, uh, because your model is only as good as the data it's trained on.
- 50:06
You need to make sure to continuously monitor quality to retune your model as needed. You also need to, um, you know... But thankfully, we have solutions like OctoAI and OpenPipe to really make this more approachable and easy to do, and it's easier than ever.
- 50:21
It's only gonna get easier, but maybe a year ago, it was only reserved for the most adventurous and sophisticated users, and now we've really lowered the barrier of entry.
- 50:30
And when you do it right, you can achieve really significant improvements in accuracy as well as great reduction in costs. I wanted to thank you for sitting here with me over the last fifty minutes.
- 50:41
I wanna reiterate a few calls to action, so go to OctoAI.cloud to learn how to use our solutions and endpoints, but also come to our booth. And, uh, so we're located at, uh, this, uh, G7 booth, and we're gonna be here today and tomorrow if you wanna chat about our SaaS endpoints, about our ability to deploy in
- 51:01
an enterprise environment. And also I wanna give a shout-out to my colleague here, Pedro. If you're curious about all the know-how that goes behind how we optimize our model in production, 'cause our background is in compiler optimization, is, is in system optimization, infrastructure optimization.
- 51:17
We've applied all of this to be able to serve our models, you know, with positive margins. We're not doing this at a loss. All right? We're not wasting our VC money here.
- 51:26
We're actually building all this know-how into making sure that AI inference is as efficient as it could be. So there's gonna be a talk on that. And also make sure if you, if you get a chance, assuming you've joined our,
- 51:42
our, uh, Slack channel, which is the following one. So if you're on the Slack
- 51:50
org for the event, go to LLM Quality Optimization Bootcamp. You can ask us any questions, and if you fill out the survey that Pedro's gonna post, we're gonna give you an additional ten dollars in credits.
- 52:02
Uh, so that doesn't seem like a lot, but that's a ton. You know, if it's fifteen cents per million tokens, that's a lot of, uh, tokens that you can generate for free.
- 52:11
So we can give you an additional ten, uh, dollars for, uh, filling out the survey, which, which should take about, you know, twenty to thirty seconds. So I'm gonna be around, and also you can find me at the booth this afternoon in case you have any questions.
- 52:24
But I'd like you all, uh, to thank you for sitting through this talk, and hopefully, hopefully you've learned something from this, and hopefully you feel like I've, I've demystified this idea of trying fine-tuning on your own.
- 52:35
Give this notebook a try, assuming, of course, we've fixed this, uh, LoRA upload issue. And, uh, yeah, thank you all, and, and maybe ask me some questions after this, uh, after this talk.
- 52:44
Thanks. [outro music]