LLM Quality Optimization Bootcamp

Talk transcript

1. 0:00

   [upbeat music] So welcome everyone. Thanks for making it to this Lunch and Learn.

2. 0:17

   Uh, my goal today is to make sure that I get to, uh, share my knowledge and experience on LLM fine-tuning. Uh, just to get a quick sort of poll from the audience here, how many of you are...

3. 0:32

   have heard of the concept of fine-tuning here? Okay, so quite a few people. How many of you have actually had hands-on experience in fine-tuning LLMs? Okay. All right. That's pretty good.

4. 0:44

   Uh, that's more than I'm usually used to. I mean, this is quite, uh, fantastic that in this conference, the makeup of AI engineer is close to a hundred percent.

5. 0:53

   That's not, uh, something I'm, I'm generally used to when presenting at other, uh, you know, hackathons and, and conferences. So I feel like I'm, I'm speaking to the right crowd.

6. 1:02

   So just to kinda contextualize this talk, really, I'm trying to address two pains that a lot of GenAI engineers face. And to get a sense of where you are in your journey, how many really identify and can relate to the first one, which is, "My GenAI spend has gone through the roof"?

7. 1:24

   Okay. Yeah? All right. And how many of you are in this other segment of this journey, which is, you know, you've built POCs, it's showing promise, but you haven't yet quite met this quality bar to go to production?

8. 1:39

   Can I get a sense of... All right. So, so I think, you know, we have a good amount of, uh, a good fraction of the audience that can relate to one of these two problems.

9. 1:48

   Uh, myself, I'm a co-founder at OctoAI, and I'm gonna talk a little bit more about what we do. But the customers I've been working with, they feel those pains in very real way.

10. 1:58

    We're talking about tens of thousands, if not hundreds of thousands of dollars in monthly bills, and perhaps even having issues trying to go to production because the quality bar hasn't yet been met.

11. 2:12

    So the overview of this, uh, fifty-minute talk, uh, is going to be spent on understanding the why of fine-tuning. Really try to understand when to use fine-tuning. It's not really a silver bullet for all the problems you're gonna face, but when used right, in the right context for the right problem, it can really deliver results.

12. 2:31

    I'm also gonna try to contextualize this notion of fine-tuning within the crawl, walk and run of LLM quality optimization because there's different techniques that you should attempt before trying to do fine-tuning.

13. 2:43

    But finally, when you're convinced that this is the right thing for you, I'm gonna talk about this continuous deployment cycle of fine-tuned LLMs. So we're going to go through today over a whole crank of that wheel of this deployment cycle composed of, you know, model, uh, you know, dataset collection, model fine-tuning, deployment and evaluation.

14. 3:05

    And really, I'm trying to demystify this whole journey to you all, 'cause in the next fifteen minutes, we're actually gonna go through this whole process, and hopefully that's something that you're gonna feel comfortable going through and, you know, applying to your own dataset, to your own problems.

15. 3:18

    And so for illustrating today's use case, we're gonna use this, uh, personally identifiable information redaction use case. Now, that's a pretty traditional sort of data scrubbing type of, uh, application, but we're gonna use LLMs, and we're gonna see that we can essentially achieve state-of-the-art accuracy, uh, while keeping efficiency at, at the highest, uh, using essentially very compact,

16. 3:42

    very lightweight models that have been fine-tuned for that very task. So again, trying to motivate this talk, what limits GenAI adoption in most businesses today based on the conversations that I've had in the field, discussions I've had with customers and developers?

17. 3:57

    The first one is there's a limited availability of GPUs. I think we're all familiar with this problem. It's one of the reasons why NVIDIA is so successful lately. I mean, everyone wants to have access to those precious resources that allow us to, to run GenAI at scale, and that can also drive costs up, right?

18. 4:16

    So we have to be smart about how to use those GPU resources. And also, uh, when people build POCs, it displays and shows promise, but sometimes you don't reach the expected quality bar to go to production.

19. 4:30

    And so on this X, Y axis, you know, on this chart where Y-axis is cost and the X-axis symbolizes quality, maybe p- many people start, uh, on that green cross here, right?

20. 4:44

    On this upper quadrant of very high cost, maybe not having met the quality bar. That's your first POC. But really, to go to production, you need to end on the opposite quadrant, right?

21. 4:54

    Lower cost, higher quality, where you've met the bar, you're able to run this in a way that essentially is margin positive. And many of us are on this journey to reach that point of, uh, you know, profitability.

22. 5:08

    So we're gonna learn today how to use and how to fine-tune an LLM. Now, fine-tuning is a method that we're gonna use to improve the LLM quality. But as a bonus, we're gonna be also showing how to improve quality significantly.

23. 5:24

    And I use quality as the title of this talk because really, I think many of us AI engineers really care about reaching the high quality bar when we're using LLMs, and hopefully, um, you know, the goal of today's talk is to instill you with some knowledge on how to tackle this journey.

24. 5:40

    And so in terms of tools that we're gonna use today, we're gonna use OpenPipe, which is a SaaS solution for fine-tuning that really lowers the barrier of entry for people to run their own fine-tunes.

25. 5:52

    Uh, you don't need hardware or cloud instances to get started. And we're gonna use this to deliver quality improvements over state-of-the-art LLMs.

26. 6:00

    And of course, since I work at OctoAI, I'm gonna also be using OctoAI here for the LLM deployment, and that's gonna be the solution that we're gonna use to achieve cost efficiency at scale.

27. 6:13

    And really the, the key here is to be able to build on a solution that is designed to serve models at production scale volumes.

28. 6:23

    And just to give you a little bit of a sneak peek in terms of the results that we're gonna showcase today after you go through this whole tutorial, and this is something that you're gonna be able to reproduce independently, so, you know, all the code is there for you to go through.

29. 6:36

    We're gonna be able to show that we can achieve forty-seven percent better accuracy at the task that I'm gonna showcase today using this OpenPipe fine-tuning. And by deploying the model on OctoAI, we're gonna achieve this, I mean, it seems kinda ridiculous, ninety-nine point five percent reduction in cost.

30. 6:55

    This is really a two hundred x reduction in cost here from a GPT-4 Turbo to Llama 3. And mostly because this is a much smaller model, it's open source, and we've optimized the hell of this model to serve it cheaply on OctoAI.

31. 7:07

    So I'm gonna explain how this is achieved, but I hope your interest at least has been piqued on those results that you yourself can reproduce.

32. 7:16

    So when to use fine-tuning. Uh, again, it's not really a silver, uh, bullet for all your quality problems. It has its right place and time. So I like to contextualize it within the crawl, walk, run of quality optimization, right?

33. 7:30

    And as GenAI engineers, uh, many of us have embarked on this journey. We're at different stages of this journey. And really, it should always start with prompt engineering, right?

34. 7:40

    And many of you are familiar with this concept. You start with a model, you're trying to have it accomplish a task, and sometimes you don't really manage to see the result you expect to see.

35. 7:50

    So you're gonna try prompt engineering, and there's different techniques of varying levels of so-sophistication. Uh, this talk is not about prompt engineering. So, you know, you can improve prompt specificity.

36. 8:00

    There's few shots at prompting, where you can provide examples, uh, to improve essentially the, the quality of your output. There's also chain of thought prompting. I mean, some of you probably have heard these concepts.

37. 8:10

    But this is where you should get started, right? Make sure that given the model, given those weights, you just try to improve the prompt to get the right results.

38. 8:17

    Sometimes that's not enough. And, uh, there's a second class of solutions which I like to map to the walk stage, um, Retrieval Augmented Generation, right? We've probably seen a lot of talks on RAG today and throughout this conference.

39. 8:31

    So, you know, there's hallucinated results. Sometimes the answer is not truthful. Well, why is that? It's because the, um, the weights of the model that is really the parametric memory of your model is limited to, you know, the point in time at which the model was trained.

40. 8:48

    So when you try to ask questions on data it hasn't seen or information that's more recent than when the model was trained, it's not gonna know how to respond, right?

41. 8:56

    So the key here is to provide the right amount of context. And so this is achieved through, uh, similar research, for instance, in a vector database through function calling to bring the right context by invoking an API, through search, uh, through querying a database.

42. 9:11

    And so this is something that I think many of us AI engineers have been dabbing in in order to provide the right context to generate truthful answer, right? Complement the parametric memory of your model with non-parametric information, and that's RAG in a nutshell, right?

43. 9:25

    So you've tried prompt engineering, you've tried RAG, you've eliminated quality problems and hallucinations, but that's still not enough, right? So what do you try next? Well, fine-tuning, I think, is, uh, the next stage.

44. 9:37

    And again, I'm generalizing a very complicated and complex journey. But in spite of your best efforts, you've tried these techniques for maybe days, weeks, or even months, and you still won't get to where you need to be to hit production.

45. 9:50

    And we're gonna talk about this journey today, right? Fine-tuning. So when should you fine-tune a model? Uh, again, after you spend a lot of time in the first two phases of this, uh, journey, so spending time on prompt engineering, spending time on retrieval augmented generation, and you don't see the results improve.

46. 10:09

    And generally, what helps is whenever you use an LLM for a very specific task, something that's very focused, for instance, classification, information extraction, uh, trying to format a prompt, using it for function calling, if you can narrow the use case, uh, to something that is highly specific, then you have an interesting use case for, um, for applying

47. 10:31

    fine-tuning here. And another requirement is to have a lot of your own high-quality data to work with, because that's gonna be your fine-tuning dataset. That goes without saying. But a model is only as good as the data that it, that the model was trained on, and we're gonna apply this principle here in this tutorial.

48. 10:48

    And finally, I think as an added incentive, oftentimes we're all driven by economic incentive in the work we do. Uh, for those of you who are feeling the pains of, uh, high GenAI bills, whether it is with OpenAI or with a cloud vendor or a third party, well, this is generally a good reason to explore fine-tuning.

49. 11:08

    So we're gonna go over all the steps now that we've kinda contextualized why fine-tuning and when to consider fine-tuning. We're gonna consider all the steps here in this, uh, continuous deployment cycle.

50. 11:20

    It starts with building your dataset, then running the fine-tuning of the model, deploying that fine-tuned LLM into production, so you can achieve scale and serve your, your, you know, your customer needs or internal needs at high volumes, and also evaluate quality.

51. 11:37

    And this is an iterative process. There's not a single crank of the wheel. This is not a fire-and-forget situation because data that your model sees in production is gonna drift and evolve.

52. 11:48

    And so this is something that you're gonna have to monitor. You're gonna have to update your dataset. You're gonna have to fine-tune your model. And I don't wanna scare you away from doing this because it sounds fairly daunting.

53. 11:57

    And so by the end of this talk, we'll have gone through a full crank of that, uh, wheel, and hopefully, you know, it... Through these, uh, SaaS toolings that I'm gonna introduce you to is gonna feel a lot more approachable, and hopefully I'll demystify the whole process of fine-tuning models.

54. 12:14

    So let's start with step one, which is to build a fine-tuning data set. Now, the data of the model has to be trained on ideally real-world, uh, data, right?

55. 12:25

    It has to be as close as possible to what you're gonna see in production. So there's kind of a spectrum of ways to build and generate a data set.

56. 12:33

    Ideally, you build a data set out of real-world prompts and real-world-generated res-- uh, real-world human responses. So for instance, you have customer service, you've logged calls with a customer agent, you have an interaction between two humans.

57. 12:47

    That's a very good data set to work with, right, because it's human-generated on both ends. This is very high quality, but not everyone has the ability to acquire this data set.

58. 12:56

    Sometimes you're starting from scratch, so not everyone has the luxury to start there. There's also kind of an intermediary between real-world and synthetic, where you have real-world prompts but AI-generated responses.

59. 13:08

    And so this is kind of a good middle ground between cost and quality because you're starting from actual, uh, ground truth information that is derived from real data, but the responses are generated by a high-quality LLM, say, GPT-4 or Claude.

60. 13:24

    And actually, OpenPipe is a solution that allows you to log the inputs and outputs of an LLM like GPT-4 to build your data set for fine-tuning an LLM. So this is something that, uh, you know, a lot of practitioners use.

61. 13:38

    And finally, there's the fully synthetic data set using fully AI-generated labels. And oftentimes, when you go on Hugging Face or Kaggle, you'll encounter data sets that have been built entirely synthetically, and that's a great way to kinda get started on this journey.

62. 13:54

    And actually, one of the data sets we're gonna use today is, uh, from that latter category.

63. 14:00

    And of course, I mean, it probably goes without saying, but in case people are not fully, uh, [chuckles] familiar with this notion, you wanna split your data set into a training and validation, um, set because you don't wanna evaluate your model on data that your, um, fine-tune has seen, right?

64. 14:16

    And so, uh, many of you who are ML and AI engineers are already familiar with this, but I just wanna reiterate that this is important. And finally, you know, this is used for hyperparameter tuning, and when you're deploying it and actually testing it on real-world examples, you wanna have a third set outside of training and validation, which

65. 14:33

    is your test set. Now you've built your data set, you're ready to fine-tune your model, and there's a lot of decisions that we need to make at this point.

66. 14:42

    And the first one is gonna be open source versus closed source, right? And so who here, just like raise of hands, is using proprietary LLMs or GenAI models today from OpenAI, Anthropic, uh, Mistral AI?

67. 14:56

    Okay, good amount of crowd. Who here has been using open-source LLMs like Llama, some of the free Mistral AI models? Okay, so maybe a smaller crowd, right? And maybe that's because these models are not as capable and sophisticated, and...

68. 15:12

    But I'm gonna walk you through how you can achieve better results if you do the fine-tuning right. So of course, the benefit of open source, and this is why, you know, I'm, I'm obviously biased, but I'm a open-source advocate, is that you have to-- you, you get to have ownership over your model weights.

69. 15:30

    So one-- once you've done the fine-tuning, you are the proprietor of the weights that are the result of this fine-tuning process, which means that you can choose how you deploy it, how you serve it.

70. 15:40

    This is part of your IP. And I find that this is a great thing for anyone who wants to embark on this fine-tuning journey. With proprietary solutions, you're not quite the owner, or you don't have the flexibility to decide to go with another vendor to host the, the, the models yourself.

71. 15:56

    And so you're kinda locked into an ecosystem. Some people are comfortable with that, others are less comfortable with it. And many of the customers that we talk to, they're very eager to jump on the open source train, but they don't really know how to get started or, uh, you know, where to start on this journey.

72. 16:12

    So hopefully this can, this can help inform you how to take your first steps here into the world of open source.

73. 16:18

    Then there's a question of like, do I use a small model or a large model? Because, for instance, even in the world of open source, you have models that are in the order of eight billion parameters, like Llama 3 8B, and then you have the large models with, uh, Mixtral 8x22B.

74. 16:32

    So this is a mixture of expert model with over a hundred billion parameters. Uh, very different beasts, and we're gonna see even larger models from Meta. And generally, my recommendations here is, well, look, the large models are amazing.

75. 16:45

    They have broader context windows, they have higher capabilities at reasoning, but they're also more expensive to fine-tune and more expensive to serve. And typically, when you have to do a deployment, you're gonna have to acquire resources like H100s to run these models.

76. 17:01

    So generally, start with a smaller model like a Llama 3B. And sometimes you'll be surprised by its ability to learn, uh, specific problems. So that's my recommendation. Start with a smaller Llama 3 8B or Mistral 7B, and if that doesn't work out for you, then move towards, uh, larger and larger models.

77. 17:21

    And today we're gonna be using this Llama 3 eight billion parameter model.

78. 17:25

    There's also different techniques for fine-tuning, and I'm gonna go over this one fairly quickly, but there's two classes of, of fine-tuning techniques. One which is parameter efficient fine-tuning. It produces a LoRA.

79. 17:39

    And the other one is a full parameter fine-tuning, which produces a checkpoint. A LoRA is much more s-- much smaller and efficient in terms of memory footprint. We're talking about fifty megabytes versus a checkpoint that is fifteen gigabytes.

80. 17:53

    And so you can guess that because of its more compact representation, you're able to serve it on a GPU That doesn't require as much onboard memory. And you can even serve multiple LoRAs at the same time, so multiple fine-tunes on a GPU for inference, as opposed to the checkpoints which require dedicated GPU for every single fine-tune.

81. 18:18

    So there's more flexibility in deployment, and we're gonna use that today. We're actually gonna serve these LoRAs, which are the result of parameter efficient fine-tuning, on a shared tenancy endpoint with other users who have their own LoRAs all running on the same server.

82. 18:33

    And that allows us to really reduce the cost of inference. And there is a benefit to checkpoints though, and full parameter fine-tuning, which is that there are more parameters to tune, so it's a more flexible, uh, fine-tuning technique.

83. 18:48

    It allows the model to have... essentially achieve, uh, better results at more expensive tasks like logical reasoning. But for very specialized tasks, which is what we're gonna look at today, like classification or labeling or function calling, a LoRA is just fine.

84. 19:04

    So we're gonna use parameter efficient fine-tuning. And also when you're doing fine-tuning, you have to decide, am I gonna DIY it or am I gonna use SaaS? So I'm sure some of you only like to DIY things, others like the convenience of SaaS.

85. 19:18

    And here I'm not gonna take a side. I think there's some great tools right now to DIY your own fine-tuning, uh, for instance, the open source, uh, project Axolotl.

86. 19:28

    And actually at the conference there's the, the, the creator behind Axolotl who you might be able to catch. Um, and, you know, the challenge here is that you have to find your own GPU resources, you have to understand how to use these libraries even though they're, they're, they're easier than ever, uh, to, to adopt, and you have

87. 19:45

    to tune and tinker, uh, you know, settings and hyperparameters. Then there's SaaS, which really aim to make it easy to embark on this journey. Companies like OpenPipe, and there's, uh, many folks from the OpenPipe at this conference today, so if you can catch them, please do talk to them.

88. 20:02

    And they're trying to lower the barrier of entry to fine-tuning, right? To make it easy, and they bring all this tooling, all these libraries to make it as seamless as possible to, for instance, move from a GPT-4 model to a fine-tune with, uh, the least amount of steps in, in collecting your data, fine-tuning, et cetera.

89. 20:17

    And so we're gonna use SaaS today, but if you feel more comfortable in this journey, uh, you might s- wanna start with SaaS and then evolve into DIY-ing it.

90. 20:27

    When it comes to deployment, you have to navigate the same options, right? Once you have a fine-tuned model, now you need to decide, well, how am I gonna serve it, right?

91. 20:33

    Because I need to generate maybe thousands, millions, or billions of tokens a day. And so you need infrastructure, you need GPUs, you need inference libraries. Some people like to DIY it using libraries like vLLM, MLCLLM, TensorRT-LLM, Hugging Face TGI.

92. 20:51

    If these are all things that you might have heard of, uh, these are all solutions to run models on your own, on your own infrastructure.

93. 21:01

    But you need to provision the resources, you need to build the infrastructure to scale with demand, and that can get tricky, especially achieving high reliability under load. That's a challenge that many people face as they scale their business up.

94. 21:15

    With SaaS, you can essentially work with a third party like OctoAI, and obviously I'm a bit biased again, I work there, so I'm gonna insert a shameless plug for OctoAI, which allows users to get these fine-tunes deployed on SaaS-based endpoints.

95. 21:32

    So endpoints very similar to the ones from OpenAI, for instance, if you're familiar with that or Claude.

96. 21:38

    And, uh, it offers the ability to serve different kinds of customizations as well. And so very quickly I wanna go over the advantages of OctoAI here. First of all, you get speed.

97. 21:49

    So Llama 3 8B parameter model, you get achieve around a hundred fifty tokens per second, and we keep on improving that number because we've been applying our own in-house optimizations to the model serving layer.

98. 22:01

    It also has a significant cost advantage because it costs about fifteen cents per million tokens compared to, say, GPT-4, which costs thirty dollars per million tokens. So that's where the two hundred x comes from.

99. 22:13

    And we don't charge a tax for customization, so whether you're serving the base model or a fine-tune, it's the same cost. There's customization as I mentioned. You can load your own LoRA and serve it.

100. 22:25

     And finally, scale. Our customers-- some of our customers generate up to, uh, billions of tokens per day on our endpoints. I think we're serving around over twenty billion tokens per day.

101. 22:35

     And so we've focused and spent a lot of time on improving robustness. And also worth mentioning, if SaaS doesn't cut it for you, you are working for a Fortune five hundred company or, you know, a healthcare company, uh, banking sector, government, you need to deploy your LLMs inside of your environment, either on-prem or in VPC, we also

102. 22:58

     have a solution called OctoStack. Come talk to us at the booth. So that's it for the shameless, uh, plug section. Let's go over to section four, which is evaluating quality, right?

103. 23:08

     We've talked about dataset collection, fine-tuning, deployment, now quality evaluation. And we could have an entire conference just dedicated on that. I'm gonna try to summarize it into kinda two classes of eva-evaluation techniques that I've seen.

104. 23:23

     First of all, you know, can your quality be evaluated in a precise way that can be automated? For instance, you generated a program or a SQL command that can run.

105. 23:33

     Uh, can you, for instance, label or extract information or classify information in an accurate way? That's a kind of pass or fail scenario, right? Or formatting the output into a specific JSON formatting.

106. 23:46

     This is something that you can easily test as a pass or fail test. And then there's more of the soft evaluation. For instance, if I were to take an answer and say, "Well, which output is written in a more polite or professional way?"

107. 23:58

     You can't really write a program to evaluate this unless you're using an LLM of course, right? But you have to put yourself into maybe two thousand, two thousand... Sorry, two thousand and twenty, two thousand and twenty-one mindset before, uh, GPT was around.

108. 24:11

     Well, it'd be hard to build a program that can assess this, right? So generally you'd need a human in the loop to say which out of, out of A or B is a better answer.

109. 24:21

     Thankfully, today we can use LLMs to automate that evaluation. But keep in mind that, for instance, if you're using GPT-4 to evaluate two answers, well, if you're comparing against GPT-4, it might favor its own answer, and people have seen that in these kind of evaluations.

110. 24:36

     So this is a whole science. I mean, we could have a whole conference just on this. I just wanted to present the high-level, uh, guidelines of this whole cycle of deploying, uh, fine-tune LLMs.

111. 24:48

     And so really there is no finish line. That's what I wanna convey to you all, that going through a single iteration is something that you might have to do on a regular basis, maybe once a week, maybe once a year.

112. 25:00

     It all depends on your, uh, use case and constraints.

113. 25:05

     Now, let's get a bit more practical. Let's switch over to our demo. And so for those of you who came, uh, a little bit late, there's a QR code here that you can scan, and that will point you to our Google Colab.

114. 25:21

     And we also have under Slack... Uh, let me see if I can pull it. If you're in the Slack channel for AI Engineers World Fair, there is this, uh, quality optimization boot camp where you can ask questions here, if you wanna follow along.

115. 25:36

     And so we're gonna go-- we're gonna try to go over the, uh, practical component in the next, uh, twenty-five minutes. I just wanna

116. 25:43

     provide some context here. The use case is, uh, personally identifiable information redaction. We've taken this from a dataset composed by AI for Privacy called PII Masking 200K. It's one of the largest datasets of its kind.

117. 26:00

     It has fifty-four different PII classes or different kinds of sensitive data, like the name, the, uh, email address, a, you know, address of-- physical address of someone, uh, their credit card information, et cetera, et cetera, across two hundred and twenty-nine discussion subjects.

118. 26:17

     So that includes conversations from a customer ticket resolution, uh, conversations with a banker, conversations between individuals, et cetera. What this dataset looks like is as follows. You're gonna have a message, an email.

119. 26:32

     Uh, here we have, you know, something that looks like it came out of an email. That contains credit card information, IP address, maybe even a mention of a role or, or, uh, anything that is essentially personal i-- per-personally identifiable.

120. 26:47

     And I've highlighted those in red because they will need to be redacted. And after redaction, we should get the following text that shows, "Look, here is this information that is now redacted, anonymized."

121. 26:59

     But instead of just masking it, we're actually telling it what kind of category this information belongs to, right? A credit card number, an IP address, or job title. And this is how we're gonna redact this text.

122. 27:11

     So where do LLMs come in? The way we would use it is through function calling. Who here has used LLMs with tool calls or function calls? Okay. So quite a few people, you know, and, um, as, as many of us are aware, this kind of what powers a lot of the agentic applications.

123. 27:31

     So this is a great use case for people who wanna do function calling and are not seeing the results, you know, out of the box from, say, GPT-4, uh, that they would like to, to, to see.

124. 27:41

     And in this case, we're actually gonna see that, that these kind of state-of-the-art models aren't doing quite well at fairly large, uh, and complex, uh, function call use cases.

125. 27:50

     Uh, so to achieve this, uh, redaction use case, we're gonna pass in a system prompt. We're gonna also pass in a tool specification. The system prompt says, "Look, you're an expert model trained to do redaction, and you can call this function.

126. 28:03

     Here are all the sensitive, uh, PII categories for you to redact." And then as a user prompt, we're gonna pass in that email or that message, and then the output is a tools call.

127. 28:15

     So it's not the redacted text, it's actually a tools call to that redact function that's gonna contain all the arguments for us to perform the redaction. Why am I doing this as opposed to spitting out the redacted text?

128. 28:28

     Well, that gives us flexibility in terms of how we wanna redact this text. We could choose to just replace that information with the PII class. We could also completely obfuscate it.

129. 28:40

     Or we could choose to use, for instance, a database that maps each PII entry to a fake substitute so that we have an email that kinda reads normally, except the credit card, the, the, the names, the addresses are all made up, but they will always map to the same individual.

130. 28:59

     And so that allows us to do then more interesting processing on our dataset, right? So that's why we're gonna use function calling here, and let's start to build the dataset.

131. 29:08

     So I'm gonna switch over to our notebook here. Uh, this notebook is meant to be sort of, uh, self-explainable, so there's a bit of redundance-- redundant context. As part of the prerequisites, you're gonna have to get an account on OctoAI and OpenPipe, um, and, and these are the tools that we're gonna use.

132. 29:26

     And if you wanna run the evaluation function, also provide your OpenAI key because we're gonna compare against GPT-4. So we're gonna install the Python packages initially, only OpenAI and datasets from Hugging Face.

133. 29:39

     You can ignore this, uh, pip dependency error here, uh, which happens when you, uh, pip install datasets in a Colab notebook. But that's okay, we can get past that.

134. 29:49

     You can enter your OctoAI token and OpenAI API key at the beginning, and I've already done this. So we're gonna start with the first phase, which is to build a fine-tuning dataset.

135. 29:59

     So we have this PII masking dataset, and I'm gonna show it from Hugging Face, so PII, uh, masking. And you can see what the dataset looks like. It has the source text information.

136. 30:12

     As you can see, these are, you know, excha-- you know, snippets from emails, for instance. You have the target text that is redacted and the privacy mask that contains each one of the PII and the classes associated to it.

137. 30:25

     So this contains all the data, all the information input and labels that we need to build our Uh, our data set for fine-tuning. And so really what we're gonna do

138. 30:37

     is that we're gonna use the system prompt.

139. 30:41

     Here, we're gonna define our system prompt here, which is again, telling the model, "You're an ex-expert model trained to redact information," and here are the fifty-six categories explaining next to each category what that corresponds to.

140. 30:55

     And this is really the beauty of LLM and sort of natural language entry, is that in the old world, when we were doing PII redaction, we had to write complex regular expressions, and here this is all done through just providing a category and a bit of a description here, and the LLM will naturally infer how to do

141. 31:13

     the re-redaction. We're also going to, uh, define the tool to call, right? And so this is done as a, uh, essentially a dictionary, a JSON object, and as you can see, there is an array that contains, uh, dictionaries containing a string and a PII type, and the string is the PII information.

142. 31:34

     The type is essentially one of fifty-six categories that we provide as an enum. So right off the bat, you can see that this tool call is, um, you know, a bit of a large function, uh, specification.

143. 31:46

     And so let's load our data set from Hugging Face. In this case, it's gonna take maybe a few seconds to load in that data set of two hundred thousand, uh, entries.

144. 31:56

     And then what I have in the next cell when I'm downloading this, uh, data set is what I'm gonna use to build my fine-tuning training data set. And here's the thing about fine-tuning, is that to build your data set, you need to make it seem like you've essentially logged conversations with an LLM, right?

145. 32:15

     You're logging the prompts and the responses, 'cause that's how you're gonna fine-tune it. You need to tell it, "This is the input. We have system prompt, uh, tools, specification, user prompt, and here's the, uh, tools call response that I expect to see."

146. 32:32

     And so this cell here just sets it up so that we essentially have each training sample as a message from an LLM that's been logged. We're gonna see what that looks like in a second.

147. 32:44

     So we're gonna build a ten thousand entry training data set for OpenPipe,

148. 32:51

     and that's gonna be downloaded as this openpipe_dataset.jsonl. And so as I run this cell, it's gonna download this, uh, from Colab.

149. 33:02

     And now when you switch over to OpenPipe, we're gonna create a new data set. So once you're on OpenPipe console, you have a project here. I've, uh, generically named it Project One.

150. 33:14

     You can access data sets, and already, as you can see, I already have built a, a few data sets, uh, before, but if you're a first-time user, you're not gonna see anything under data sets.

151. 33:24

     So you can create a new data set here by clicking on this button. And if you go under Settings, we can name our data set, so I'm gonna call it,

152. 33:34

     uh, Lunch and Learn, and today is, uh, June two six. All right, so this is today's Lunch and Learn. I'm gonna, I'm gonna call this my dataset. And under General, I can upload the data that I just downloaded from my notebook, openpipe_dataset.jsonl.

153. 33:54

     So this upload operation is gonna take a few seconds or maybe a couple of minutes, because what's gonna happen o-on OpenPipe is not only we're uploading this data set, but it's gonna do some, uh, pre-processing here to split it into a training and validation set.

154. 34:11

     It's also going to get it all formatted in a nice way so we can essentially look into the data set.

155. 34:18

     Uh, so you can see there's this little window here that shows that you're uploading the data set and that it is essentially being processed. So while this is happening,

156. 34:28

     right, we've prepared our data set, and we're gonna take a look at it in a second while it's being processed on OpenPipe, but let's see how we're gonna do the fine-tuning in the next stage, right?

157. 34:38

     So once we have our data set uploaded, we're gonna have this view on the data set that shows every single entry that we can peek into and how it's split into training and test set, generally a ninety/ten percent split.

158. 34:52

     And from that UI, we can launch a fine-tune, and this is where we get to choose our base model, and what we're gonna choose is a Llama 3 eight billion parameter model with 32K,

159. 35:03

     uh, context width, which is a, uh, fine-tune from News Research called a Theta model. And you can see that there's essentially a pricing here that is being estimated for this fine-tune.

160. 35:17

     We have a substantial training set because it can range from, say, hundreds of samples to thousands to hundreds of thousands, and the cost can scale up as you, um, as you feed in more training samples, but it will improve the accuracy, and it also provides an estimated training price of forty dollars.

161. 35:35

     Now, that might seem like a lot, especially when you're tinkering with fine-tuning. But keep in mind, some of the people that we work with, they tend to spend tens of thousands or maybe hundreds of thousands of dollars a month on GenAI spend, so this is absolutely something that you can do up front that will pay off.

162. 35:50

     And I believe that on OpenPipe, if you get started, you get a hundred dollars credit, uh, so that allows you to, to run some fine-tunes off the bat, uh, without having to necessarily, uh, have to, to pay.

163. 36:03

     So, um, let's go over to OpenPipe. And it is still uploading. I think maybe the network is, uh, [chuckles] is a bit slow. But we're going to essentially start training at this point, and once the training is happening, we're gonna then deploy the fine-tune LLM Uh, when training is done.

164. 36:24

     And what happens on OpenPipe is when you're done with training, you're gonna get an email when that training job is done. It can take a few minutes, so I'm gonna pull a Julia Child here.

165. 36:33

     I'm gonna stick the, you know, the turkey in the oven and, and the second oven, I'm gonna have a, a pre-baked turkey, just so that we don't lose time.

166. 36:40

     But as you're going through this on your own, keep in mind it's gonna take a little bit of time to just kick off that whole fine-tuning process. But it's not that long because, um, you know, you're training a fairly small model here.

167. 36:54

     All right, so this is still, uh, saving.

168. 36:57

     But let's kinda take a look at what we've done so far, right? So we've built our data set using a synthetic data set from Hugging Face. Uh, we format each input/output pair from the data set as logged LLM messages, and this is essentially stored as a JSONL file that we upload to OpenPipe, and we produce ten thousand

169. 37:15

     training samples. We're fine-tuning a model from OpenPipe, and we're-- OpenPipe uses a parameter-efficient fine-tuning, which produces a LoRA, and we choose Llama 3 eight billion parameter model as the base.

170. 37:27

     And when we deploy, what we're gonna use here is OctoAI.

171. 37:32

     So let's see. This didn't finish uploading, so I'm gonna go into the one that I uploaded just a couple days ago, just to essentially show you what you should see on the, uh, user interface.

172. 37:44

     So as you peruse through the training samples, what you're gonna see is an input column and output column. And so on the left, you have the input with the system prompt.

173. 37:54

     As you can see, it's a, it's a big boy, uh, because it has all these different categories, right, that it needs to classify. It also has the user prompt, which is the message that we need to redact, the tool choice, and the tool specification here with all the different categories of PII types.

174. 38:11

     And then the output will be, will be this tools call from the assistance response, and that will have this redact call, along with these arguments, field to redact as a list of dictionary entries containing string and PII-type information, right?

175. 38:27

     And so this is what we've passed into our fine-tuning, uh, data set into OpenPipe.

176. 38:35

     And, uh, this is still saving, so I'm just going to go ahead and go to the model. So

177. 38:42

     once you have the data set uploaded, again, you hit this Fine-tune button, and this is what's gonna allow you to launch a fine-tuning job, right? I can call this blah.

178. 38:51

     And this is where you select under this dropdown the model that you wanna fine-tune. Uh, this is again what we saw before. Training size is substantial. I'm not gonna hit Start training because I already have a trained model.

179. 39:02

     But when you do that, it's gonna kick off the training, and when it's done, you'll get notified by email.

180. 39:07

     Now let's fast-forward. Let's assume I've already trained my model. So I'm gonna have this model here that's been fine-tuned from this data set. I'm gonna click on it. As we can see, it's an Llama 3 8B model.

181. 39:20

     It's been fine-tuned over these ten thousand data sets split into nine thousand, uh, training samples and a thousand test samples. Uh, we can even look at the evaluation.

182. 39:33

     But going back to the model. And the nice thing is that it's taking care of the hyperparameter, like learning rate, number of epochs. It kinda figures it out for you, so you don't really have to tweak those settings.

183. 39:46

     And I find that to be very convenient, especially for people who haven't yet built an understanding of how to tweak those values.

184. 39:52

     And the beauty of using OpenPipe is that you can now export the weights and be the owner of those weights, right? Remember when we talked about open source, it's really important to own the result of the fine-tuning.

185. 40:03

     So you can download the weights in any format you want. You have LoRAs, but also merged checkpoints, so you can have a, a, a parameter-efficient representation as well as a checkpoint.

186. 40:13

     And so we've selected to export our model as a FP16 LoRA, which is what we're gonna use to upload our model on OctoAI, which is what we're gonna use to deploy the model.

187. 40:23

     So now I can download the weights as a zip file, and it's fa-fairly small, only fifty megabytes. But I can also copy the link, copy the URL, and this is what we're gonna need to do in this tutorial.

188. 40:36

     So to deploy the model, what we need to do is copy this URL. I'm gonna download in the cell the OctoAI CLI. This is a command line interface for users to upload their own fine-tunes to what we call our asset library.

189. 40:53

     So this is a place where you can store your own checkpoints, your own LoRAs, for not just LLMs, but also models like Stable Diffusion if some of you are developers who also work in the image gen space.

190. 41:03

     And so we can serve these customized models, uh, on our platform. And so we're gonna upload this LoRA from OpenPipe to OctoAI. So we're gonna log in just to make sure credentials are good, and here we have a confirmation that our token is valid.

191. 41:24

     And in this cell, we have to replace the LoRA URL from set me to that URL that I just copied here from Download Weights.

192. 41:32

     And keep in mind, this might take a couple minutes to get the link to appear. But once you have that link... And again, I'm kinda skipping ahead because when you're gonna run this at your own time, it might take a, you know, a few minutes to run the fine-tune, it might take a few minutes to download the

193. 41:47

     weights. But everything that I'm running here is essentially the steps that you'll take yourself. And what I'm doing here is, uh, passing in this URL here and setting a LoRA asset name in my OctoAI asset library.

194. 42:02

     So I can then create this asset from this LoRA as a SafeTensor file and, uh, based on the Llama 3 8B model. I'm gonna name it, uh, let's see.

195. 42:19

     Seems like something has, uh, failed here. So let's try to run it again.

196. 42:38

     And so what this is doing is, uh, let's see.

197. 42:47

     Usually that, that, uh, that should have worked. [chuckles] So what's, uh, what should happen here is, at this point, once you've taken the, uh, URL of your fine-tune asset,

198. 43:01

     should be able to host it on our asset library, and then from there, serve it to start running some inferences. So this, uh, [laughs]

199. 43:12

     this, this LoRA upload step didn't quite work here. So Pedro, are you able to maybe double-check with product whether this capability is working?

200. 43:23

     Uh, this isn't a good demo unless something fails and... [laughs] So, uh, yeah, you know, I just tested it earlier today and it was working flawlessly. So, uh, let's see.

201. 43:36

     I might have to list my assets so I can pull an old one.

202. 43:41

     Um, actually, one second. Pedro, can you, can you tell me what the, uh, command is to, to list the assets that are on... I think it might be OctoA- OctoAI asset list.

203. 43:55

     All right, let's... Okay, there we go. So I'm gonna pull from an asset that I uploaded earlier.

204. 44:10

     It could be the third one. The third one?

205. 44:16

     All right. Yeah, because... All right, so I'm gonna take an asset that I uploaded earlier. I'm gonna have to look into why, uh, that step failed, but, uh, let's,

206. 44:34

     let's try this. Okay, so I'm gonna use an asset that I uploaded earlier. I'm not sure why this didn't work, but I'll make sure that this is working for you all to reproduce this step.

207. 44:45

     And I'm gonna set LoRA asset name equals this. All right, so these other LoRAs I uploaded using the exact same steps as I used for this, uh, tutorial.

208. 45:00

     So we'll make sure to get to the bottom of this and, uh, we'll use the Slack channel here for folks who want to run through this, uh, step. But I'm just gonna run an example inference here on this asset that I pulled from OpenPipe.

209. 45:13

     And so again, we have our, uh, system prompt. We're gonna pass in this, uh, ex... you know, this message, this email as our test prompt. And then when we're invoking this OctoAI endpoint, we're using the standard chat completions, uh, from OpenAI.

210. 45:30

     And what we're passing here is this OpenPipe Llama 3 8B-32K model, and we pass in this argument for parameter efficient fine-tune, and pass in the LoRA asset name that we just uploaded to the asset library.

211. 45:47

     And as we can see, the response here contains the tool calls and the call to the function that will do the redaction. So this is behaving exactly as we intend it to.

212. 45:57

     So now we can move on to the quality evaluation. For quality evaluation, what we've done is use essentially an accuracy metric. Thankfully, we have a ground truth, right, from our dataset.

213. 46:09

     All the exchanges have been labeled with privacy mask information that we can use as ground truth. So that makes evaluating scoring or results fairly easy. We don't have to use an LLM, for instance, for that.

214. 46:21

     We can actually use more traditional techniques of accuracy evaluation. And so we have a metric that we've built. It assigns a score that can be penalized when PII information was missed or mistakenly added, i.e., false negative or false positive.

215. 46:37

     And then we use a similarity distance metric to kinda match the responses from the LLMs compared to our ground truth. So for illustration purposes, we have, for instance, this PII information that's been redacted.

216. 46:50

     That's a score of one point oh because it's the perfect match. Our fine-tune might, for instance, miss the fact that Billy was the middle name and might interpret it as first name.

217. 47:00

     In that case, we're still attributing a high score because it's close enough, and probably for a practical use case, that would be good enough. But for instance, upon calling GPT-4, it fails to identify two out of the three information that we had to redact, and so the score is about a third here, right?

218. 47:17

     So in this case, what we're gonna do here, I'm just gonna reduce the test size to a hundred samples, and I am going to run this evaluation inside of, uh, this cell.

219. 47:30

     It's gonna bring us a hundred test samples that we can then run our evaluation metric and get our overall scoring out of. Uh, so if we look at, you know, the, uh, output from the cell, essentially, we're just evoke, invoking back-to-back the fine-tune running OctoAI, and we're invoking GPT-4 on

220. 47:55

     OpenAI to do the results collection. So we're gonna collect some results here.

221. 48:02

     And, uh, once we've collected the results, once we get to a hundred, I think we're getting pretty close here, we can run the quality evaluation metric. And of course, I invite you to run it on more samples, maybe a thousand or ten thousand.

222. 48:14

     Uh, it just gets more expensive as you're using GPT-4. You know, to run a hundred samples, it costs about a dollar in inference. So, uh, then a thousand samples costs, uh, ten dollars.

223. 48:27

     And now we're gonna score it. All right, so we're gonna go through every single entry. We have our ground truth information. We have our...

224. 48:35

     Eval and labels from GPT-4 and our eval and labels from our fine-t-tune. And we can see that right off the bat, the fine-tune is actually better at finding the PII to redact.

225. 48:47

     Here, GPT-4 scored only a score of, uh, zero point four nine, whereas our fine-tune achieved zero point eight five, and here zero point three for GPT-4, one point zero for the fine-tune.

226. 48:58

     So the fine-tune overall is performing better, and once we aggregate and average the score, GPT-4 achieved zero point six eight out of one, whereas our fine-tune achieved zero point nine seven.

227. 49:12

     And so that's the difference between prototype and production, right? You're expected to achieve somewhere in the single nine or two nines of accuracy, and this is what this technique shows, uh, allows you to achieve.

228. 49:23

     And again, I want to reiterate that in terms of cost, GPT-4 costs upwards to thirty dollars per million tokens generated, whereas Llama 3 8B on OctoAI costs just fifteen cents.

229. 49:34

     That's a two hundred x difference, right? So with that, I just want to conclude with some takeaways on fine j- j- uh, on, on fine-tuning, right? Fine-tuning is a journey, but a very rewarding journey.

230. 49:48

     There's truly no finish line here. You need to attempt fine-tuning after you already tried other techniques like prompt engineering, retrieval augmented generation. But once you en- decide to embark, data is very important, collecting your data set, uh, because your model is only as good as the data it's trained on.

231. 50:06

     You need to make sure to continuously monitor quality to retune your model as needed. You also need to, um, you know... But thankfully, we have solutions like OctoAI and OpenPipe to really make this more approachable and easy to do, and it's easier than ever.

232. 50:21

     It's only gonna get easier, but maybe a year ago, it was only reserved for the most adventurous and sophisticated users, and now we've really lowered the barrier of entry.

233. 50:30

     And when you do it right, you can achieve really significant improvements in accuracy as well as great reduction in costs. I wanted to thank you for sitting here with me over the last fifty minutes.

234. 50:41

     I wanna reiterate a few calls to action, so go to OctoAI.cloud to learn how to use our solutions and endpoints, but also come to our booth. And, uh, so we're located at, uh, this, uh, G7 booth, and we're gonna be here today and tomorrow if you wanna chat about our SaaS endpoints, about our ability to deploy in

235. 51:01

     an enterprise environment. And also I wanna give a shout-out to my colleague here, Pedro. If you're curious about all the know-how that goes behind how we optimize our model in production, 'cause our background is in compiler optimization, is, is in system optimization, infrastructure optimization.

236. 51:17

     We've applied all of this to be able to serve our models, you know, with positive margins. We're not doing this at a loss. All right? We're not wasting our VC money here.

237. 51:26

     We're actually building all this know-how into making sure that AI inference is as efficient as it could be. So there's gonna be a talk on that. And also make sure if you, if you get a chance, assuming you've joined our,

238. 51:42

     our, uh, Slack channel, which is the following one. So if you're on the Slack

239. 51:50

     org for the event, go to LLM Quality Optimization Bootcamp. You can ask us any questions, and if you fill out the survey that Pedro's gonna post, we're gonna give you an additional ten dollars in credits.

240. 52:02

     Uh, so that doesn't seem like a lot, but that's a ton. You know, if it's fifteen cents per million tokens, that's a lot of, uh, tokens that you can generate for free.

241. 52:11

     So we can give you an additional ten, uh, dollars for, uh, filling out the survey, which, which should take about, you know, twenty to thirty seconds. So I'm gonna be around, and also you can find me at the booth this afternoon in case you have any questions.

242. 52:24

     But I'd like you all, uh, to thank you for sitting through this talk, and hopefully, hopefully you've learned something from this, and hopefully you feel like I've, I've demystified this idea of trying fine-tuning on your own.

243. 52:35

     Give this notebook a try, assuming, of course, we've fixed this, uh, LoRA upload issue. And, uh, yeah, thank you all, and, and maybe ask me some questions after this, uh, after this talk.

244. 52:44

     Thanks. [outro music]