The GenAI Maturity Curve (or: You Probably Don’t Need Fine-Tuning)

Talk transcript

1. 0:00

   [upbeat music] Hi, everyone. Uh, my name is Kyle Corbitt.

2. 0:16

   Um, thank you for the introduction. Uh, I work for a company called OpenPipe, which is a fine-tuning platform. Um, the title of my talk is that you probably don't need fine-tuning.

3. 0:27

   This is an awkward title. Um, given that I am doing a fine-tuning platform, there's a little bit of a catch here. Uh, this is a little bit better. Put a question mark there.

4. 0:37

   Maybe you don't need fine-tuning. Um, all right, I'm gonna take it one more thing. Do you need fine-tuning? All right, let's start from this basic, we're not assuming anything, and we're gonna be talking about that in the presentation.

5. 0:48

   So, um, as we're going through this presentation... Now,

6. 0:52

   going back a couple slides, um, I actually do believe, despite the fact that I am in fact working on a fine-tuning platform, that my guess is that most people in this audience, as of right this moment, probably do not need fine-tuning.

7. 1:04

   And we're gonna be talking about in this presentation kind of what the maturity curve looks like and where the right moment is for you to bring that into your infrastructure and whether it even makes sense to do that, um, in your particular case.

8. 1:14

   Um, but my hope is that by the end of this presentation, um, a few things have happened. One is, um, you have a better que- answer to that question for yourself.

9. 1:21

   Um, does this make sense for me? Is this gonna apply? And then just as importantly, um, if it does make sense, what is the right moment? Um, how will I know that this is something that, like, I, I, I should be doing that's actually going to help my company, um, you know, whatever I'm trying to do, uh,

10. 1:35

    get, get the outcome I'm looking for? Okay, um, splitting this up into three pieces. Um, one is why, okay? So kind of addressing the, the title of the talk, um, do you need fine-tuning?

11. 1:47

    Um, we're gonna talk about what are the benefits, like, why, why is this even something you would put time into? What potentially could you gain from it? Um, and then the second one is when, uh, again, at what moment in the process does it make sense, uh, to, to start doing this, if it does?

12. 2:01

    And then finally, how? Um, like, what are the actual steps you take? Kind of like, what is the, what is the general process, and even some specifics on how to do that.

13. 2:09

    Um, hopefully in that last section we find that it's actually maybe easier than you thought. Uh, that's usually the case when, when I talk to people, uh, who have not done it before.

14. 2:19

    Okay, first we're gonna talk about why. Um, so anyone-- This, this is, this is a graph we're looking at, uh, you know, the, the quality of output ac-across several tasks we averaged, um, to generate this.

15. 2:30

    And then on the, on the x-axis, we're looking at kind of tokens per dollar, basically the cost-effectiveness of this model. All right, um, can anyone... If you've seen a graph kind of like this before, raise your hand.

16. 2:41

    This is, this, this is a pretty common, um, kind of way to, to look at this. And I, and I suspect many of you who are, um, practitioners and are using LLMs, you've, you've, um, maybe even generated a graph like this on your own specific use case.

17. 2:52

    Maybe you've evaluated a few different base models. You've looked at, um, either on your own evaluations or some kind of vibe check or something to figure out, okay, um, for this mo- for, for these different models, uh, how well does it do on my task?

18. 3:03

    And, um, and you compare them. Um, and usually, uh, there's a correlation, at least at, at sort of like the frontier level, between how expensive a model is and how good it is.

19. 3:13

    Um, typically, the larger a model is in number of parameters and the more time and effort that has gone into training it, um, the higher quality outputs it's able to produce.

20. 3:21

    This is, this is all, um, pretty basic stuff. Um, and then of course, as, as you get into smaller models, uh, or, or tr- models that haven't had as much training time put into them, then, then maybe the response quality on sort of a zero-shot basis is, is not quite as good.

21. 3:34

    Um, okay, all, all pretty basic stuff. Um, one thing you'll notice is, uh, is that you can sort of think of this as like a proto-frontier, right? Where it's like you f- if you figure out how good a quality you need, then you can, from that number, sort of like backfill, all right, which, uh, which model hits

22. 3:50

    that bar that's at, at the lowest price point. Or alternatively, if, um, maybe you've got some, some constraint on your unit economics or like, you know, how much you wanna spend on this project, um, and so then you, you work from the other direction and say, "Okay, I, I can't spend more than this.

23. 4:03

    What's the best model I can get?" And so this, this frontier kind of maps, all right, given, given a certain constraint, um, what should I choose?

24. 4:11

    Um, there's, uh... This, th-this next slide is a little bit of an aside, but it's actually very important for, for where we're getting to next. Um, one way you can push this proto-frontier, especially kind of the low end in the sense of like the very expensive but high quality end, is, um, by basically using more compute at

25. 4:28

    inference time. So this is, um... I put a gra- an example on the graph. This is actually just a very, like, particular way of doing that. It's, it's called a Mixture-of-Agents approach.

26. 4:36

    Um, Together AI published, uh, some papers, and we've also done some research in that area. Um, but this is-- You can think of this as like a very specific proct- prompting technique that uses extra time at inference.

27. 4:45

    You're, you're going through several layers and kind of like improving your answer, and then eventually you come out with something that's higher quality than in this case you would've gotten with GPT-4 by itself.

28. 4:52

    Okay, so why is this important? Um, because it actually does prove it's possible to, to move outside this frontier, um, if we're a little bit smart of how we do it, in this case by using more prompting.

29. 5:02

    Now, this particular approach, um, there's some downsides. As you can see, it's all the way over on the, uh, left side of the graph, which means it's ex-extremely expensive 'cause you're doing all these calls, um, but you get higher quality.

30. 5:13

    Okay, that's cool. But if, um, a lot of people can't use that directly, again, because of those constraints where it's, it's just really expensive, latency's really high, you're doing multiple prompts, um, and so that by itself maybe isn't going to, going, g-going to be that interesting.

31. 5:25

    What is interesting is if you then take, say, you've found some way like using this Mixture of Experts to get-- or sorry, Mixture of Agents to get really high-quality data, um, that is the moment, uh, where fine-tuning can really help you.

32. 5:39

    And I'm gonna put up some more dots on this graph. Um, okay, so let's take a look at what I just put through up here. So this-- And by the way, this graph, th-this, th-these are not, um, this, these are not conceptual.

33. 5:50

    Like, this is actually, uh, you know, like this is, uh, evaluations that we ran internally on real customer data sets, um, using the evaluations our customers had defined. And, uh, and we used four different data sets.

34. 6:01

    We averaged it. The quality is, um, an LLM-as-a-Judge, uh, and then the tokens per dollar is, is basically just like a industry standard, uh, pricing for those models. So, so this is based in fact.

35. 6:11

    Um, this is not a, a theoretical graph.

36. 6:13

    And what I've put up here are three additional models, um, which are fine-tuned versions, and two of them are actually fine-tuned versions that are also on the graph. So if we look, yeah, on my next slide, you can actually see through the fine-tuning process for, for Llama 370B and then Llama 38B, we were able to move the

37. 6:29

    performance on, like, the average across these four tasks that we tested on up substantially. And not only are they much higher than they were from just prompting, they're actually in both those cases higher than any of the base prompted models.

38. 6:41

    And that's really important because, um, what we're doing here is we are actually moving the Pareto frontier outwards, right? So it is a different trade-off. If you're willing to do fine-tuning, if you're willing to put in the effort to create a fine-tuned model, you can actually move that Pareto frontier outward and get much more bang for your

39. 6:57

    buck. So whatever the amount of money you're willing to spend on inference is, um, you can, you can get just much higher quality completions, um, than you would have without this.

40. 7:06

    And that basically in a nutshell is the pitch for fine-tuning, is that it gives you this ability to move outside that Pareto frontier and for whatever budget you have, um, you end up, uh, with, with a better model.

41. 7:18

    Um, so anyway, I've, I've redrawn here. We have, we have a new Pareto frontier, um, you know, which, which kind of takes this into account. Um, and so now all of a sudden the best model you're gonna get at any price point is gonna be one that you fine-tuned if, if you're willing to go to that work.

42. 7:32

    So, um, you know, actually I'm gonna go... I, I, I really like this, this transition, just go back and forth, you know? It's, like, better, right? Cool. Everybody got, got it?

43. 7:42

    Okay. So, so this is awesome. Hopefully I've convinced you, like, you know, moving that outward is better. Whatever your price point is, um, you're, you're gonna get a better completion.

44. 7:50

    Um, okay, so this is awesome, right? Like, Pareto frontier, like, by definition means that y- we've expanded it. No matter where you are on that, um, on that point, you're getting a better completion.

45. 7:58

    There are no trade-offs, right?

46. 7:59

    Whoo. [clapping]

47. 8:00

    That's right. Thank you. Okay, I lied. There are trade-offs. There's always trade-offs. Um, the trade-off that you typically run into with fine-tuning is there, there's a couple of different things.

48. 8:12

    Um, the biggest one honestly is just the time upfront it takes, right? Like, this is an extra thing. Um, you've gotta actually do it. You've gotta gather your data set.

49. 8:20

    You've gotta make sure that the, the model's working. Um, and that is a real cost. That takes some time. Um, and then the second one is, is you get...

50. 8:27

    You lose a little bit of flexibility. Um, when you fine-tune a model, what you're doing is you are making it really, really good at one specific thing, um, but it's not necessarily really, really good at everything else.

51. 8:37

    In fact, by definition it's, uh, it's not good at everything. Um, that's, that's sort of like exactly what we're trying to achieve by fine-tuning. And so, um, because you're losing some of that flexibility, it's really important that you understand before you go into that fine-tuning process what your task actually looks like, uh, because you wanna be able

52. 8:54

    to specialize on that task and get really great performance on that task, and you wanna make sure that when you're actually using this model at inference time, um, you aren't sending it a bunch of stuff that's kind of out of domain that is not related to what it was trained on because you're not gonna get that same

53. 9:07

    performance boost on, on anything that was out of domain.

54. 9:10

    So that's the primary trade-off. Um, that's, that's the reason why maybe it's not right to, to fine-tune immediately. And so that brings us to our second of the three sections we're gonna talk about.

55. 9:19

    And the second one is like, okay, when? How do I make this decision? Um, how do I know when the right moment is to invest the time to do this?

56. 9:27

    Um, and there is a little bit of sort of like, uh, s- you know, uh, specificity, but, um, there are generally three different reasons why you might want to move from a prompted model to a fine-tuned model.

57. 9:39

    Um, and the three reasons are, first of all, cost, um, if you just can't deal with the unit economics. If you're using GPT-4 in production, that's gonna be really expensive.

58. 9:47

    Um, the second reason is latency. So as we're moving down towards these smaller and cheaper models, we're also... You know, one advantage of ha- being a smaller model is there's just, like, less multiplications, less math going on for every single generation, and that means you can actually bring your latency floor much lower than you can with a

59. 10:02

    large model like, for example, a GPT-4. Um, so cost, latency, and then the final one is actually the, uh, the, the quality, um, particularly the consistency of the quality.

60. 10:11

    So if you find that GPT-4 maybe 80%, 90% of the time is doing a great job and it is solving your task well, but that last 10, 20%, you know, it makes mistakes, it doesn't get the format right, it forgets one instruction, um, typically those kind of issues where it, it usually gets it right but not consistently,

61. 10:26

    um, a fine-tuned model, that's the kind of thing that fine-tuning excels at, and you're gonna very consistently get the, get the output you're looking for, um, at a minimum much more consistently than you would just purely through prompting.

62. 10:36

    So the way I like to think about it is, um, for almost every application, not, not universally, but for most applications I, I see people using, um, if you can get away with a prompted model to start with, um, you should do that.

63. 10:48

    Um, you, you, you should just eat the cost. It- this is, like, as you're first deploying it, you're probably not spending that much money as you're just testing it, um, and you're gonna benefit from the iteration speed you get from being able to, uh, to just change the prompt and change the behavior.

64. 11:03

    Once you've figured out, okay, this is the, this is the product shape, this is like, kind of like, you know, um, what we're actually gonna be deploying at scale, then it's worth taking, you know, the, the couple of hours you're gonna have to invest to go from that prompted model to a fine-tuned model.

65. 11:17

    Um, and, and at that point you're gonna start getting those benefits from way cheaper, way faster, and higher quality completions. Um, so that's, that's basically the, the way I think about that.

66. 11:26

    Okay, so we have six minutes left. I'm going to take six minutes and teach you how to fine-tune a model, okay? And if you have not done this before, it might feel intimidating.

67. 11:35

    I'm hoping by the end of this six-minute section it is no longer intimidating because honestly, we have had engineers that come in with no experience with machine learning, no experience with data science, and have a successful fine-tuned model that is outperforming their base model they're coming from, um, with less than an hour of work.

68. 11:49

    Um, so hopefully this is something that, that, uh, we come out of this with, with a good understanding at least, um, of what you need. Okay, so yeah, if, if you can prompt, you can fine-tune.

69. 12:00

    Uh, it's, you know, there, there's, uh, we can walk you through this process. All right. Four steps. You have to get the data. You have to actually train the model.

70. 12:08

    You probably should evaluate. Technically step three is optional, but [laughs] um, I think, uh, we all, we all know, uh, on this track that evals are important. Um, and then you have to deploy the, the model Okay, so preparing data.

71. 12:20

    One of the reasons why it's extremely, um, useful if you can get away with it to start with a prompt model like, say, GPT-4 in production is because it solves, like, a huge class of problems that anyone who has a background in traditional machine learning is very familiar with, which is it helps you figure out what your

72. 12:37

    actual input distribution is before you have to, like, actually train the model. Um, if you're deploying GPT-4 in production, you can actually see, okay, what are my users using this for?

73. 12:47

    Um, like, what are the areas where, where, like, I-- it was unexpected, but, like, this is something people are doing. And you can actually capture that distribution really well.

74. 12:54

    You just have to log the prompts, um, and then you can use that to know exactly what your model has to be good at. So this is actually... When people talk about how hard training models, like, used to be in the dark ages, um, this was actually where most of the effort was, was figuring out what, what

75. 13:08

    does the data actually have to look like? Like, what does my input look like? What do the good outputs look like? Um, and if you can get away, and not every problem can, but most can in my experience, if you can get away with starting with, um, a prompt model like GPT-4, then you're in a great spot

76. 13:20

    because you can very accurately capture that input distribution. You know what your model's gonna need to do, and you can just train a model that's really, really good at doing that.

77. 13:28

    Um, we do that, uh, first part with an OpenPipe. It's-- Don't overthink this step, by the way. I- I've talked to tons of people who just, like, in whatever database you're using on your application or whatever observability or logs provider you're already using, you just, every time you do a prompt, you throw it in there.

78. 13:41

    Um, the important thing is you're just, you're capturing that somewhere. Um, and then, and then you can go ahead and use that to figure out what your, what your data actually looks like.

79. 13:49

    Um, and yeah, this is, this is what it looks like within OpenPipe. Okay, once you've captured that, the next step is to make sure your data set is as high quality as possible, and the amount of work that needs to go into this step is pretty variable.

80. 14:00

    Um, if you're lucky enough that, like, GPT-4 is doing a great job on your problem already and you're really just trying to decrease latency or cost, um, then you're in a really great spot.

81. 14:09

    You can literally just use that exact data that you've been generating from just your normal production, um, and, and you can fine-tune and do great. Um, if there-- it is making mistakes, there may be extra steps required.

82. 14:19

    You may have to do some human review, um, or you may have to do some relabeling with, say, a stronger model. Um, this is something I, I mentioned that that Mixture-of-Agents thing that I mentioned earlier, this is something we've seen a lot of our customers be successful with, is basically going through multiple iterations.

83. 14:32

    It's still fully automated, still with an LLM, but it can solve many of those, uh, reliability issues by going through multiple iterations as part of the labeling. Of course, it's expe- more expensive because you're doing this multiple times and it's slower, but this is for generating training data.

84. 14:45

    You don't have to do it on, on that many, so it's, it's still very manageable. Anyway, um, once that data's prepared, then you actually have to train the model.

85. 14:52

    Um, this part I feel like can be pretty intimidating to people. This is what people think, uh, if they, if they're coming in without familiarity with fine-tuning, this is what people think is gonna be the hard part, is figuring out the hyperparameters.

86. 15:02

    You know, like, how do I actually train it? How do I get the infrastructure? Um, honestly, at this point, like, I feel like hon-- this, this is mostly a solved problem.

87. 15:09

    Um, of course, like, uh, we do it just, like, with one click on OpenPipe, um, but there's lots of other providers out there. I've listed some that also do fine-tuning.

88. 15:16

    It's quite simple. There's also some pretty easy-to-use at this point open source libraries you can use, um, that I've listed on the other side as well. And, um, and ultimately, like, this part is actually the, the infrastructure and tooling has gotten good enough, the architectures involved have gotten, like, standardized enough that you don't really have to do

89. 15:33

    that much custom work. Um, so, so the, the training part is actually relatively easy. Um, and then you want to evaluate. So evaluations, obviously there's gonna be a lot of discussion on that on this track.

90. 15:45

    Um, really quickly, I like to think of evaluations as being in two categories, um, which I call the inner-loop and the outer-loop evaluations. Inner-loop evaluations just means something you can run really fast and get feedback on fast.

91. 15:56

    So what we do at OpenPipe is we use LLM-as-a-Judge evaluations. We let you define, uh, you know, the criteria, and then you can run that evaluation on your fine-tune output versus your data set and see that, um, you know, really quickly, does this seem right?

92. 16:09

    And then there's the outer-loop evaluations, which are incredibly important as well, and that is gonna be very business specific, very use case specific, but it's basically, what do I care about?

93. 16:18

    How do I measure it? Is this actually achieving that? And usually that requires actually deploying your model or at least doing some kind of A/B testing or something like that in production and seeing, is this driving-- is, is this doing what, um, you know, what, what I'm expecting it to do?

94. 16:32

    Okay. Within, um, OpenPipe, we do have that inner loop really strongly integrated. We can-- You can define LLM-as-a-Judges. You can see the, uh, basically how your fine-tune models are comparing, um, and very quickly decide if it's something you, you, you wanna promote to prod and, and get that outer loop done with.

95. 16:47

    Okay, got a minute left. Deployment. Um, this is actually also an area where things have gotten much more mature recently, and actually it's not that scary anymore. Um, you can self-host.

96. 16:57

    There's great projects out there that can give you, like, vLLM and TRTLM. They both give you, like, an OpenAI-compatible API that you can just, like, host on your own GPUs and kind of call the same way you would.

97. 17:06

    There's also a lot of hosted providers. We will host your model for you if you want. There's lots of other inference providers out there. Um, this space, like I said, has gotten very mature recently, so, um, not something that, uh, you, you should be too scared of.

98. 17:20

    Um, okay. This is just a recap of the, the, the workflow. You gotta prepare your data, train a model, evaluate it, and deploy it. Um, and then finally, um, this is what you're trying to achieve.

99. 17:31

    Uh, you know, you, you can get for one twenty-fifth the cost better performance than GPT-4, um, and, and for a lot of use case it is worth it. Um, and hopefully, uh, this is enough information so you can decide for yourself when the right moment is to do that.

100. 17:43

     Thank you. [outro music]