Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI

Talk transcript

1. 0:00

   [upbeat music] Amazing.

2. 0:15

   I think there's still some people coming in toward the back. Um, so I'll do, like, the quick maybe like, uh, administrative stuff in the meantime. This is gonna be super interactive.

3. 0:23

   Uh, there's also, like, mics around, so if at any, at any point you guys have questions, um, just raise your hand. Um, someone will go to you, or I think we also have mics nearby.

4. 0:33

   Um, cool. I think that's pretty much it. I'll let a few people finish trickling in, and then we'll get started.

5. 0:44

   Okay, I think that's long enough. Um, okay, so welcome everyone to Model Maxing with OpenAI. Uh, we're gonna be talking about RFT, SFT, DPO. My name is Ilan Bigio.

6. 0:56

   Uh, I'm on the developer experience team at OpenAI, and so I do a lot of very early testing on new products and new directions that we're taking the API.

7. 1:05

   So whenever there's a new feature coming out and it's still changing, like every day they're like, "Can you tell us how good this is?" Uh, and make a demo with it.

8. 1:13

   So this is a little bit of what we're gonna be doing today. It's gonna be a bit of a cross between a presentation. If you've been to any of my talks before, they're very code-heavy.

9. 1:23

   This one's a little bit less so because it's fine-tuning, and I don't wanna have you waiting for multiple hours watching a progress bar. So I have a lot of like pre-compiled data and some stories to share.

10. 1:35

    Um, cool. So this is you. Uh, you might be asking, you know, SFT, RFT, DPO, what the heck are all these letters? Uh, that's a great question. You might have some experience with fine-tuning, but be a little bit more asking like, you know, will fine-tuning solve all of my problems?

11. 1:51

    It's another good question. Um, you might also be wondering if you're, you know, in the right room. Uh, you are most likely. If you wanna learn about fine-tuning, stay here.

12. 2:02

    Cool. So yeah, we're gonna be talking about optimization at a high level, um, and then the specific subset that is fine-tuning. Um, we're gonna go into each of the different fine-tuning options that OpenAI provides, um, and then some stories, examples, best practices, and then we're gonna do some Q&A.

13. 2:18

    Um, I think I have like one hour and a half or two hours. I have no idea how long this is gonna take. We'll take as long as we need.

14. 2:23

    Uh, if we end early, you can all have some food. Uh, cool. So this is optimization. This is everything that it is, right? It's like it's a graph, uh, with a line going up and to the right in some nonlinear path.

15. 2:37

    Um, this is what you're going for whenever we talk about optimization. Uh, how you actually accomplish this can vary a lot. It can vary, um, you know, you can do like manual optimization, fine-tuning, changing your prompt.

16. 2:50

    There's a million ways to do this in a million different industries, but, uh, when you say optimization, this is all it means.

17. 2:58

    Specifically when it comes to LLM optimization, we have a few different ways we can do it. Um, but every LLM system is sort of composed of these three, uh, events or parts, right?

18. 3:09

    You have your input, which involves the prompt and the context. Um, you have your model, which is the, uh, like actually represented by the weights and whatever was pre-trained.

19. 3:19

    Um, and then you also have this system that goes around the model. It's the scaffolding, it's the tools. Um, I will go ahead and say this is every single LLM application ever.

20. 3:29

    Uh, so everything we say will cover all of this. Now, today we're gonna be focusing mostly on this middle part, um, optimizing the weights and optimizing the model choice.

21. 3:39

    Now, how to optimize like prompt context, uh, like prompt and context and tools and scaffolding, there's like a million ways to do that. Um, I think there's other talks today that are gonna go a little bit more in-depth into that, but we're gonna be focusing on this middle one.

22. 3:51

    Um, especially since it's a little bit nebulous, uh, and I wanna help clarify some of the topics and especially how you can do it with, um, the different like fine-tuning offerings that you can use on the OpenAI platform.

23. 4:02

    So this might be your initial reaction to fine-tuning. It's like, so it's like gross idea. It's, it's hard, it's confusing. Um, I would rather just like write prompts, uh, like this and have the model do what I want.

24. 4:14

    Now, that's fair. I think for a lot of, for a lot of problems, that's actually the case. Um, fine-tuning is really just continued training that optimizes a model for a given domain.

25. 4:24

    So it might be useful for your case. Um, you might be better off sticking up with prompting, and this is one of the things that we're gonna be talking about today.

26. 4:34

    But I think the idea is to open up your eyes to like all the options that you have. So let's do a quick comparison between prompting and fine-tuning and then move on.

27. 4:42

    Um, prompting is kinda like a set of tools, like a hammer, um, pliers, whatever. I think those are some made-up tools in the back that don't exist, but we're gonna ignore those.

28. 4:52

    Uh, it has a very low barrier to entry. Anybody can do it. Um, it's very easy and quick to do small changes to like quickly progress. Um, and it's actually enough for, I'd say, most jobs.

29. 5:03

    Now, fine-tuning is like a CNC machine, right? You have, uh, like it's a way higher upfront investment. You have to collect data, make sure it's clean. Um, the iteration loops are longer, right?

30. 5:14

    Like you, you can't just like make a quick tweak and like rerun because the rerun is gonna be multiple hours. So, um, however, they're more automated, right? Like if you have a pipeline set up for fine-tuning, it's a little bit more hands-off.

31. 5:26

    You don't have to do all this manual thinking yourself. Um, and finally, it is a specialized tool, right? It's not useful for every single use case, but there are certain use cases where it really, really shines.

32. 5:35

    Um, and with certain kinds of fine-tuning, you can actually push the boundary of what is possible and do things prompting cannot accomplish.

33. 5:44

    So these are the three main types of fine-tuning that we, uh, support during OpenAI. If you're gonna take pictures, there's another slide later that has this and more. Um, so the first one is SFT, supervised fine-tuning.

34. 5:57

    And what is this? It's just imitation, right? You just show, uh, the model what you want it to output, and it'll learn to copy exactly that. The next one is direct preference optimization, and it's about doing-- uh, showing the model two samples and you say, "One of them I don't like, one of them I like.

35. 6:16

    Do it more like the one I like and less like the one I don't like." And then finally, we recently introduced reinforcement fine-training-- uh, reinforcement fine-tuning, which is actually, like, the magic sauce that went into making o1 and o3 and o4-mini and all these reasoning models.

36. 6:33

    We, like-- You can use this algorithm as well on this platform. Um, and what you're doing, you're essentially providing a way, um... Th-the model essentially figures out how to think about your given problem.

37. 6:47

    So let's get a little bit deeper into each one.

38. 6:51

    What does the data for supervised fine-tuning look like? Like I mentioned before, you just collect a set of inputs and outputs that you want. Um, for direct preference optimization, you have inputs, and then for each input, you have a positive output and a negative output.

39. 7:06

    Um, I think we call them preferred and non-preferred. And then finally, for reinforcement fine-tuning, you provide a set of inputs, um, and then slightly differently, you provide a grader.

40. 7:17

    And optionally, with the inputs, you can provide some reference in order for the grader to be able to, like, have a ground truth if you want. But really, this grader can be one of a variety of things that we're gonna be talking about later.

41. 7:27

    Um, which is why s- um, reinforcement fine-tuning is so powerful because it, it does not constrain what you can do as much as these other two, but it still has its caveats.

42. 7:37

    So we're gonna be going into all of that. Now, what do each of these cases learn? Right, like you can look at the data, maybe you have some idea, um, but, like, from first glance, there's a lot of cases that supervised fine-tuning looks like it might be able to do, like, direct preference optimization versus reinforcement fine-tuning.

43. 7:54

    Like, how do you think about what each of them is doing? And the way that I find it mo use-- most useful to think about this is thinking about what is the model learning in each of these cases.

44. 8:03

    So supervised fine-tuning is essentially learning a continuous mapping or, like, a soft mapping from your inputs to the outputs. Um, and so, you know, it, it's, it's not exactly like a, like a direct mapping like a table, but in essence, that is what you're teaching the model.

45. 8:20

    It's very, very direct. Um, it's just gonna learn to imitate the data that you provided. Um, direct preference optimization is a little bit more interesting. Um, it doesn't actually learn, like, each of the examples.

46. 8:34

    It learns the delta between the examples. And so if you imagine if you can, like, pull up your, like, mental, uh, embedding, like, latent space, and you project these two ideas there, um, the positive output and the negative output, um, if you can imagine, like, what that difference vector is between them, that is essentially what you're trying

47. 8:53

    to teach the model. And then finally, and this is also it looks straightforward, but it's actually pretty interesting, is learning to reason, right? So reinforcement fin-fine-tuning learns to reason about a problem by learning to tune and change its chain of thought so that it is able to get this, like, higher success rate on whatever problem that you're

48. 9:13

    giving it. Um, so what each-- What is each of these good for? So supervised fine-tuning is the simplest of the bunch. Um, it's great for classification use cases.

49. 9:26

    Um, it's great for formatting and structured data extraction. Pretty much any case where you really wanna constrain what the model is doing, um, and have a very specific kind of output, supervised fine-tuning is it.

50. 9:39

    And this is helpful for, like, distillation if you wanna teach, like, a very, very, very small model how to do this, like, structured output example, and, like, prompting isn't cutting it, supervised fine-tuning is the way to go.

51. 9:49

    Now, direct preference optimization is a little bit more, um... I'd say it's the hardest to think about of the three. Um, it's very good at learning tone matching, right?

52. 9:58

    Because tone and style are these kind of little bit more, um,

53. 10:03

    intangible things that are hard to evaluate but easy to see if you put two, uh, comparisons together, right? And so this was really made, um, in order to do tone matching and also, um...

54. 10:15

    I mean, primarily it is when you have A/B tests, right? Like, when you have, uh, positive data and negative data, preferred data, not preferred data. If you've been using ChatGPT and you get, like, both, like, two responses, this is why, right?

55. 10:30

    Like, this is the algorithm that is being used in order to, like, tune for those preferences. And so it's not saying output exactly the one that I prefer. It's saying output slightly more in that direction, whereas supervised fine-tuning is much more exact.

56. 10:45

    And then reinforcement fine-tuning is for gradable hard problems. Um, some examples are, like, medical, legal, coding, just, like, things that are pretty hard to do but pretty simple or straightforward to verify.

57. 11:00

    Um, you really, really, really want unambiguous solutions. And I feel like in a lot of these talks, we always talk about successes. I snuck in a little failure here later, so you're gonna get to see that.

58. 11:10

    Um, and it's also a really interesting candidate for training an LLM judge model, right? Because grading a problem is actually a problem itself. And so if you... Like, grading that problem is a pretty verifiable answer if you have these, like, golden results that you want your LLM judge to be able to mimic.

59. 11:36

    Cool. So one thing to note before I get into each of these is that, um, each of these approaches, while

60. 11:48

    they're very different, we are training only a subset of the model in each of these cases. And so this is why they're not amazing for learning, like, tons of new data, right?

61. 11:58

    Like, RAG, um, and other solutions like agentic retrieval are much, much better if you wanna teach the model new information. Um, but- This is called, uh, I think low-rank adaptation.

62. 12:11

    Can anybody check me on that? Yeah, I see some nods. Cool. Um, and essentially for those who are interested, you should look more into it. It's a really, really clever way where you take the model weights, and you decompose them into two matrices.

63. 12:23

    Um, and then that essentially gives you a much smaller set of weights that you need to update in order to affect the overall model behavior. And so you don't fully readjust the entire model.

64. 12:34

    You don't make it forget the things it did before, but you still impart some new behaviors into it.

65. 12:40

    Um, cool. So let's start with a story. Um, when I joined OpenAI, I was working as a solutions architect, and what this meant was I was working directly with companies helping them deploy their, like, AI use cases and applications into whatever, um, domains they are working in.

66. 13:02

    Now, uh, one of our partners, one of our customers was working on a low latency function calling use case, kind of assistant style based on, like, taking action, right?

67. 13:13

    And so you would... Uh, the user would say something, um, and the, the model had to act. Um, but it was in an extremely latency constrained, uh, scenario, right?

68. 13:23

    So they wanted to use, back then it was 3.5, GPT 3.5. Um, GPT 3.5 was very, very fast, um, but not very accurate. Uh, GPT-4 was roughly the level of performance that they were expecting in terms of accuracy, but it was way too slow.

69. 13:38

    Uh, and so they asked us, "What can we do?" Um,

70. 13:43

    so, uh... Ooh, actually, [laughs] before I get into this, this would actually be fairly straightforward, and I feel like a lot of you could imagine how to do this if we had a lot of existing examples with, like, inputs and outputs, um, which [laughs] they did not have.

71. 13:58

    And so we were left to figure out, how do you fine-tune a model when you don't have, like, this, this great amount of data or, or these, like, good examples?

72. 14:05

    Um, and so we took a few different approaches, but I'm gonna cover the two main ones that we used that proved to be pretty effective.

73. 14:12

    So the first one is we did have all of the functions that they wanted the agent to use. Um,

74. 14:21

    one more thing I didn't say. They were trying to do this with 120 functions, which was unheard of at the time. I remember asking, um, some researchers like, "You know, what is like...

75. 14:32

    Uh, how many do we support usually?" And they were like, "Eh, five to ten maybe." I was like, "Okay. Okay, cool. Cool." Um, so this is actually the first time that anybody fine-tuned the model with over 100 functions because, uh, it did...

76. 14:47

    It-- The capability was not possible before. We had to have this built for this use case. Um, so what you're seeing here is the first step that we did.

77. 14:54

    We actually took the function, uh, schemas that the customer provided, um, and, uh, figured out a way to create every possible, like, permutation of the invocations of that function.

78. 15:09

    So for example, if we have like a toggle... Ooh, typo. If we have like set lights, uh, and it takes one parameter, and it's an enum, uh, here I'm essentially showing that the three permutations that you can have are set lights off, set lights on, and set lights red.

79. 15:22

    This was actually done completely deterministically. This was just like a, a Python script, um, that I spent way too long working on. Then, uh, search song is kinda similar.

80. 15:32

    Uh, there's more than one parameter. Here I skip the cases where you provide like multiple parameters, but essentially what this function did is it took a schema, uh, from a function, and then it spat out essentially every possible permutation of that function.

81. 15:45

    Um, and if it was like more... If like it exploded in complexity, we would like randomly sample from, from those.

82. 15:51

    Um, so this was the first step. Oh, and then if there were some samples that were non-deterministic like Katy Perry or like were open-ended, we would ask a model to like provide plausible values.

83. 16:04

    The next step was once we have those function calls, then we asked GPT-4 to go essentially from, "Here's a function call. Give me a command that would have resulted in that function call."

84. 16:18

    Um, and we did this for a large number of the permutations. Um, and so some of you might already be noticing what we're doing, but essentially if you like flip this in your head, now we have these input/output pairs where the input is the GPT-4 generated command, and the output is the function call itself.

85. 16:38

    So we're already on a pretty good track, right? We have a lot of data, um, but it's pretty... It's, it's synthetic, and we don't just want it to be synthetic.

86. 16:45

    We don't know if these are the kinds of inputs that the users will be giving, and if we are very off base, if we are like very out of distribution, we won't have as good, um, of a model in the end.

87. 16:56

    I will say the models are pretty good at generalizing, even if you do stuff that is slightly out of distribution because they have such a large set of weights in like a preexisting world model.

88. 17:07

    The second way we did it was distilling some data. Um, so, uh, the engineers we were working with did actually have, um, some unlabeled inputs, and we were like, "Okay [laughs]

89. 17:18

    . Well, here's what. You said GPT-4 was good enough for most of these cases. So let's do something. Uh, let's just run them through GPT-4 and get what outputs GPT-4 would give us."

90. 17:30

    Um, after this, I didn't include this in here, we actually did some filtering as well, and like we did a couple stages in this pipeline to essentially generate a lot of like distilled, um, still syn- still, still, uh, synthetic data, but, uh, hopefully very high quality.

91. 17:46

    And in the end, we took these two data sets, uh, passed them into GPT-3.5, um, and we achieved GPT-4 level performance at 3.5 accuracy. Um, and so this was one of the first like major successes that I saw with a supervised fine-tuning.

92. 18:03

    Now, note that this is a very constrained use case, so it's actually perfect for fine-tuning. The model doesn't have to do this like open-ended response. You're not trying to strive for a certain personality.

93. 18:13

    You just want it to call one function, and you want it to do it well, and you want it to do it fast. So this is a perfect use case to do distillation with supervised fine-tuning.

94. 18:23

    Actually, I wanna keep this interactive. I'll pause every now and then. Feel free to ask any questions. If I don't see any hands, I'll just, like, barrel along. Okay, I see a couple back there.

95. 18:37

    Uh, yeah, I think they're coming to you

96. 18:40

    with a mic.

97. 18:45

    Could you have...

98. 18:49

    Is it on?

99. 18:49

    There we go.

100. 18:52

     Six?

101. 18:53

     I can just... Yeah. Could you have distilled-

102. 18:55

     No

103. 18:55

     ... without synthetic?

104. 18:57

     So the question is, could we have distilled without synthetic data? Um-

105. 19:00

     Just for-

106. 19:00

     I think we would have preferred to distill without synthetic data, right? I think the, the use case that we... Like, the perfect case here is the engineers c-come to us and they're like, "Hey, we wanna get this model to be really, really fast, and we have 1,000 labeled examples."

107. 19:12

     And we're like, "Great. We'll get back to you in, like, a day." Um, instead, this is what we had to do. But yeah, that's a, that's a good question.

108. 19:21

     I think there's one back there.

109. 19:25

     What was the ratio between synthetic and real data? And how many examples would we need to do the

110. 19:33

     Yeah. So the question was, what was the ratio between synthetic and slightly less synthetic data, and how many examples did we need? Um, I think I'll, I'll answer the second one first, which is...

111. 19:47

     And we, we, w- I have this in a demo, and we'll, like, we'll look at the performance. So, like, um, at around, like, 50 to 100 examples, you start to see signs of life.

112. 19:55

     Um, and that might be, like, already enough for some smaller problems. Then if you start, like, pushing more toward production, you might wanna scale up to, like, few hundred, 500 examples.

113. 20:05

     In the demos I have later, I used fun... 150 examples and 500 examples. Um, but this is for supervised fine-tuning. For the other two kinds, you can actually use, uh, fewer and it's more forgiving, and we will talk about that in a second.

114. 20:19

     Um, and then the first one was, what is the ratio? I don't remember. [laughs] I, I think I tried some ablations where, like, I only provided one of the, um, one of each, and, like, um, it was best when I provided both together.

115. 20:31

     But I can't tell you what... I, I don't remember what the ratio was.

116. 20:35

     Yep.

117. 20:36

     Okay, so you, you said that there was around 100 functions?

118. 20:41

     Yes.

119. 20:41

     Okay. Yeah. I pretty much had the same question. So you're, you're basically saying for each function you had around 50 to 100 synthetic examples.

120. 20:50

     Uh, I don't rem... It, it was... I-I've said a lot of numbers around 100 and 500.

121. 20:56

     Yeah.

122. 20:56

     But I-I think they're being maybe conflated. We had 100 functions, around 100 functions. Each one we generated, um, I think it was, like, between 20 and 200... Like, we tried different amounts.

123. 21:09

     Like, between 20 and 200 examples, depending on how many permutations each function would give us.

124. 21:14

     Mm-hmm.

125. 21:15

     Um, and then also depending on which, which of the inputs. So all in all, I think it was in the, like, l- like, high hundreds, low thousands. Maybe, like, yeah, l-low to mid thousands of examples that we used.

126. 21:29

     Yeah.

127. 21:31

     And, uh, sorry if I'm skipping ahead here, but was there, um... Did, did that improve the function call accuracy for, like, choosing which function to call? Or did it just improve, like, how well it would call a function?

128. 21:45

     So, like, choosing the function versus, like, populating the function parameters?

129. 21:49

     Yes.

130. 21:49

     Both.

131. 21:50

     Okay.

132. 21:50

     It improved both.

133. 21:51

     Awesome.

134. 21:51

     Yeah, yeah. I mean, we, um, if I remember correctly, we got to, like, within 2% of GPT-4 at, like, a very small fraction of the latency.

135. 22:00

     Nice.

136. 22:00

     Yeah. Uh, something else that we tried was, like, what happens if you remove all the context, uh, all the functions from context, or, like, the parameters from context? Like, can it learn the parameters?

137. 22:09

     Like, 'cause we wanted to, like, really squeeze out as much context as we could to, like, reduce latency. And you know when you're cutting input context for latency reductions, you're, like, at, like...

138. 22:20

     Y-you're out of options, right?

139. 22:21

     Mm-hmm.

140. 22:21

     You should never start with that, but we were trying everything. Um, and that was actually not very good results. We, we, we realized, like, having things in context, like, removing them from context makes them much harder to learn-

141. 22:32

     Mm-hmm

142. 22:32

     ... through fine-tuning. It's not as, um, as good.

143. 22:36

     Thanks.

144. 22:37

     Yeah.

145. 22:39

     Yeah.

146. 22:39

     Yeah.

147. 22:40

     S-So how... Uh, follow-up questions that-

148. 22:42

     Mm-hmm

149. 22:43

     ... in the future if they have more functions, then do they need to retrain everything?

150. 22:50

     Uh-

151. 22:51

     Let's say they have 100-

152. 22:52

     Yeah, yeah

153. 22:52

     ... and, you know, they just have two more.

154. 22:55

     Yes.

155. 22:55

     So they would not scale, right? I believe.

156. 22:57

     This, this does not scale in that way. If you, if you, like, add new functions, you... Like, in this exact setup that we have, you would need to, to retrain.

157. 23:06

     Um, there are ways to just pick up training from where you left off. Like, you can fine-tune an already fine-tuned model. Um, and it just continues the fine-tune process essentially.

158. 23:15

     And would that be okay? Like, if you fine-tune on top of this with few examples-

159. 23:19

     I can try. Uh, I feel like it, it could be, especially for this case. It's so constrained.

160. 23:23

     Yeah.

161. 23:23

     I think you can get away with a lot in cases that are this constrained.

162. 23:26

     Got it.

163. 23:26

     Yeah.

164. 23:27

     Thank you.

165. 23:27

     I'll take maybe one, one more and then we will keep going. Yeah. How did your... How did the prompts change when you did this? How did the prompts change?

166. 23:36

     Yeah. Uh, how did the prompts change when I did this? Like, did you have... Were you able to reduce context of, like, what does each tool do? I see.

167. 23:44

     Uh, yeah, in terms of... So I think, um, the, the question is around, like, how much we managed to reduce the prompt size and, like, context that we put into the model.

168. 23:52

     The answer is a bit, but not that much. Like, we were able to get rid of most of the prompt and just say, like, "Classify into, like, one of these, and then here's all the functions."

169. 24:02

     But we weren't able to remove, like, types or the different parts of the functions themselves, um, from the context without losing performance. Yeah. Which was much more of an issue then because we were working with, like, a 4K context window.

170. 24:16

     Uh, this is maybe, maybe... I, I think we, like, had to, like, push, like, uh, Apply to, like, let us fine-tune a 16K model, and it was, whoa, it was crazy.

171. 24:24

     Now it's like you're fine-tuning, like, 1 million models. No, no problem. Okay. Um, cool. This is this case. I'll move on and- We'll have, we have plenty of time for questions.

172. 24:34

     Um, and if you're more, if you're curious to learn more, we did actually write a lot of this up in, uh, a cookbook called Fine-Tuning for Function Calling. Uh, you can run it, you can play with it.

173. 24:44

     It has most of what I talked about.

174. 24:47

     Okay, uh, time for the first live demo. So let's say we have a customer. I, I wanted to find a, um, a data set that would be easy to find.

175. 24:56

     So here I have cursor, um, with like a very basic, you know, loading in OpenAI keys, yada, yada, yada, but then I'm loading in this banking data set, and essentially what it does, if we like print out, um,

176. 25:16

     the first thing in the data... It's not happy.

177. 25:23

     Oh. Cool. Um, we have some input text, you know, I'm still waiting on my card and a label, and the label is, uh, some like index into an array of labels, right?

178. 25:43

     Uh, of like actual text labels like, um,

179. 25:50

     uh, user is [laughs] not waiting for a card, but like, like arrival time or something like that, right? And so what we wanna do is we wanna get a model that can like classify this really, really well, and this is a good case for fine-tuning or for s- uh, supervised fine-tuning.

180. 26:04

     It's very straightforward, um, it's very direct, uh, and it's very constrained, right? So, okay, so here I load in the data. I load in the label names. I guess I could have just [laughs]

181. 26:16

     printed these. Cursor abandoned me today. Okay, yeah, activate card. Yeah. Yeah. We got like many of them. And so as you can see, this is not like a super easy task for the models.

182. 26:30

     This is like a lot of different labels they can pick from. Um, and this is as hard as I tried in my prompt. [laughs]

183. 26:37

     That's it. That's it. Um, and then it passed it all the labels. Um, and I guess this is a good time to say in supervised fine-tuning, your prompt does not matter as much, right?

184. 26:47

     You are showing like a direct, uh, like a direct examples, and so it, like you can skip a lot of the prompt engineering. Not all of it, right? If you give it like nothing to work on, it's not gonna like learn very well.

185. 26:58

     Um, but like... Or sorry. You could actually completely remove your prompt, and if you have a constrained enough example, it will still learn to do the right thing. This is not true about DPO and RFT.

186. 27:09

     You still need good prompts for both of those. Um, SFT, I, I would recommend having at least a decent prompt. Okay, so we have our data. We have our label names.

187. 27:17

     Uh, I loaded in the test data, uh, and I wrote my prompt. So now I have my messages. Uh, I'm using the chat completions API because fine-tuning right now I think works in that format, so I wanna keep the format consisting, consistent.

188. 27:33

     Um, so we have our system prompt, [laughs] which is this beauty, and then the user is just the input te- uh, the user messages is just the input text, right?

189. 27:42

     And then I write this very simple function that calls the API and gives it a classification, right? And so if I like test this out right now, um, look at that.

190. 27:53

     So how do I look at my card? Card arrival.

191. 27:57

     I guess that's correct, but like if I run it a few more times, you know, card not working, like

192. 28:02

     it's not good. To be fair, I don't know if any of us c-could do this with this prompt. Uh, it's pretty bad, but that's part of the point. Okay, so we have this sort of like inconsistent thing.

193. 28:13

     Um, now before we even start doing fine-tuning, we have to ask like, you know, should we do it, right? And this is one of the most important questions in fine-tuning.

194. 28:23

     I think this is part of why a lot of you are here. Should I do it? The answer is, how well is it doing on your evals, right? So I wrote this very small like parallel evaluation function, um, that essentially takes in a model string, a system prompt, uh, some like input samples to run over in parallel,

195. 28:40

     and then a number of workers since I'm like shooting them off in parallel, sort of like asynchronously. Um, I have a semaphore just so I don't like violate any like rate, uh, rate limits or like the number of workers that I have is like fixed.

196. 28:53

     Um, I'm like logging the progress, and then I have my score function, which just calls classify intent, um, updates the progress, and then returns whether it equals the correct label.

197. 29:04

     And I shoot them off in parallel, and there we go. So, uh, this is my evaluation function. You know, people talk about like different, uh, evals, um, platforms. We have a way to do evals as where, as well on our platform.

198. 29:20

     Um, I think I'm gonna stick to this for this example since I wanna keep things like mostly in this notebook. Um, and we can like see little pretty progress bars that I made.

199. 29:30

     Okay, so now we have GPT, uh, four O mini and four O.

200. 29:36

     This will come in later, uh, and we can run it, right? And so if we run this evals, we're essentially taking the testing set, um, made up of one hundred and fifty examples, uh, running it.

201. 29:48

     My progress bars aren't working correctly. That's fine. Uh, it still finished, right? So I just ran four O mini, and it ran four O, and now we get the relative accuracies, which is seventy-five percent accuracy for four O mini and eighty-three percent for four O.

202. 30:02

     So, you know, not bad. Let's see if we can do better.

203. 30:06

     Um, cool. So this is just, I'm taking the actual data sets and then dumping them out into files that we can use for training. Now, if the important part here is the format.

204. 30:20

     So the format of a fine-tuning model, and like let me see if I can just... I, I just love pulling up docs in talks. Uh, it's, it's what everyone else will be doing.

205. 30:32

     Um, let's see. Oh, by the way, you should all check this out. Agents SDK TypeScript came out today. Uh,

206. 30:43

     where are we? There we go. So if we look at the format... By the way, I know this stuff. I'm not doing this for me, okay? This is for everyone to see.

207. 30:55

     Uh, cool. So here we go. This is the JSONL, which is just like JSON lines, data format that you'll, you'll wanna be passing, which is, um, a messages, uh, array, like a, a field.

208. 31:07

     It's array that you're used to, the one you know and love. Um, and all it's gonna do is it's gonna take the last assistant message and learn from it.

209. 31:16

     Um, here, the last assistant message is a function call, so it has, like, the whole schema. Um, but if we look at the data that we're producing here,

210. 31:27

     these are like, you know, this is my amazing prompt. Uh, these are like the... This is all part of the prompt, right? And then when I get to the very end...

211. 31:37

     Oh, boy. Okay. Uh, then we just have... So that's the developer message. Then we have the user message, which is the input, and then the assistant message, which is what we want it to respond.

212. 31:49

     Um, and that's the whole setup, and what it's gonna do is it's gonna learn from that last message. Uh, so we can either shoot this off from the, uh, fine-tuning UI, but I am gonna show you...

213. 31:58

     I'ma... I'll show you how to do both. Um, cool. So now what we're doing is we're taking the dataset, putting in this format, and saving it, and calling it train.

214. 32:08

     Uh, we then, like, upload the files. So it's important that we... Like, for, for us to reference files within a fine-tuning job, we have to upload them to the OpenAI API first.

215. 32:17

     So client.files.create. We have to set the purpose to fine-tune. Give it the file. Here are the file IDs that we have. And now fine-tuning is as simple as, like, this API call, right?

216. 32:29

     We pass in the base model. Uh, the reason I had the mappings between, like, the base model, like 4o, and, like, the full snapshot is because it can't take in, um, aliases.

217. 32:39

     It has to take in, like, the full snapshot. Then we pass in the training file, which is gonna actually train on validation file, which is gonna tell us how well it's doing, uh, so it doesn't overfit.

218. 32:49

     And then I'm just passing the number of epochs. There's many more parameters you can play with. I just wanna give you a sense.

219. 32:55

     Um, so cool. I already ran this. [laughs] I don't want you all to sit here waiting. Uh, f-fine-tuning jobs can take anywhere from thirty minutes, a-and this is, uh, on the OpenAI API, anywhere from thirty minutes to twenty-four hours.

220. 33:12

     So give yourself time. Never do this right before a talk.

221. 33:17

     Uh, cool. So we have our fine-tuning job. Then we load the model, right? We, like, check the model. Uh, we, we check the job. We see if it succeeded.

222. 33:26

     We load them in. I actually already saved them here. And now the question is: How well did it do? I fine-tuned, um, as you can see, I made some examples with...

223. 33:36

     I guess I skipped over this, but I made some examples with 150 examples, and I made some with 500. And I trained both for, uh, GPT-4.0 and GPT-4.0 Mini with each one, just to do, like, a little comparison.

224. 33:50

     Uh, and so I ran it, and I also compared it against o4 and o3. And the final results at the very, very end, this is what they look like.

225. 34:00

     So, uh, baseline eighty-three percent, seventy-five percent. Uh, reasoning models a little bit better, eighty and ninety. Um, but the fine-tuning models blow everything out of, like, blow everything else out of the water.

226. 34:12

     Um, and this is kind of to, to show, like, how convenient fine-tuning can be, right? Like, it's doing better than o3 at a task that, like, I barely wrote a prompt for.

227. 34:23

     And I think even 4o Mini... Yeah, 4o Mini with 500 examples is as, is, like, actually technically better than 4o. But, like, they, they sort of re- like, are reaching much higher levels.

228. 34:33

     So this is what I wanna show how you can, like, get fine-tuning and, like, when it's worth it. Now, if this was something I could have done with more prompt engineering, maybe.

229. 34:41

     But, like, this is a case where I have the data, I wanna run it. Pretty straightforward. Uh, maybe, yeah, I'll pause here again for, for questions. Yeah.

230. 34:52

     Is the fine-- Oh. Do the fine-tune runs check for over fitting? And if they do, do they stop, or do you have to be, like, watching it-

231. 35:03

     Nope

232. 35:03

     ... while you're like, "Ah."

233. 35:04

     Nope. So the question is: Do the fine-tuning runs check for over fitting? No, they don't. I mean, you can see it. You can see it on, like, the diagrams.

234. 35:10

     Let me see if I can actually pull up these jobs. Um,

235. 35:15

     maybe this last one.

236. 35:19

     I guess the second question then is: Do you just kill it after the diagrams show you that it's crappy? Or, like, how do you know what to do next? [laughs]

237. 35:26

     Um, am I in a different org? Okay. I may have done this in a different org, [laughs] so I'll pull it up at a different time. But, uh,

238. 35:39

     yeah. Um, I don't wanna leak anything. Sorry. Uh, sorry, what was the question again? Uh, do you ha- You, you do have to s-

239. 35:45

     When you see the over fitting, like, what do you do?

240. 35:48

     Well, I mean, yeah, you, you can stop it and try again. Uh, I think you're asking me a very, like, data science general question. What do you do when you see over fitting?

241. 35:56

     So many things you can do.

242. 35:57

     But in this case specifically, it's like, you know, you don't wanna have a twenty-four hour job-

243. 36:01

     Yeah

244. 36:01

     ... doing a thing. Like, yeah.

245. 36:03

     You, you can stop it. Yeah, you can stop it. Uh, you can stop it, and I think you'll still be able to test with the latest snapshot maybe. Uh- Right here.

246. 36:12

     Here? Yeah.

247. 36:12

     Yeah. Hey. Um, couple of questions. How do you decide between, like, when do you wanna fine-tune versus not? Like, do you just lean to a-always using o3, or, like, when do you sort of pick up to, like, actually go fine-tune?

248. 36:23

     I'll, I'll get into this more later, but I think the short of it is always start with prompt engineering and see how far you can go, and then run evals, right?

249. 36:33

     And so i-if, if it's as good as you want it to be, that's great, right? Um, and if you, if you are still changing the prompt and you are still getting better results- There's no reason to fine-tune yet, right?

250. 36:44

     I think fine-tuning is something you wanna consider, like, when you have enough data for it to be worth it, and the data is, like, pretty clean, and, like,

251. 36:54

     it's not really like a, eh. Like, y- y- you kinda have to have, like, the, the, the resources ready for t- to do the fine-tuning.

252. 37:01

     Fair enough. And then, like, which models do you, like, support fine-tuning for right now? And, like, does it keep changing over time? Like, I know it used to support like, you know, 3.5 and then 4, like-

253. 37:09

     Yeah

254. 37:09

     ... is everything, like, when it comes out, like, instantly gonna support fine-tuning or is there like... What's the lag period behind it?

255. 37:14

     I think we have... Ooh. [laughs] [laughs] [laughs] Oh, man. Uh, where are we? I think we now, like, say... Yeah, which ones support fine-tuning on the models page.

256. 37:30

     This is why I like docs [laughs] and live, uh, browsers. Uh-

257. 37:35

     Yeah, did you have to do any, uh, hyperparameter tuning?

258. 37:38

     Not for this.

259. 37:39

     Okay.

260. 37:39

     Uh, not for this. Yeah.

261. 37:41

     Does it have any, any impact on the results, like updating, running late?

262. 37:46

     Sh- sure.

263. 37:46

     Yeah.

264. 37:46

     But I'd say that falls into the, like, the, the ca- the, the field of, like, it's gonna, it's gonna be,

265. 37:53

     uh, very case by case. You have to, like, run different, different things. Th- there's not a lot of good intuition that I have built that I can share with you.

266. 38:01

     Um, but I'm sure if you, like, look online, everyone will have their own opinions. Yeah.

267. 38:05

     Thank you.

268. 38:06

     Yeah, I'm curious on, um, after you fine-tune, can you still tweak the system prompt? Or if you do, do you need to then fine-tune again?

269. 38:15

     Yeah, that's a good question. You can tweak the system prompt, but, um, you are now out of distribution, and you are sort of hoping the model was able to learn from your, like,

270. 38:27

     enough out of distribution behavior that it won't. I would recommend if you fine-tune with a prompt, always use that prompt. If you're gonna change the prompt, fine-tune again. Um, or just, like, use many different prompts during your fine-tuning.

271. 38:40

     You don't have to use just one. Uh, if you use multiple different prompts, you might, like, you might prevent it from collapsing too much into, like, one specific kind of behavior.

272. 38:49

     Um, but g- good question. Uh, yeah.

273. 38:55

     Uh, can you go back to the notebook for one second?

274. 38:57

     Yep.

275. 38:58

     It appeared to me like fine-tuning the 1540 made it worse. Is that true? And if that happens, how do you know when to stop and tune?

276. 39:06

     Fine-tuning for 4o mini. For f-

277. 39:13

     4o.

278. 39:13

     4o. Oh.

279. 39:13

     The fine-tuning made it worse, but fine-tuning made it too so that... How do you know when it's gonna start to go up? Because really it went up.

280. 39:22

     Well, yes, that's a good question. I don't know what happened. [laughs]

281. 39:25

     Yeah.

282. 39:26

     I think, um, either I made a mistake or I, um...

283. 39:34

     I would love to pull up the graphs right now. Uh, the reason I'm not is because they-- I think I ran these on the OpenAI org, and I don't wanna pull that up on screen live.

284. 39:44

     Um, but that's what I recommend you do.

285. 39:46

     Understood.

286. 39:46

     Yeah.

287. 39:46

     Ask them.

288. 39:48

     Uh, yeah.

289. 39:50

     Um, regarding like overfitting, do you have like, uh, ergonomics to do that in the SDK? Like, like defining-

290. 39:57

     No

291. 39:57

     ... a test set and then-

292. 39:58

     Oh, yeah, yeah. I mean, you, you, you can define a test set, validation set, and it'll like do both.

293. 40:02

     And can you determine when, when you see the, like, the curve go, go up of like the test set?

294. 40:08

     Yeah.

295. 40:08

     Like when you see them cross, do you stop the job or you have to check it ma- manually?

296. 40:12

     I guess we can just pull up like an old, older one. Um, I just don't know how successful it'll be.

297. 40:19

     But like, you know, you get these... What were, what was I doing?

298. 40:25

     Yeah, but like the, the blue line is validation loss and the, the green one is... This is clearly like not an amazing fine-tune run. I think this was pretty old, but this is like what the graphs look like.

299. 40:36

     Yeah. Okay, uh, I'm realizing, uh, we still have a lot left, and I'm running out of time, so let's keep it chugging along. Um,

300. 40:47

     cool. Okay, so rules of thumb for supervised fine-tuning. It's best for simple tasks. Um, models can regress on other tasks, so I mean, this regression that you called out, confusing.

301. 41:01

     I'm not sure what it was. Um, and but the thing is if you, if you then test it on different things that you did not fine-tune for, you may see regressions.

302. 41:09

     So this is for very constrained cases. Data diversity is critical for that reason. Anything you don't include might regress. Um, hundred samples to start, five hundred plus is best.

303. 41:19

     Okay, DPO. Uh, training a model on jokes. Uh,

304. 41:27

     so [laughs] I, my, my, my thinking here was like if we take a mo... Like jokes, like this funniness principle is like very hard to evaluate for, but you know between two options whether something is funny or not, so this feels like a good candidate for DPO.

305. 41:41

     Uh, so I generated a bunch of, a bunch of jokes for, with 4.5, hoping they were gonna be good. Um, m- most of them were not, some of them were.

306. 41:49

     Uh, I really wanted a data set, and this is really what it comes down to. All of these are about the quality of data. I did not have high quality data.

307. 41:56

     I had some medium quality data with a, um, like mediocre validator. So like w- uh, then did some manual filtering. Like, did I laugh? Uh, did I, did I like exhale a little bit more?

308. 42:09

     And then, um, okay, the interesting part is once I have this subset of good jokes, uh, which really should have been like a nice data set,

309. 42:17

     uh, remember what we're trying to do, right? We're trying to get some inputs and then two pairs. So I went backwards again t- from joke, I extracted the topic,

310. 42:26

     um, and once I had a topic for each joke, I once again went forward, uh, with 4o and asked it to like generate a corny joke for that topic.

311. 42:36

     Um, and so what we're left with is once again input topics and then good jokes and bad jokes. For that topic. Um, and so hopefully the, like, main delta between those jokes is gonna be one is good, one is bad, they're about the same topic, so, like, that delta should be stronger, um, as opposed to just giving

312. 42:54

     it two unrelated jokes. Maybe it's like, ah, like, you know, he's looking for, like, cat jokes or something. No, it's like similar jokes, one's funny, one is not. Um, did it work?

313. 43:05

     Sort of. Um, this is one of the results that I got. So two guys are hunting when one collapses, suddenly stops breathing, and appears lifeless. Panicked, his friend calls emergency services.

314. 43:16

     "Help, my friend collapsed. He's not breathing. I think he's dead. What should I do?" The operator calmly replies, "Stay calm. First, let's make sure he's really dead." Then there's a short silence followed by a loud gunshot, and then the guy comes back on the phone, "Okay, now what?" [laughing]

315. 43:30

     Okay, so, uh, is it original? I don't think so. No. But, like, you know, it's funny. Uh,

316. 43:39

     cool. So, um- It's based. Yeah, but, like, can you tell me an original joke right now? It's exactly that joke. Exactly that joke? Ten years ago, I would have been with it. [laughing]

317. 43:51

     I agree. Me too. I would say almost all the jokes that the models generate, h-- like y- you are hard-pressed to make, like, a, a good one. But I think it's still, like, it's not a null result because the fact that it was able to, like, give me a good joke on command is a hard thing to

318. 44:08

     do, right? Like, I think if you ask ChatGPT for a good joke now, it's, like, harder to get that to happen. So even though these are, like, not new jokes, the idea is to, like, elicit the good jokes.

319. 44:18

     Um, what was I gonna show you? Where's my notebook?

320. 44:24

     Um, yeah. So here is the data in the format that we said, right? So we have the user input, then tell me a joke about topic, and then we have the preferred output, and then the not preferred output.

321. 44:40

     Amazing. Cool. Let's keep going. Um, okay. So DPO, it's more forgiving about data diversity because if you give it, like, a bunch of examples that are all specifically jokes, and then I ask for, like, um, a speech about something, um, it's not gonna give me a joke necessarily, right?

322. 45:04

     Because it didn't learn to say jokes, it learned to get funnier, hopefully. Um, so it's like th- that delta is, is important, and it is less constraining than SFT.

323. 45:15

     Um, it's, it's much more for, like, these which do you prefer, uh, scenarios, and, like, it's hard to evaluate unless you have, like, humans or, like, some, like, natural signal.

324. 45:25

     Um, and you can actually run it after SFT. Okay, reinforcement fine-tuning, the one I wanted to save time for, which I didn't. Eh, I still have some time. Um, the shape of it, as we talked before, th- this is, I think, a pretty exciting one is, like, as time goes on, um, I think more and more use

325. 45:42

     of this is gonna, is gonna start happening because this is the first way, um, that I see that, like, we are really making this algorithm public, like, this reinforcement learning algorithm public, and you can, like, become, like, get SOTA, like, state-of-the-art results on whatever your task is, um, like, beyond o1 and o3.

326. 46:01

     Um, cool. So what does this algorithm look like?

327. 46:05

     You have a bunch of inputs. Give it to the model. Model, for each of the inputs, generates this, like, chain of thought to the output. Then you have your grader.

328. 46:13

     Your grader evaluates the results, right? Um, how does it do this? However you want, uh, and we'll get to that in the next slide. But importantly, when it performs well, it reinforces those behaviors specifically and whatever led to that chain of thought.

329. 46:29

     And so good chain of thoughts, um, that, like, get high scores are more likely, like, to, to, to keep going, like, to... It's more likely to generate those chains of thoughts that lead to high results, which is exactly how, like, o1, o3, and these reasoning models are trained in the first place.

330. 46:45

     So you can do this. Um, now, these are the very, very important things to keep in mind. You wanna have, uh, unambiguous grading. Um, in an example you'll see later, the grading was a little bit less, uh, unambiguous, and the results varied.

331. 47:02

     Um, you want very low noi-- and, and by unambiguous, it just means, like, if you ask a bunch of people, uh, who are, like, fairly smart and have the same rubric how to get to the answer, they will all give you the exact same answer.

332. 47:13

     Um, you want very low noise data, so if you have, like, any mistakes in your data, they're gonna, like, magnify in a way that is not normal for, like, SFT and DPO.

333. 47:22

     So it's very, very, very important for you to clean your data and not have, like, noisy signal. You want, you want it to be high signal, like, a- and this kinda goes with the last one.

334. 47:31

     You want it to be able to actually get to the solution, um, 'cause it's learning to reason. It's not necessarily learning, like, new facts or, like, new, new things.

335. 47:39

     It's like, it's learning how to, how to get to the solution. Um, and because it is, like, so... It's, it's very data efficient. You can give it, like, forty to eighty very high quality, very high signal examples, and it can, like, learn to reason over them and generalize, uh, to other examples in that set.

336. 47:58

     Uh, cool. So now let's talk about graders. Uh, there's a whole myriad of graders you can pick from. There's stringy, like, string checks. Um, there's text similarity. Um, you can write Python code, sandboxed, no internet access, a few libraries preloaded.

337. 48:14

     Um, and you can have that Python code be the grader itself. You can have a score, like mo- uh, model graders. So model graders are just, like, the, like, using the LLM with a prompt, um, one that outputs a score, one that outputs a label.

338. 48:28

     And we have a multi-grader. You can nest these as much as you want, and then you write a combination function. Um, and so this is how you can define the grader.

339. 48:37

     I'd say it covers most of the cases where you would wanna have something graded. Um, and you can get pretty, pretty good results.

340. 48:46

     So, uh, this was my idea for the talk. Um, I hate email. Let's make a model that can, uh, do the thing... Like, I-- It's not that I want it to respond to email.

341. 48:56

     It's I wanna know whether it has to show it to me or whether I can ignore it and never have read it in the first place. Uh, so that was the idea.

342. 49:04

     So I made this little, like, email labeler for myself, and I became a little data labeling [laughs] person for a day. Um, turns out if you make an interface that is nice, you can go so much faster.

343. 49:18

     Um, so I labeled around six hundred emails. Uh, my, my categories were like, uh, am I gonna glance, ignore, archive it, which is like receipts, store it, which is like I wanna see it later, respond within different time frames, take action within different time frames.

344. 49:33

     I actually didn't end up using the topic.

345. 49:36

     Um, and I can download it as a JSONL. Now, what do the instructions look like? I have some formatting. I have some classification instructions with, like, a little bit of extra instructions, but I didn't try very, very hard with this prompt.

346. 49:51

     This will come back to bite me. Um, and then the input itself I formatted as, um, uh, like, th- this, like, XML-ish. So I wrapped each, uh, each in an email.

347. 50:02

     The input is in an email with sender, subject, date, and then the stringified body, uh, plain text, uh, email.

348. 50:11

     How, um... Yeah. So this is... Then I formatted the data for reinforcement fine-tuning, which is a set of messages in the same way that you would for SFT or all the others.

349. 50:21

     Um, and then I use reference answer. You can use any other fields that you want, but that is essentially, uh, fields that are accessible to your grader. Uh, and then the grader, which I define below, uh, is just a string check grader that takes the, uh, reference answer action and compares it to the sample output.

350. 50:39

     And if they're equal, we're, we're good. Um, now, how'd it go?

351. 50:45

     Like, okay. Not amazing. Um, and I ha-- I, I, I think I know why, right? So first of all, like, this is just my whim. Like, I was just, uh, like, wh-whimsy deciding whether an email is important or not.

352. 51:01

     It's a very hard function to model. If I w-- If I was asked to label the same emails again, would I hit, like, a hundred percent the same classifications for all of them?

353. 51:10

     Probably not. And that's exactly the scenario you wanna avoid. Um, it was somewhat noisy. I think I, like, screwed up, like, one or two, uh, and that, like, uh, compounds.

354. 51:20

     Um, a hundred and fifty examples, that was actually fine. Like, if they were very high signal, this would've been, like, a better result, but they weren't. Um, and I didn't really try with a prompt.

355. 51:30

     The model doesn't know what I want, and it's trying to figure it out. It's like, it's like as if I gave you this task with the emails and I locked you in a room and you get to, like, maybe write some notes, uh, and I, like, slip you some food and water.

356. 51:43

     Um, and then the grader, uh, you know, string quality, that was fine, but there's only, like, a few categories, so the model could guess some, and you wanted to make it not, uh, very guessable.

357. 51:55

     Um, okay. So that was my failure. Now let's take a look at something that, like, reinforcement fine-tuning is actually quite good at, right? So, um, predicting the number of hydro bond, uh...

358. 52:05

     Sorry. Yeah, hydro bond... I think it's... Yeah, hydro bond donors and acceptors. Um, this is actually fully out of what I understand, um, but the model can learn to do this quite well.

359. 52:16

     So the task is to take in some chemicals, um, formatted in a specific format, and then predict the donors and acceptors, uh, to analyze it. And this is an example of one of the inputs.

360. 52:29

     Um, now the output schema is defined on the left. We have acceptors and donors, uh, which are just two numbers, uh, and it just has to output these two fields.

361. 52:38

     And then on the right, uh, I have my grader, which is a multi-grader. It just checks for each of them, right? And the weight is, it gets half if it gets the donor right, half if it gets the acceptor right.

362. 52:53

     Um, I'm gonna pause here 'cause this is an important slide, if there's any clarifying questions. Otherwise, I'll move on.

363. 53:01

     Amazing. Um, so what were the results? Much better. This started at roughly, like, sixty-five percent. Um, it ended up going to... I think it started, like, touching eighty percent, right?

364. 53:15

     And this is o4-mini. Um, o4-mini, like, does not have a very vast world knowledge. It really has to, like, learn to reason through things. Um, I'd say even this result is not, like, fully exemplary of how good it can get.

365. 53:29

     The reason it's not as good is because it does need some world knowledge to understand, um, like, this sort of simulation that you have to do with the hydro bonds.

366. 53:37

     Um, but you still see this, like, very, very high performance that it-- like, you would not get with something like supervised fine-tuning.

367. 53:45

     Yes.

368. 53:46

     Did you use DPO before this? Would you like to hear an email example with DPO first and then do this second?

369. 53:53

     Yeah. One thing I forgot to put in the slides, SFT and DPO you can do with non-reasoning models. RFT you can do with reasoning models, and there's no interplay.

370. 54:01

     Can you repeat his question?

371. 54:03

     Yes. Sorry. He asked, um, if there... [laughs] I'm blanking.

372. 54:09

     It-- whether you can do DPO before RFT, and the answer was, uh, SFT and DPO are for non-reasoning models, RFT is for reasoning models, and it's like an IFF.

373. 54:19

     Um, cool. So some thoughts on, uh, notes on reinforcement fine-tuning, only for reasoning models. Uh, it's extremely sensitive to noisy data. Um, agentic use cases are limited by a single turn, right?

374. 54:31

     So you can't yet have these, like, long horizon tasks with multiple turns and function calls. Uh, and reinforcement fine-tuning, I would love to see that. We'll see if we ship it at some point.

375. 54:42

     And then, um, you want the data to be high signal, uh, and you want a solid grader, and if you provide both, you are likely gonna reach, like, state-of-the-art performance on the task that you are going after.

376. 54:55

     Um, cool. So this I think is one of like the most telling slides. I believe

377. 55:03

     that is most of what I have for today. Um, I have a notebook. I have a ton of stuff that I didn't quite go through. Um, I'm happy to dive into that, but I kinda wanna leave this open for questions and like poking around, um, different parts.

378. 55:18

     Actually, how much time do I have left?

379. 55:20

     It's like an hour.

380. 55:22

     An hour? Oh, great. We can do so much in this time. [laughs] Uh, you guys are, are captive. Okay. Yeah. I, I do wanna do a little quick questions, and then we'll move on.

381. 55:33

     One thing I started and didn't do is another kind of tuning, um-

382. 55:38

     He's gonna-

383. 55:38

     ... which I can do live, which is prompt tuning, um, which I'm gonna make up on the spot. So if you wanna see that, stick around. Um, but yeah.

384. 55:47

     Questions. Yeah.

385. 55:48

     Yeah. Hi. Uh, so I had a small question. Uh, so you're saying that if I have a non-reasoning model, and let's say I do an SFT step with it, uh, can I induce thinking into it?

386. 56:01

     For example, if I create a dataset input/output pairs, which has, let's say, thinking into it as well. So would you call it the same amount of thinking as you would get in a, a reasoning model?

387. 56:15

     Um, you would get thinking. It's a very kind of... a very different kind of thinking because it's, it's purely, it's learning by imitation. Um, so it's not learning to like, uh,

388. 56:27

     reason in order to get to an answer. It's learning to imitate the thought process, um,

389. 56:33

     regardless of whether it's actually useful to it or not, right? Um, however, on the other... Like, whereas reinforcement fine-tuning, like it is, it is the one producing the chain.

390. 56:43

     And so, like if it gets to the right answer, it means that chain was useful like, in, in practice. And that is true for like every single time that it gets reward, it's because like it managed to get there.

391. 56:54

     So it's, it's like, it's pretty different. I'd say like, yeah, SFT with a, um, reasoning is like what we started to see a little bit like maybe a year ago.

392. 57:04

     Um, but reinforcement, uh, learning is really how you get like high quality learning. Yeah, like chains of thought.

393. 57:11

     Uh, yeah.

394. 57:12

     Yep.

395. 57:13

     Or sorry, somebody has a mic. Yeah.

396. 57:14

     Here we go. Yeah. Um, thanks, Ilan. Could you go back to the slide, um, where you showed the inputs to the RFT? Yeah, this one. Um-

397. 57:21

     This one

398. 57:22

     ... showing the weights. Is-

399. 57:23

     Sure.

400. 57:23

     Does it, um... So for the calculated output, can I, um, basically instead of just like a, a, a lookup into a dictionary, can I have functions there, like, like to...

401. 57:35

     So, so can I pass in, uh, a function to check if the code runs and output a number, like a one or a zero to see if it runs, for example?

402. 57:44

     Um, can you, can you say it again?

403. 57:46

     So if, um, I was just thinking, um, one of the guys in the team is fine-tuning a model at the moment to write, uh, Triton code, and one of the checks he has is like, does the code actually run?

404. 57:55

     Yeah.

405. 57:55

     Like is it valid Python? Is it possible to encode that as a grader, this-

406. 58:00

     You could do that in a hacky way.

407. 58:01

     Yeah.

408. 58:02

     Right?

409. 58:02

     Yeah.

410. 58:02

     So you could use like a, um, a Python grader and inject the model response. Like just do like, um, eval or like-

411. 58:15

     Yeah

412. 58:15

     ... exec.

413. 58:16

     Yeah.

414. 58:16

     Which if you've come to my previous AI engineer talk, I did the same thing. Don't do this.

415. 58:20

     Yeah.

416. 58:20

     But you could, you could, uh, get it to work.

417. 58:23

     Yeah.

418. 58:23

     Uh, i- if it has a lot of right libraries and doesn't need internet access.

419. 58:26

     Yeah.

420. 58:26

     Which it might not. Yeah.

421. 58:27

     And, um, just the... And then on the, the weights you have there, does it matter the

422. 58:33

     magnitude, I guess, of like what you're calculating? Like you should, should you like aim for it to sum to, to one or, or a hundred or like it doesn't really-

423. 58:40

     Um-

424. 58:41

     ... like match, as a difference?

425. 58:41

     I don't think it matters.

426. 58:42

     Yeah, yeah.

427. 58:43

     Yeah. Um, it asterisk in the way that like...

428. 58:49

     It, it doesn't really matter, asterisk. You can look into the asterisk.

429. 58:52

     Yeah, yeah.

430. 58:52

     Yeah.

431. 58:52

     Okay. Cool. Thanks. Um-

432. 58:56

     Um, I wanna do some in the back 'cause I feel like I've been missing some in the back. Yeah.

433. 58:58

     Y- yeah. You, you mentioned that it wouldn't, the RFT wouldn't work for multi-step, uh, agent stuff. Could you talk a little bit more on like why, and if, if you, if OpenAI were to make it work for that, what that would look like?

434. 59:14

     Mm.

435. 59:14

     I just kinda didn't understand.

436. 59:15

     Yeah, yeah. So, so the reason I said that, um, is because right now it only does a single turn, and it only evaluates the output of the model. If the model say like calls a function, there's no way to provide it the result and let it keep going, right?

437. 59:32

     And so that's really what you want for like an agentic evaluation loop, which RFT does not currently have. Um, I, I think that would be awesome if like we did have a way to like, um, provide the output of the model and like do these more end-to-end training.

438. 59:49

     Um, but this is what we have right now.

439. 59:52

     Gotcha.

440. 59:53

     Yeah.

441. 59:53

     Thank you.

442. 59:56

     I, I wanna do in the back first a little bit. Yeah.

443. 59:58

     For, for the hydro bond example with RFT-

444. 1:00:01

     Yeah

445. 1:00:01

     ... you used a generic string check, but you could inject more domain expertise to check if it's balanced files, if it's only using like chemical examples. You know, when do you need to design that type of custom grader, and when is a generic grader sufficient?

446. 1:00:17

     Yeah. So the beginning of the question was in the, uh, hydro bond example, um, we use like yeah, this, this string checks. When is it worth using more complex, um, graders?

447. 1:00:31

     I think the answer is like, where you think it would benefit. Uh, like is it a more accurate score? Is it a higher signal grade? Because if the answer is yes, then you might actually just get higher, like better results, right?

448. 1:00:44

     It's gonna maximize whatever this grader says is good. And so if you can better tune what that means, like what good means, you will just get better results. Yeah.

449. 1:00:54

     Um, or does someone else have the mic other?

450. 1:00:57

     Hi.

451. 1:00:57

     Yeah, yeah.

452. 1:00:58

     Uh, can you tell us more about the RFT algorithm or some details about how it works?

453. 1:01:02

     Yeah, yeah, yeah. Here, here. Just come here. No, I'm kidding. No. Um, I mean- It, I think obviously I can't share like the, the exact details, but like it is, it is fundamentally like a, a very, very similar algorithm to the exact algorithm that we used for o1 and o3, right?

454. 1:01:17

     Like, this reinforcement learning with language models is like exactly how we did our reasoning models, um, and we are putting as much out as we can, right? And so like you can actually get these like really, really insane results, um, if you have clean data in, uh, in your scenarios.

455. 1:01:36

     Yeah. The reason... I, I really wanted to come with like a lot of really good examples. The reason I can't come with a lot is like, A, a lot of them are like proprietary and I can't share, and B, I wasn't able to do many myself because I don't have a lot of really good data that I

456. 1:01:48

     work on. Um, yeah. So back there.

457. 1:01:54

     So our time is our most precious resource.

458. 1:01:57

     Mm-hmm.

459. 1:01:58

     How do you determine when fine-tuning is worth it versus integrating with a new model or getting better data?

460. 1:02:06

     Yeah. Maybe this is a little bit of a hot take, but I would avoid fine-tuning until you need it. Um, right? Like if you, um,

461. 1:02:20

     if you can get away with prompting, you should just use prompting. Yeah. Uh, and fine-tuning, the, the way you actually know... Sorry, I blanked for a second. Um, you really wanna evaluate, right?

462. 1:02:31

     Like if you, if your evals are showing that like you are reaching the limits of what you can accomplish through prompting, but you know it's better to do pos- uh, you know it's possible to do better, then that's when you can pull out fine-tuning, if you have the data, if you have everything set up.

463. 1:02:46

     Um, it is a big upfront cost, right? But if you, if you are in a space like, uh, you know, legal or like, uh, in like, um, hard sciences, like it can really be worth it to do something like reinforcement fine-tuning because the results you get are like impossible to achieve otherwise.

464. 1:03:06

     I have a question.

465. 1:03:06

     Mm-hmm.

466. 1:03:08

     Uh, what's happening on the OpenAI platform side when we do fine-tuning? So is it generating like a new set of weights, and then when you actually go to run inference, does it load those weights into memory and run on that?

467. 1:03:20

     Or like how does that work?

468. 1:03:23

     Yeah. So it is generating new weights. It's generating this LoRA component, right? So it's generating like this little piece that we use with the models. Um, the rest of the weights are the same in the model.

469. 1:03:32

     And so what it does when you use your inference is it loads in that part of the weights and then uses those. Yeah.

470. 1:03:40

     I had a question on DPO.

471. 1:03:42

     Mm-hmm.

472. 1:03:42

     Um, how bad should your bad examples be? [laughs] Like with a joke, should it be just barely bad, like it's non-obvious, and so you're squeezing out nuance, or should it just be like really bad?

473. 1:03:53

     I, I think you sort of answered your own question, right? If you make it too bad, then you are learning, um, how to go from like really bad to like good, which might not be what you wanna teach the model.

474. 1:04:05

     You might wanna teach the model how to go from like mediocre to really good.

475. 1:04:09

     Mm-hmm.

476. 1:04:09

     Right? And so yeah, like you wanna sort of mimic what you would expect to see. Um,

477. 1:04:15

     or, or you wanna think like what is that, um,

478. 1:04:18

     delta. I, I think... I haven't actually heard of too many other people doing like synthetic DPO with generated bad examples. Um,

479. 1:04:28

     most often it's just used for A/B tests, because the, the signal from an A/B test is real and pure, right? The signal from synthetic DPO is as good as your imagination, maybe, and like your intuition.

480. 1:04:43

     So I... Yeah. Like the best would be to like actually have people sit in front... I was thinking about doing this, like sending, sending everyone like a little website and having you label a bunch of examples.

481. 1:04:53

     But then I realized even if we did that, I wouldn't be able to train it live. Um, but that would be, that would be like good real signal, right?

482. 1:05:00

     Perfect signal for DPO. Uh, yes. You're gonna have to yell.

483. 1:05:07

     So I'm kind of curious about why RFT doesn't show up for non-reasoning models. Like say I wanted to give a model that only produced output with precisely 30% emojis.

484. 1:05:19

     I don't think I can do that with DPO, could I?

485. 1:05:22

     Uh, the question is why RFT doesn't work with non-reasoning models. What was the bit about emojis? I missed that.

486. 1:05:28

     Sorry. Um, if I wanted to get a non-reasoning model to-

487. 1:05:34

     Okay. Let me get you a closer

488. 1:05:35

     ... only output with 30% emojis exactly, I don't think I could do that with DPO, could I?

489. 1:05:42

     Um, exactly is tough, but I, I would bet that you could get somewhat close. Um, I mean, it depends. Are you trying to do like a v-- If it's a very specific constrained example-

490. 1:05:56

     The... Sorry. No, the more general one was there was an example for like where reinforcement learning with non-reasoning models might be, might work.

491. 1:06:06

     Yeah. Um-

492. 1:06:07

     Or it wouldn't be appropriate.

493. 1:06:08

     The main reason it doesn't work is because we don't support it.

494. 1:06:11

     Oh. [laughs]

495. 1:06:12

     Uh-

496. 1:06:14

     Can you repeat her question?

497. 1:06:15

     Yeah. So the question, the question is why doesn't reinforcement fine-tuning work with non-reasoning models? And it's because we don't support it. [laughs] Yes.

498. 1:06:23

     Stand up if you have a question.

499. 1:06:26

     Yeah.

500. 1:06:26

     Yeah, I have a couple of questions. So yeah, yeah. Uh, so does, does custom GPT you have right now, does it use one of those fine-tuning?

501. 1:06:34

     Custom GPTs?

502. 1:06:35

     Yeah.

503. 1:06:35

     No.

504. 1:06:36

     No.

505. 1:06:36

     It's just a prompt.

506. 1:06:37

     Okay. And second question is that, uh, for us certain applications, it's also important to have good thinking, CoT, you know?

507. 1:06:46

     Yeah.

508. 1:06:46

     But it looks like there is no reward for or penalty or anything for that. Like if, uh-

509. 1:06:52

     Um, yeah, I mean, we released a paper recently that talked about how you should avoid, um, directly evalu- like directly, uh, rewarding or penalizing certain behaviors in your chain of thought.

510. 1:07:02

     Because if you don't touch the chain of thought portion and only, um, put pressure on the results, um, essentially You get faithful chains of thoughts. You, you, you can actually see, um, you hopefully close to what the model is like actually thinking through.

511. 1:07:20

     Um, and so if it's doing something you might not want, it'll be clear. Um, but if you like, for example, reward, like investigate the chain of thought and like penalize it if it starts thinking about bad things, um, turns out, uh, you do reduce the ratio-- like the, the percentage of bad things by a little bit, but

512. 1:07:38

     you reduce the cases where you can catch them by looking at the chain of thought to like zero. And so it's like not a good idea to do it.

513. 1:07:45

     There's a, there's a paper on this you should, you should look it up, we released.

514. 1:07:48

     So o-o-other question is, uh, you know, like if you have a really good training data, right? Uh, then you can just distill the any small models or even open source ones, right?

515. 1:07:59

     I believe. So how... So like supervised training, and you also showed DPO.

516. 1:08:05

     Mm-hmm.

517. 1:08:05

     If you have a really good quality data and where things are deterministic, especially the questions, then I believe there is no point in going to bigger models like OpenAI, right?

518. 1:08:17

     Any open source models would do it. What, what, what do you think about that, you know?

519. 1:08:20

     Well, you know, if it's, if it's doing what you want, that's awesome. Like, yeah. Like j- go with every-- go with whatever is gonna get you the performance that you're looking for.

520. 1:08:30

     I think just in practice, you know, the... I think the real consideration is like, do you wanna fine-tune an open source model and deal with like, um... Like if you have the data for it, that's amazing.

521. 1:08:43

     If an open source model does what you want, use that. But if you like, if you have the choice between like fine-tuning an open source model, um, which involves all the like pitfalls that I just talked about, versus just like asking 4o, like with no fine-tuning, it's like

522. 1:09:00

     o-one of the is gonna let you move faster. You can always start with a bigger model and then like tune a smaller one, right? So I'd say always start with, always start with the biggest, most expensive thing that'll make your thing work, and then work on getting more efficient.

523. 1:09:15

     Yeah.

524. 1:09:16

     Over here.

525. 1:09:17

     Yeah. Given, uh, the knowledge that you shared with us, uh, today, can you give us a little bit of insight why o3 scores higher but, uh, hallucinates more than o1?

526. 1:09:30

     In general? Like why o3 hallucinates more than o1?

527. 1:09:34

     Yeah.

528. 1:09:34

     No.

529. 1:09:34

     You have that on the, in the system card, right?

530. 1:09:37

     Yeah, yeah, yeah. I mean, I don't have a, like a reason that I can give you. I think that's something we're investigating. Obviously, it's something that we wanna avoid, but like, um,

531. 1:09:47

     yeah. I don't think we have like a super solid un- like understanding we can share about why that happens. Um, and speaking for myself, I don't know. Yeah.

532. 1:09:58

     I, uh-

533. 1:09:58

     Yeah.

534. 1:09:59

     Thanks a lot. Th-this is great. Um, it seems like the hardest thing to do for, uh, reinforcement learning is the reward modeling or like defining the reward functions. Do you have any heuristics or tips or resources on like, uh, the number of reward functions or like the type of problems that it suits or doesn't?

535. 1:10:17

     I just feel like that there's like not that much to know when I'd like to reach for reward functions now and, you know, how, how I'd find that... It's, it's, it's almost like it's a s- a, a solution in search of a problem, you know?

536. 1:10:30

     I don't know when I would, when I would do this, you know?

537. 1:10:33

     When you would, when you would-

538. 1:10:34

     When I would use reinforcement learning and like kn- like find five perfect reward functions that model some behavior I want, want to capture.

539. 1:10:43

     Um, yeah. I, I think, I think when... It's not for a beginning of the problem, right? Y- let's say you already have something in place that's already working and like you're wondering if you could do even better.

540. 1:10:56

     Like the fact that you're asking yourself can you do even better means you already have a way to evaluate it and already have a way to grade it. And so that's the beauty of it, is like by the time that you're asking yourself, like, "Should I use reinforcement fine-tuning?"

541. 1:11:11

     You should already have all the data ready or almost close to ready, because all you need is like your-- whatever eval setup you have. Because that-

542. 1:11:19

     Yeah. So I, so I have like a good LLM judge, uh, that I'm using for evals, and then I take that LLM judge and use it as a reward function-

543. 1:11:26

     You can

544. 1:11:27

     ... to, to, to fine-tune.

545. 1:11:27

     You can, yes.

546. 1:11:29

     Okay. So that's the kind of path. It's like start off, build a good LLM judge that captures the task you wanna do well on, and then-

547. 1:11:35

     That's-

548. 1:11:35

     If, i- you know, try milk a model with like OR- OREL to make it better.

549. 1:11:40

     Yeah. I mean, that, that's one way for sure. And like, if that is the path that works for you, that's great. I wouldn't say there's one, like necessarily one path.

550. 1:11:48

     Okay.

551. 1:11:48

     So actually in the crowd is Teo. Can I-- Can, can you wave? So, um, he wrote a cookbook that I used to base a lot of this examples for.

552. 1:11:57

     Um, you should check it out. It's, I, I believe... What's it called? Can you yell out the name?

553. 1:12:03

     Exploring... I think it's Exploring Model Graders for Reinforcement Fine-Tuning.

554. 1:12:09

     Yeah. Yeah. Incredible cookbook. You should all check it out. It goes into like the nitty-gritty of like how to do each of these different parts. It talks about different graders.

555. 1:12:17

     It talks about like all the different pieces. So you should really check out, uh, that cookbook and the function calling one if you're curious. I'll see if there's a way for us to send it out.

556. 1:12:26

     Maybe I can just like pull it up real quick. Um,

557. 1:12:34

     I think it's this one. Yeah. So you should really check this out. Um, it's long, it's comprehensive, and it's like really good. Um, cool. Any-

558. 1:12:43

     Yeah. Uh, I had a question. Here.

559. 1:12:46

     Yeah.

560. 1:12:46

     Uh, does the fine-tune API support different modalities as well, or it's just text and-

561. 1:12:51

     Oh, we do have image fine-tuning. [laughs] I forgot. Yes. Wow. Okay. We have image fine-tuning. Uh, and so you can do image input and, um,

562. 1:13:04

     I believe it's text output. So it's really good for like bounding boxes and all these other parts. Wow. Thank you. It's like you're a plant. Uh-

563. 1:13:13

     Stand up.

564. 1:13:14

     Yes.

565. 1:13:15

     Um, so how do you, um, teach model, uh, new knowledge or domain knowledge? So, and then how do you use that work with, uh, reasoning? And, and one example is, like, if I have my personal experience, personal preference, should I do fine-tuning or should I kind of put it in context?

566. 1:13:33

     Um, I would say always try starting by putting things in context. Um, yeah, fine-tuning you wanna, like, leave for, like, a later stage thing. You already, you already have some data to, like, show one way or another.

567. 1:13:46

     Um, in order to, like, give models new knowledge, you know, you have, like, all, like, different ways you can do RAG and search. Um,

568. 1:13:55

     technically, when you fine-tune, you can put in small amounts of new information into the models. But in general, you should treat it as more of a methodology, right? Um, like, if you consider an algorithm data, like, in that it requires bits to describe the procedure, you're sort of teaching that.

569. 1:14:15

     But you, you can't really give it, like, um, too, too much data. However, you... I-it can, it can learn a bit, right? Like, SFT starts to memorize certain things that you give it.

570. 1:14:27

     Um, I just wouldn't say it's, like, the best way to give, like, correct information or, like, correct, like, referenced information. I'd say maybe it's good to, like... I would focus it more on formatting.

571. 1:14:41

     You can use it. Right, like, think, like, two years ago at this point, um, when we released, um, fine-tuning for GPT 3.5, one of the interesting things you can do is, um,

572. 1:14:55

     if you do retrieval, you can teach a model, like, with, um, like, embedding search, and you need to, like, embed the query. First, sometimes you wanna k- like, transform the user's query into something that looks more like your search results.

573. 1:15:10

     Um, I think it's, like, hypothetical document embeddings.

574. 1:15:13

     Mm-hmm.

575. 1:15:14

     Um, you can fine-tune a model to, like, do, like, essentially hallucinate that better, right? Like, if you train it on, like, the actual outputs, uh, uh, like, in your actual, like, knowledge base, then it's more likely to out- output something that is in distribution, um, for your knowledge base, and so you'll get, like, more direct mappings, if

576. 1:15:33

     that makes sense. So, but, but it's like you're not really teaching it too much new information. It's just helping you find... It's giving you a better intuition to find it.

577. 1:15:39

     Yeah. So, like, for, for a reasoning model, sometimes if I want to reference to custom definition or custom, custom knowledge in certain domain, is it good to kind of combine with RAG and then generate that kind of training data, train with RL or, or just, like...

578. 1:15:55

     I guess that's in general how to solve that problem.

579. 1:15:57

     Um, for that case, uh, I would just let the model... Like, give the model some search functions.

580. 1:16:04

     Oh.

581. 1:16:05

     Like, let the model do search. Um, if you have explicit search steps that you do beforehand, um, then you can maybe fine-tune, like, specific parts. But we-- Like, because search is, like, inherently a two-step process, it's like searching and then interpreting, you can't do that with reinforcement fine-tuning right now.

582. 1:16:23

     Mm-hmm. I see, I see. Thank you.

583. 1:16:26

     Yeah.

584. 1:16:26

     Uh, so for your ques- for your thing about the email classifier-

585. 1:16:30

     Mm-hmm

586. 1:16:30

     ... did you do the reinforcement learning just to kinda show us, like, a bad example, or?

587. 1:16:33

     No, no, no, I really hoped it worked.

588. 1:16:35

     Oh, okay. Uh-

589. 1:16:36

     That was me learning.

590. 1:16:36

     Well, it's so ambiguous, right? And then you were like: "Well, don't be ambiguous with it." And I'm like, "Okay." Uh, okay.

591. 1:16:40

     Yeah, yeah, no, thanks for calling me out publicly in a talk. But keep going. [laughs]

592. 1:16:44

     Well, I'm not trying to do that, but, um-

593. 1:16:45

     I'm joking, I'm joking

594. 1:16:46

     ... for the supervised fine-tuning, that would have been the... What, what was the correct answer there, then? Like, supervised fine-tuning given your dataset?

595. 1:16:53

     Um, so what I, what I wanted to try and, like, maybe if we have enough time, I'll, like, do a little bit of this, was, like, there are a lot of nuances around the email that I ha-- like, my preferences that are extremely low signal.

596. 1:17:09

     Like, I might see it once. Like, I might get two very similar emails from two very similar, like, uh, people, uh, or, like, like, sources, and one I decide to, like, glance and one I decide to, like, ignore.

597. 1:17:24

     How is it gonna learn that, right? Um, like, what I was imagining is, like, if you find cases that are similar, and you could do that with embeddings, um, but that have different results, you might be able to give them to, like, a reasoning model and be like, you know, figure out what's different here, and then make

598. 1:17:41

     a note of it. And then you can compile this set of, like, principles, essentially, that hopefully guide what it does. And so this is, like, what I would consider almost like prompt tuning, where you, like, are finding the right context to provide the model based on a lot of examples.

599. 1:17:59

     Uh, w-we can try in a little bit.

600. 1:18:01

     That would be cool to see. Uh, like, I feel like embedding-

601. 1:18:04

     Here

602. 1:18:04

     ... fine-tuning the embedding model itself, though, would almost be critical then to get... Right? For an individual, for an individualized, like-

603. 1:18:12

     Why? What-- How so? Why?

604. 1:18:14

     Because embeddings naturally don't understand the nuance of, like, how you-

605. 1:18:17

     Oh, no, no. I'm not saying using embeddings to find ca-- to, like, find the nuance. I'm saying using embeddings to find similar ones.

606. 1:18:23

     Right, but how would it know what's similar based on your preferences?

607. 1:18:26

     Not based on my pre- preference. Based on, like,

608. 1:18:29

     whatever embedding similar... Like, does it... Like, finding similar in the naive sense and then showing the model sim-- like naive similar cases that are actually different and being like: "Look, these are...

609. 1:18:43

     The embedding model thought these were similar, but they're actually profoundly different for some nuanced reason. Figure out what that is." Does that make sense?

610. 1:18:52

     That makes sense. Yeah.

611. 1:18:53

     Yeah. Um, yeah. I feel like people kinda wanna see me try this. [laughs] Okay, so I did start trying this. [laughs]

612. 1:19:07

     Um, I'm also gonna try not to flash too much of the data because this is my actual emails. Um, okay, so I have the email dataset.

613. 1:19:17

     I'm loading it. I have this, like, eval function. I have my instructions, and then I just made a little, like, helper that

614. 1:19:27

     essentially, I just did a little, like, fancy evaluation loop where you can define

615. 1:19:34

     an evaluator, and you can... Where's my code? Oh,

616. 1:19:41

     yeah. I'm sorry. The eval model takes in a run function that just returns a result. This is where I'm gonna pass the model. There, it, uh, takes in an evaluate function and then a dataset of workers.

617. 1:19:51

     This is just like a generic version of the other function I implemented. Um, cool, and let's see how it's doing. So,

618. 1:20:00

     you know, I have this prompt. Tried out all the models, and what am I getting right now? For a mini, it's fifty-eight percent, fifty-one percent worse, fifty-six, fifty-seven. So this kinda goes to show [chuckles]

619. 1:20:14

     there's no signal to learn right now, right? Like, uh, it's kinda random. Or like there's, there's no-- the very little signal there. So

620. 1:20:23

     let's think about this. How would you do this sort of prompt tuning? Um,

621. 1:20:29

     maybe even without the embedding models. We can, like, do these-- Oh, I think I started writing an algorithm somewhere. Where is it?

622. 1:20:39

     Ha. There we go. Look at that. Um, so this was sort of the idea that I had for fine-tuning, for prompt tuning, was, like,

623. 1:20:48

     run a forward pass, um, then sample from that training subset. Or, or sorry, split up into mini batches. Run a forward pass, um...

624. 1:21:02

     Sorry. Split into mini batches, each of which has a training subset and a test subset. Run a forward pass on the training subset, and then reason over it to see how it could have done better,

625. 1:21:16

     um, and then take that, put it back in the prompt, and then use that, run it forward again on the test subset, and then update the prompt and keep going.

626. 1:21:25

     Does this make sense? I don't know. I'm kinda, like, coming up with this on the spot, so we can see how it goes. Maybe before I jump into this, I wanna make sure there's no more questions if I, like, lose myself here.

627. 1:21:37

     No? Okay. People wanna see me. [laughs] Okay. So,

628. 1:21:44

     you know, what are we gonna do? Uh,

629. 1:21:49

     let's do training loop. And yeah, f- Should we just do this? Let's, let's do one run, right? So, like, we,

630. 1:22:03

     we have some, like, mini batch. Um, and then we have training

631. 1:22:19

     and te-- like, train test. Yeah. Now, if we have that,

632. 1:22:31

     then I want to run a forward pass. So for...

633. 1:22:40

     Is it just evaluate? I'll do this in a lazy way right now. So, you know, so sample and train. Let's do results.

634. 1:23:01

     Okay. So-

635. 1:23:06

     With the train test split, no?

636. 1:23:08

     Hmm?

637. 1:23:09

     How did you write the model?

638. 1:23:10

     Sorry, the model was so fast. Cursor is amazing. Uh, okay. So,

639. 1:23:19

     yeah, let's pretend that's correct. Okay, so we get train, we get test, we sample train. Um,

640. 1:23:29

     we should, yeah, we should also have some labels.

641. 1:23:34

     Here, shall we print what one of these... So we have-- Let's, let's go, let's do this, like, more accurately. So the samples have an action, and they have a text.

642. 1:23:45

     So for each sample, we wanna run it on...

643. 1:23:58

     Does the model take in the whole thing?

644. 1:24:01

     Yeah, it takes in a sample. Okay. So, so then this should--

645. 1:24:07

     Let's just, let's start here. Um, you know, process mini batch of, like, I don't know. Let's do twenty. [laughs]

646. 1:24:24

     Do I wanna print this? Well, let's see if this runs correctly, if this works. Okay. Um, I'll risk one.

647. 1:24:35

     This is one of the worst ideas I've had. [laughs]

648. 1:24:41

     Ah. I didn't know you could do top-level weights, by the way, in Notebooks. It's really nice. Um,

649. 1:24:51

     yeah, this should be in parallel. I'll just do ten for now.

650. 1:24:56

     Did it-- Was it happy? Uh, okay, so it's just the results. So maybe I wanna append,

651. 1:25:05

     like, sample. And then model output

652. 1:25:36

     Okay. Wow. Very, uh, topical. And in [laughs] what, what did it say was the output? Was it ignore? [laughs]

653. 1:25:44

     I don't know, that feels correct. [laughs] Um, okay, so now we have a... Oh, and then what was the

654. 1:25:52

     actual... Okay, wait, let's print, you know, uh, let's dump it, right? Um.

655. 1:26:11

     Huh, we have a blank inputs. My package has

656. 1:26:17

     ignored us. [laughs] Okay. Okay, no, this is good, right? So we have-- Let me just, like, quickly, eh.

657. 1:26:24

     There's a lot of blank ones. Hmm. Uh.

658. 1:26:41

     It's... I don't know what it's doing.

659. 1:27:10

     Whoa, whoa, whoa. Why is it still running?

660. 1:27:17

     Okay. Hmm?

661. 1:27:34

     Hmm. Good call. Oh, this is so much more fun with people, like, debugging with me. [laughs] [sighs]

662. 1:27:49

     Okay. Uh, okay. So how should we do this? Should we like-- I guess we could just give the entire mini batch to a model. That's the easiest thing. And then just be like, you know, what did...

663. 1:28:04

     Like, think about what you got wrong, and then give me some notes.

664. 1:28:09

     Uh, so we'll have results, and then we can make a, you know, like...

665. 1:28:15

     Where is it? You know, um, extract insights, and then we can do, uh...

666. 1:28:32

     I'll use o3.

667. 1:28:57

     Okay. Any prompt engineers wanna shout out some prompts? Let's see. So we have model.

668. 1:29:03

     We have the results. We'll say like, you know, um,

669. 1:29:10

     uh, you are provided a set of results, um, from a forward pass to a model.

670. 1:29:24

     Um, given these results, task is to... Let's see. [laughs]

671. 1:29:42

     It doesn't need to know the format. Um, it's critical these results follow,

672. 1:29:50

     um, [laughs] uh, not, not that fast. [laughs] Uh, you know.

673. 1:30:05

     Um, um- You know there's an app that's really good at making prompts for you, ChatGPT. Yeah. Well- It's insane. Okay, here's the funny part. So there is a,

674. 1:30:21

     um, uh... It was actually with, uh, with, um... I helped create this, uh, prompt generator.

675. 1:30:30

     I just don't wanna go that far 'cause then it's more for me to read. [laughs] Okay. Um, gen to,

676. 1:30:39

     you know, emails outside of this batch of results. Um, the duplicated. In general, the output should be at most

677. 1:30:57

     one word. Uh, eh. And then what's the last one? It's like...

678. 1:31:21

     Is this, uh... Oh, uh, you know, do not...

679. 1:31:28

     I'll do this here Make sure the sites do not

680. 1:31:37

     overfit to the current batch of results. Uh, should it be a JSON array? I'll keep it as a string for now.

681. 1:31:54

     Simple string. Okay, so now we have that. Let's see how it does on this.

682. 1:32:05

     Um, man, this is like... Okay, yada, yada, yada.

683. 1:32:16

     Code. What did I call it? Extract insights. Maybe I'll give it reasoning. Uh, I'll, I'll do like that, and then results.

684. 1:32:44

     Huh?

685. 1:32:53

     Okey-dokey.

686. 1:33:13

     How do we feel about this prompt? I don't know. I feel like everyone's kinda just watching, thinking this is gonna work. I don't think this is gonna work, but, like, it'll be an interesting experiment.

687. 1:33:26

     Okay. I'd say this is too long. Process.

688. 1:33:46

     Um, okay. So let's run this on some other ones,

689. 1:34:12

     and now let's take this and use this as a negative example. Be like,

690. 1:34:20

     numbers. And I'm not using the exact same one for anyone who

691. 1:34:41

     might point that out. Am I? Hmm, maybe I was.

692. 1:34:49

     Well, it's fine. We'll see. In the meantime... Okay, so let's say let's pretend this is good, right? Let's pretend this is good. Then what do we continue doing in this, in this process mini batch?

693. 1:35:01

     So we, um, so here we did the forward pass, so we did results. Um, and so now we can say insights is await those. Maybe I'll move that up.

694. 1:35:18

     What's it say? Eh. We can, we can, we can see. We can work on the prompt later.

695. 1:35:29

     Um, so I'll take extract insights, move this up.

696. 1:35:43

     Now I have it in here. Okay, so now we have insights. So now what do we do? Um, maybe we can do, like, a while true. So,

697. 1:35:59

     so we get the insights, and then maybe let's... [coughs]

698. 1:36:04

     Actually, let's measure performance on tests. So we can do, no, eval model.

699. 1:36:17

     This is correct, valid sample. Um, validated results.

700. 1:36:27

     Let's print these. Let's skip this for now.

701. 1:36:42

     So zero point six. So then I guess the idea is if we keep doing this and putting it back into the prompt,

702. 1:36:50

     it should get better, maybe. And then I'll try the bigger, with a bigger mini batch.

703. 1:36:59

     Well, you know what? Let's, let's cheat a little and try with both the tests and the training. So, you know,

704. 1:37:08

     uh, test results, and then we can see how, how much we're over fitting. Train results.

705. 1:37:18

     Um, cool. Okay, so we have-- We're printing both. We have our insights and what we wanna do. Maybe just, like, we could just, like,

706. 1:37:31

     have a little portion that is ready. So model,

707. 1:37:38

     make model Of... Oh, are we even passing in the prompt?

708. 1:37:52

     Yeah, 'cause it's default to that. So we could,

709. 1:37:59

     you know, always make a new model. Instructions equals,

710. 1:38:11

     you can say equals that. So at each, and then we have a notes that starts out empty.

711. 1:38:29

     And then we do instructions plus notes. I don't know. [laughs]

712. 1:38:53

     What do people think? What's gonna happen? Okay.

713. 1:39:01

     Are you okay in the prompt?

714. 1:39:04

     No. [laughs] Very good call, very good call. So we could just make the notes this.

715. 1:39:16

     Um, maybe I should print out what the notes are each time.

716. 1:40:02

     So while this happens, I think the idea here would be... Oh.

717. 1:40:16

     That's, that's pretty surprising. We're already doing pretty good. Um,

718. 1:40:24

     so yeah, I, I guess I have zero confidence this is gonna work, but I think this style of approach can work, right? Where, um, you can take, and you can do this in, in like a tree-like structure as well, where you can take, um, maybe subsections, these mini batches, and then let it reason over the failures, and

719. 1:40:43

     then maybe produce these insights. Then you have a choice as to how you want to, like, c- um, aggregate those insights. Here, I'm just s- replacing the notes and within each mini batch, and then the plan was to like, um, append all the different notes together from all the different mini batches and put it in the prompt

720. 1:41:02

     and see if that can, um, perform better. Why is it stopping?

721. 1:41:08

     You gotta return in the loop. [laughs]

722. 1:41:13

     Thank you. Um, I'll leave this up in case anyone sees any other mistakes. This is good to have there. But yeah, uh, this is like the skeleton of something that could be very interesting, obviously with a better prompt, obviously with better pieces.

723. 1:41:30

     Um, but yeah, I don't know. Any, any questions, ideas, thoughts? If not, I might just like call it, uh, after we see what the results are.

724. 1:41:41

     Can you share some examples that work?

725. 1:41:45

     For this?

726. 1:41:47

     Yeah, between the tasks.

727. 1:41:47

     For this? I'm making this up in front of you.

728. 1:41:50

     Anything, yeah.

729. 1:41:51

     Any examples that work for anything? [laughs]

730. 1:41:54

     No, no, no. Any example in your file work for it, you know? Except this.

731. 1:42:00

     Um, can I show examples that have worked for this technique?

732. 1:42:04

     You don't have to share this data, but any prompt engineering that you provide, can you share some examples? That would be good to get inside people's minds. Like, I understand, uh, this example, but is there any other example you can share?

733. 1:42:20

     Well, so y- you're, you're asking... Are the numbers moving? Not significantly. Okay, I'm gonna call it. This, this doesn't work right now, but it's good, it's good enough to like be an example.

734. 1:42:30

     Um, I think to, to answer your question about, you know, what, what are the good techniques, um, for prompt engineering, like if I can share some, I, I think they've just been changing so much.

735. 1:42:38

     There's many, many, many like prompt engineering guides and like what you would wanna do is like...

736. 1:42:44

     I guess you're, you're asking both about prompt engineering, but maybe also something like this. Prompt engineering, there's so many resources. I'd say the biggest one is just be clear and don't have any like contradictions.

737. 1:42:55

     Um, and then few-shot examples are really, really good.

738. 1:42:58

     Um, for something like this, I'm like making this up as we go. Uh, I think like, again, I think the shape of this is something that can make sense.

739. 1:43:06

     Um, I would've been very surprised if I would've gotten any like good results. Maybe I'll say like a few minutes after trying to get this work. But, um, yeah, question back there?

740. 1:43:19

     Can you explain what the hell you're doing?

741. 1:43:21

     Yes. So the question is, can I explain what the hell I'm doing here? Um, I think the idea here is to instead of tuning the model itself, um, tuning the, the prompt and specifically tuning a subsection of the prompt that I'm calling notes that essentially, hopefully

742. 1:43:46

     contains like- jots about the emails that it's seen and anything that they might have found unintuitive about classifying that email. Maybe that's something I should include in the prompt.

743. 1:43:59

     And so the hope is that as it sees more emails, uh, like with the what it actually classified it as and what it was supposed to classified as, um,

744. 1:44:11

     b- by using these reasoning models, hopefully extract some like explicit insights of like, oh, you know, like maybe Ilan like, um, wants to like, you know, if, if, if he sees like a package has been delivered, um, I'm ac-- This one's probably glance.

745. 1:44:27

     This is what I mean, like my data is very noisy. Um, but like he will want to like glance at like, um... Oh, I said ignore. [laughs]

746. 1:44:36

     Like he-- I might wanna glance at this instead of like archiving, right? And so like, but, but the question is like, but why, right? I didn't say why. I'm just saying what the correct answer is.

747. 1:44:47

     Um, and this is why I thought RFT might be, might be good, but like this is way too noisy and subjective. But that doesn't mean you can't use reasoning models to still extract interesting insights.

748. 1:44:57

     So the idea would be like give enough examples of both successes and failures, um, that the model itself has done to a reasoning model and have it come up with like

749. 1:45:09

     notes or techniques that the model can use to hopefully improve the next time.

750. 1:45:15

     So in this scenario, could you go a step further and also like use this to identify, um, data points that you might be missing, like what time of day does your hard work happen?

751. 1:45:28

     Is it helpful to-

752. 1:45:29

     Yeah. So the question is like, can I use this to identify pieces of information that are missing? And I think yes, absolutely, and I think that's like very important for even our...

753. 1:45:38

     Like if you w-- if you do wanna fine-tune, but even if you don't, like this, this approach can like tell you a little bit about your data as well.

754. 1:45:46

     Yeah. So like just... It's like having someone read through it, um, and like write these things. Yeah. Um, if there's more questions, I'm happy to keep answering. Um, otherwise, I'm going to officially call this.

755. 1:46:00

     You have now seen two failures today, but that's okay. We learn from those. Um, so thank you. [laughs] [clapping] [outro music]