OpenThoughts: Data Recipes for Reasoning Models

Talk transcript

1. 0:00

   [upbeat music] Yeah, I'm Ryan.

2. 0:16

   I'm a founding engineer at Bespoke Labs, and today I'm gonna talk to you about OpenThoughts, which is our project to create the best open source reasoning datasets. And I'll be switching tack a little bit from our earlier discussions on reasoning and RL, and focus on the reasoning part, and you'll see why.

3. 0:35

   So just so we're on the same page, we've talked a lot about reasoning, but what's actually going on here? So I like this graph from Jason, which shows this incredible performance that's happened in the last several months, where models are getting much, much, much better on certain benchmarks.

4. 0:52

   Um, and if you look at that, this is reasoning, this is test-time scaling. I think everyone here is quite familiar with this, and it seems that certain tasks like AIME, which are competitive math problems, really respond to models when they're able to think step by step and do these long chain-of-thoughts.

5. 1:11

   Um, so let's go back to DeepSeek-R1. Now, DeepSeek-R1 was really impressive for a lot of people for a lot of reasons, and RL was a big part of that.

6. 1:21

   But I was also particularly interested because DeepSeek-R1, at the end of the day, is an SFT model. So the final weights that they've released are actually from DeepSeek-V3-Base, which is fine-tuned on 800K SFT examples, 600K of which are reasoning.

7. 1:40

   Of course, you can see here that RL was a, a big part of it, and RL was used heavily to create that model which generated this data. Um, but at the end, it was SFT and a little bit of RL for alignment.

8. 1:53

   So this is really interesting and surprising. The other thing that was really interesting and surprising to us was these small reasoning models that DeepSeek released, which were incredibly strong.

9. 2:03

   Um, and this for us was a, a huge motivation, a huge motivation to try to do this ourselves. And why is that interesting? Because if we go back to here, none-- no a-additional detail was really given on these datasets here.

10. 2:19

    So if you wanna create strong reasoning models, we now sort of have a training recipe, but we don't have the data recipe. That's the missing link. Okay. I want to also include a slide here on why is it interesting to train your own reasoning models.

11. 2:35

    So, uh, I'm partially taking this from Amir's talk yesterday on open source and enterprise, which I really liked. But there's these main points: performance, privacy, speed and cost, and then ownership and destiny.

12. 2:46

    I think, um, using reasoning is a, is a great tool to solve a problem, and you shouldn't lim-limit yourself in your toolbox if you're trying to solve a specific domain task.

13. 2:59

    So, uh, as we talked about before, RL is a great tool in this toolbox to tackle, to tackle reasoning tasks, but we're gonna see here that SFT is, as Nathan put this morning, extremely easy and extremely effective.

14. 3:12

    Okay, great. Now, the missing link. How do we actually solve for this, this reasoning data recipe? There's all these questions that we had when we started. How much data do you really need?

15. 3:24

    What data creation steps are necessary? What are the optimal choices for each step in that data creation pipeline? And then how do you even go about figuring all this out?

16. 3:35

    And this, this is the meat of the OpenThoughts project. So today, we're excited to announce OpenThoughts 3, which is hot off the presses, just came out two hours ago, which is our latest and greatest version of our reasoning datasets.

17. 3:48

    And yeah. [laughs] [audience applauding] Thank you. And now we-- this is the state-of-the-art reasoning dataset recipe.

18. 4:01

    So you can see here, these graphs are showing accuracy on three of these reasoning benchmarks: AIME, which is competitive math; LiveCodeBench is competitive code; and GPQA Diamond, which is our science questions.

19. 4:13

    Um, on the Y-axis, you see accuracy is going up. Uh, on the X-axis, you see the data scale is going up. So we, we heard before that scaling is difficult, particularly difficult with RL.

20. 4:25

    The good news is for SFT, scaling is quite easier. Um, you can see here we compare to other open reasoning datasets. So Nemotron Nano. NVIDIA released this great model, Nemotron Nano.

21. 4:36

    It's a AP model, and they also released the dataset to train on it. So we compared directly by training on the same base model between our dataset, which is our dataset recipe, and the Nemotron Nano data, which is the NVIDIA recipe.

22. 4:49

    And you can see here there's a significant gap. So we, we shifted this scaling curve upwards.

23. 4:55

    Great. So the-- Yeah, this is the state-of-the-art 7B open data reasoning model. You can see we've had-- we have measured across the domains of interest, so science, code, and math, and then a couple held-out benchmarks.

24. 5:09

    So our original goal was to, to reproduce, to find the missing link for the DeepSeek Distill models. And you can see here we've crushed that goal. So we're, we're significantly outperforming the DeepSeek-R1 Qwen 7B model, which we started off trying to reproduce.

25. 5:26

    And then compared to the Nemotron Nano model, which is trained on a different base model, um, we are also outperforming on some benchmarks and similarly competitive on some others.

26. 5:37

    So okay, let's actually talk about how we achieve this. This is the interesting part for you. So if we go back to the scaling graph, you can see, um, once again on the X-axis, we're scaling dataset size.

27. 5:49

    So, uh, this is a, a, a huge method to in-increase accuracy, and the thing here is it gets more and more expensive. Exponentially, it gets more expensive as you keep going.

28. 6:01

    Um- And then, uh, um, vertically, you can see that we've shifted this, the scaling curve up. So this is what I was talking about before. This is the improving the dataset recipe.

29. 6:11

    So given a fixed dataset recipe, you can always scale it larger and you can always have higher performance. But, um, if you wanna push your performance to ax- absolute maximum, the real question is: How do I create the best dataset, and therefore, what is the best recipe for the dataset?

30. 6:27

    Okay. So, uh, en-enough teasing here. Let's go, let's go into the meat of it. So this is, this is how we approach this problem.

31. 6:36

    We broke down the dataset pipeline into sourcing questions, mixing different sources of questions, filtering those questions, filtering out the high- highest quality questions, generating answers with a teacher model, so that's distillation, and then filtering out bad answers.

32. 6:52

    Um, and, and lastly, at the end of this entire experimentation, we looked at what, what are the best teacher models? Which, which teacher model should we select? So through this entire pipeline, we've, we've come down to this final dataset recipe.

33. 7:04

    Now, this was a ton of work. This is a screenshot of our, our Hugging Face page. So you can see, created over five thousand datasets and almost three thousand models.

34. 7:13

    Um, for this project, it was only around a thousand experiments, but just to give you an idea of how rigorously we looked at the different decisions in each of these steps of the pipeline.

35. 7:23

    And also, I think this is interesting because it, it peels back the curtain a little bit on maybe what the frontier labs are doing. Uh, finding signal at the smallest scale possible and trying out as many things as possible and empirically choosing the best, and then scaling.

36. 7:39

    And often sometimes when you scale, you see, okay, what was the best of the small scale doesn't actually work. But if you're lucky, um, and you've done good science, then you'll, you'll-- your YOLO run will be the best possible, right?

37. 7:54

    Okay. So these are the, the key learnings that we had from our dataset recipe and, and this is what you can take away.

38. 8:02

    So the first thing is that pretty surprising, sampling multiple answers, so multiple reasoning traces per question in your dataset works really, really well. Um, the, the performance does not go down at a fixed scale if you take a fixed scale of questions, say thirty k questions, um, or so thirty, thirty k examples, and of those, you-- if

39. 8:25

    you take just thirty k questions and you only sample once per question, that performs pretty similarly to, um, if you took one sixteenth, so thirty k over sixteen, and then for each, you sampled sixteen times, which is quite cool.

40. 8:41

    So this allows you-- This is really cool 'cause this allows you to scale by sixteen x, which is more than an order of magnitude. And if you remember the graph from before, that corresponds to a pretty large increase in accuracy.

41. 8:52

    But the other surprising thing that we found was that a better model in terms of its own performance on evaluation benchmarks does not necessarily mean it's a better teacher model.

42. 9:03

    I think a good way to think about this is a brilliant researcher who's maybe a terrible lecturer, right? [laughs]

43. 9:10

    Um, we found specifically Qwen-32B was a stronger teacher model than DeepSeek-R1. So we switched to that in our, in our recipe, even though previously everyone has been using R1.

44. 9:23

    We also found that the syn-- the sources of data that had synthetic questions were actually quite good. Um, some of the top sources that we selected were entirely synthetic and better than sources, say, that scraped from forums or had humans manually write things.

45. 9:39

    And this is also really good news because synthetic question generation is scalable. So once again, we go back to the x-axis and we can push even further, which is, is accuracy boost.

46. 9:51

    So question filtering also works well. Here we, we filtered questions by having-- asking a language model how difficult is this question, and then taking only the hardest questions.

47. 10:05

    We also had a language model try to answer that question and looked at the length of that answer. So these are sort of proxies for the same thing. You can imagine that if a problem is a lot harder, then a language model will think more and it will produce more text, so its, its answer will be longer.

48. 10:22

    And these things worked better than embeddings-based approaches or fast text classifiers, which is interesting as so much that those, those approaches were typical for pre-training. So it seems that the, the filtering for data in post-training is quite different than pre-training.

49. 10:39

    Okay, some things that didn't work that were also quite interesting. Uh, through our experiments, we saw that choosing a smaller number of high-quality sources was much better than trying to optimize for diversity by going for a larger number of sources.

50. 10:50

    That's very counterintuitive, right? You'd think, "Okay, I'm always gonna go for, for higher diversity," but this is actually not what we saw. Um, the last thing we-- it was interesting is that people talk a lot about, um, verification, which is obviously very important for RL, and we actually see for SFT and distillation, it didn't seem that filtering

51. 11:08

    based off of the answer or verifying the answer really helped at all. This was quite surprising. Um, and I think there's, there's some, some good research in the literature about maybe why this is because if you have the, the hardest problem, it might be still helpful even if you have an incorrect answer to that hardest problem, um,

52. 11:27

    keeping it in and s- and seeing how the teacher model attempts. It's not just the final output that matters.

53. 11:34

    Okay, great. Okay, so this is-- those are all, like, the amazing learnings that we had for OpenThoughts 3, which super excited to share. But now you're probably thinking, "Okay, they've, they've done a thousand experiments.

54. 11:44

    I don't wanna do a thousand experiments. I still wanna create reasoning models. Uh, how do I adapt this if I wanna create specialized reasoning models?" Um, so I guess the first thing I would say is Be aware that based off of your domain, these exact choices might be a little bit different.

55. 12:00

    I would suggest, okay, start with our recipe and then iterate on it. If you have, um, capacity and compute, try a couple different choices for each step in the pipeline.

56. 12:08

    And I think a good example of this is we studied each step in the pipeline differently by domain. So we studied it distinctly for code, science, and math. And we saw, for example, in the question filtering, which I talked about before, um, using difficulty labels worked well for code questions, but for math and science, it was a

57. 12:28

    response length. And if you think about that for a second, it makes a little-- it makes sense because the response length for coding questions are very different, right? For, for, um, AIME math, it's literally just a number between zero to thousand.

58. 12:41

    So the, the answer is not-- it's not considering a, a large portion of the length. But you can imagine there's very simple coding questions in which the answer is still a lot of lines of code.

59. 12:51

    Um, so yeah, this is one thing to be aware of. The other thing which I talked about previously is synthetic question generation. Because it works so well, um, and if, if your specialized domain, if you're-- if you don't have a lot of data for your particular problem, then, uh, go ahead, transform that existing data into questions, expand

60. 13:09

    it, um, throw those as, as in-context examples, and just-- and generate more data. So yeah, we built an open source library for this. It's called Curator, and you can, you can try that out.

61. 13:20

    And then lastly, I feel like everyone says this, but it can't be said enough. Like, the, the evaluation is paramount. If you don't know how well your models are doing or improving, then you cannot make good principled decisions about your dataset recipe.

62. 13:33

    Um, we spent a lot of time on this. We also have this open source library on GitHub called Eval alchemy, uh, which takes a ca- takes care of this and also takes care of the, um, sharding and parallelism.

63. 13:45

    And, and the key thing here is for very small evaluation sets, if you, if you only have a handful of questions, you should run your model on those evaluation sets many times in average.

64. 13:55

    So going back again to AIME competitive math questions, there's only thirty per year. So, uh, for our evaluations, we gave the model those thirty questions ten times, and then we averaged to get the, the f- the final signal to determine, um, which data strategies were working better than others 'cause otherwise there's too much noise.

65. 14:16

    Okay, this is also very, very interesting and surprising and promising for you if you're specializing.

66. 14:23

    It seems that you can actually surpass the teacher in some domains with distillation. This is, this is super cool. Usually, you think about only RL can push the frontier.

67. 14:31

    Distillation's just about catching up to the teacher. But no, that's not the case. So we have an example, it's in our paper, where, um, we looked at the legal reasoning domain, so the problem of classifying Supreme Court decisions.

68. 14:45

    And what we did is we took two K unique questions, we sampled five answers per question, and then we did do verification here, which, which did matter. So we threw away any questions, any answers that were incorrect.

69. 15:00

    Um, and when you fine-tune the 7B model, it surpasses R1, which is a very strong reasoning model and also a very huge reasoning model. So this is very exciting, and there's a lot more, um, research and also application to be done here.

70. 15:16

    Okay, cool. So everything's open. It's OpenThoughts, and OpenThoughts means open. Go out and build. We have all of our, um, we've got our detailed paper. It's just out this morning.

71. 15:27

    We've got the weights dataset. Uh, we have a ton of repos, uh, for code, for data generation, for evaluation, and synthetic data.

72. 15:36

    So check those out. Um, this is, this is the team. It was a huge group of people, uh, a lot of work over many months. Uh, I think we're all very proud of what we did, but there's lots of people to recognize here.

73. 15:47

    If you, uh, scan that QR code, it goes to the tweet, and everything, uh, about the OpenThoughts project is linked in from there. Yeah. Thank you. [audience applauding]

74. 16:02

    All right. Thank you so much, Ryan. Um, that was fascinating. Looks like we're already getting-- We have at least one question lined up. Again, we have time for maybe a couple of questions.

75. 16:11

    So if you have questions, um, please, uh, line up and, and we'll do it. Um, actually, before we get to those questions, I will say as people are leaving, um, we are gonna be back here at two o'clock.

76. 16:23

    We've got an excellent afternoon planned on this track. We've got Nathan Lambert. Um, we've got the, uh, we've got Christian Seide, who's the co-founder of X. Um, and it's gonna be a really great track at two o'clock back in this room.

77. 16:34

    Also, one more thing, if you do have questions for any of the speakers from this morning, um, hopefully they're gonna be able to stick around. Don't let them go to lunch.

78. 16:40

    They're gonna be-- They're, they're sitting up here at the front, so swarm them as soon as we're done. But for now, let's, uh, let's get a couple questions for, uh, for...

79. 16:46

    Go ahead. Um, yes. Over there.

80. 16:48

    Uh, thank you. Great talk. So, uh, two questions. One is, um, if you're just using SFT on this data, what's the difference between this and regular SFT?

81. 16:57

    This is just regular SFT.

82. 16:58

    Oh.

83. 16:59

    Yeah.

84. 16:59

    Oh, okay. So then how is regular SFT able to make the models, like, think longer? Because I thought for the reasoning models, they have, like, this thinking block, and they think-

85. 17:08

    Yes

86. 17:08

    ... for, you know, hours and minutes and-

87. 17:10

    Exactly.

88. 17:10

    So how do you, how do you-- how does F- SFT make it think for hours?

89. 17:14

    So you're, you're doing supervised fine-tuning on the questions, and the answers also contain the thinking. So the model learns to use its context window and produce these long thinking traces.

90. 17:24

    So i-it can do this, people call SFT imitation, um, but it, it can learn to learn this format in the same way. Yeah.

91. 17:32

    Thanks.

92. 17:33

    All right, we'll take one from this side.

93. 17:35

    Um, great presentation, Ryan. Uh, one question. Uh, why do you think, um, a smaller model like Qwen 32B was a better teacher than a DeepSeek-R1? What was your insight in-

94. 17:48

    Yeah

95. 17:48

    ... figuring out that, like, a good professor makes a bad lecturer?

96. 17:52

    Yeah, that's a great question. Um, I think this is something we need to investigate more, but you can see that- Uh, when you look at charts of the length of reasoning traces, you can see the distributions are different.

97. 18:04

    So, uh, it might be the case that you're using more of your context window, using more tokens, more steps. It also might be the case that you just have, uh, a better formatted response, better output.

98. 18:14

    Um, this is like an- another o- great open research, research question. I- interesting, I'll also say on this point, we also tried Claude as a teacher, which is like a very- is a good strong model, and it was just a terrible teacher.

99. 18:25

    Um, so there's the, it, it, yeah, it's interesting what can s- what actually creates a good teacher. Yeah.

100. 18:31

     All right, we'll take one more very brief question from this side, and then those of you still waiting on questions, um, after, uh, after we have closed this up, swarm in.

101. 18:39

     So- so, um, great talk, Ryan. Um, we're doing similar kind of thing, but I just had a question. Do you guys have any like pattern map as to in the reasoning chain of thought when things don't work, at what level, you know, in the eval do you find out that things are not working or do, it's not

102. 18:56

     reasoning correctly? Is there a pattern map or something that you have in your open source wrap?

103. 19:01

     Is, sorry, I didn't catch that. Is there a-

104. 19:03

     So if there are five steps of reasoning-

105. 19:04

     Yes

106. 19:05

     ... to reach a final conclusion, uh, at what step does the reasoning go awry?

107. 19:09

     Yeah, this is, this is a great question. We don't do this fine grain analysis, but there is a ton in the literature about this, um, where, yeah, there's a sort of critical step where it get, gets things wrong.

108. 19:19

     Um, there, w- we did like the simplest thing possible, right? You could also go in and try to do more complicated things, um, at evaluation time where you're doing interventions to, uh, maybe detect steps that have gone awry and, and, and change.

109. 19:35

     Or you can do this in the, when you're creating the data set. So you could potentially rewrite things. But everything that we tried in terms of like messing with the reasoning trace, it wasn't helpful.

110. 19:46

     Um, so yeah, I think there's still more to explore there. There's like, this is really just the start of everything in reasoning. [upbeat music]