Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It

Talk transcript

1. 0:00

   [upbeat music] This is a discussion that I'm particularly excited about because the field is moving so fast, and we have two people that, that kind of have this unique vantage point on the field.

2. 0:22

   And what I wanna do is kind of ask questions to see if we can learn from that. So, uh, I wanna start off with intros. Talk a little bit about your role, what you're thinking about, what you're working on.

3. 0:31

   Maybe Dan, if you can go first.

4. 0:33

   Yeah. Uh, hey everyone. Uh, I'm Dan. I'm the VP of Kernels at Together AI. Uh, I lead inference, uh, GPU optimization. I'm trying to figure out how to use GPUs most effectively to serve AI models.

5. 0:45

   Yeah. So one of the things that I wanted to dive, uh, in, uh, with, with Dan about is like new model drops. What are... What is everything that goes on behind the scenes to serve it so that everybody here, there's a lot of builders here that can use it.

6. 0:57

   Um, Olive, I want to throw it over to you to talk about your role and what you're focusing on.

7. 1:01

   Yeah. Um, I'm Olive, and I am the research lead of RL at MiniMax. And I'm responsible for the final training of the model and the shipping of the model.

8. 1:09

   So basically, everything before the inference, right?

9. 1:12

   Okay. Awesome. Um, so maybe I want to start off, this panel is focusing on open source. I wanted to start off, uh, with this is your strongest model yet, MiniMax-M3.

10. 1:23

    Um, why open source it? What's the kind of the, the idea behind that as a company, uh, as you're releasing these models?

11. 1:29

    We do believe that, um, the open source community as a whole is very str- strong and powerful. While we open source the model, everyone can use it, so it aligns with our mission that we want to have intelligence with everyone.

12. 1:42

    And also different developers can contribute to the model through feedback, through their own PRs, and we can build the models even stronger. And also like, for example, then you, you will be able to optimize on our open-weight model and make it inference faster and then serve better for everyone.

13. 2:00

    Yeah. Yeah, we're, we're big believers in open source at Together-

14. 2:03

    Yeah

15. 2:03

    ... and, um, yeah, I think when we, we've been, you know, following you guys for, for a while, I think from way o- old, older MiniMax models. So seeing M3 and seeing how far it's come is, is really impressive and really great.

16. 2:15

    Mm-hmm.

17. 2:16

    Yeah, so I wanted to kind of pick on, uh, this a little bit more. Can you explain... So we've got the model creators themselves, MiniMax. We've got, um, experts on the inference side of things.

18. 2:26

    How did this partnership come to be? So they launch an open source model, uh, and we're now distributing it. I, I checked this morning. We have the lion's share of, uh, token usage for MiniMax-M3.

19. 2:37

    Um, w- how does this partnership come to be, and how do we serve a model like this at scale?

20. 2:41

    Yeah, yeah, great question. So at Together, I think one of the things that, that we're really interested in is how do you, um, make intelligence abundant. So how do you get more tokens for more people to do, um, more useful things and, and, and get, uh, get all these capabilities into more people's hands.

21. 2:58

    So, uh, we follow all the open models very closely. Um, I, I don't remember when exactly we, we started part... Oh, actually, I think I do know this. Um, we had a car event in Las Vegas sometime last year and w- someone from MiniMax came.

22. 3:12

    And there he was like, "Guys, you really gotta serve our next model. It's gonna be really, really great." Um, so I think from there we, we started talking. We were serving MiniMax 2.5-

23. 3:21

    Mm-hmm

24. 3:21

    ... um, and I think 2.7 for, for a while. Uh, and then when leading up to the launch of, of M3, uh, we were, uh, we were quite excited about it.

25. 3:30

    I think we, we were seeing the, the usage and what people were, were doing with it. It was, it was really quite exciting. Um, and, and so from there, uh, that's really where we partner.

26. 3:39

    We start working on the model, the architecture, optimizing it, uh, figuring out, you know, what's the best way to serve inference on it, um, and, and all those great pieces.

27. 3:49

    Yeah. I wanted to actually get into more on the model side of things. And so as the creator of a model, as, as somebody who's like post-training this thing, the model lands and all the builders that are here start using it.

28. 4:01

    From your perspective, um, what are the kinda the unique capabilities that you, you love to see people use it for? And what are maybe some of the hidden gems that you haven't...

29. 4:09

    You thought, "Oh, people would love to build this," but you haven't seen a little bit. Could you shed more light on that?

30. 4:14

    Mm-hmm. Mm-hmm. So MiniMax-M3, which was different from the M2 series, was that it was actually multi-model.

31. 4:20

    Yeah.

32. 4:21

    So it not only understands text and it not only writes code, it also understands videos and images. So we did see a lot of applications on agentic, uh, multimodal agents, which is very cool.

33. 4:33

    Um, and I would say there are a couple that we can highlight, right? For example, computer uses. The model can-- is able to navigate through a computer and then do some pretty good creations with, uh, the tools that they, they can, uh, utilize.

34. 4:47

    And also, um, you can develop games with the model. Um, it's very fun that, uh, I, I don't... I think that's one of the hidden gems is that we actually worked on, um, game development.

35. 4:58

    So the model can help you develop, uh, real cool games. Um, yeah.

36. 5:02

    Yeah. One, uh, so when the, when the blog dropped and then, uh, the paper dropped, one of the things I noticed was that you guys highlighted, uh, SVG bench, you guys highlighted KernelBench.

37. 5:12

    You also touched on OS World.

38. 5:14

    Yeah.

39. 5:14

    So can you talk more about, like, what does it take to post-train, especially for those particular domains?

40. 5:18

    Mm-hmm. Mm-hmm. Uh, I would say is, uh, the very important thing is the data and how we define the problems.

41. 5:26

    Yeah.

42. 5:26

    Um, and it could be very different from different tasks. For example, uh, let's say the kernel one, right? The, it would be very important to design the environments of the data so that we can deliberately train reinforcement learning in those very complex environments and to let the model to optimize the kernels themselves and iteratively improve the performance.

43. 5:47

    And, uh, one, one aspect that I wanted to talk to you about on, on the kernel development side of things-

44. 5:52

    Mm

45. 5:53

    ... um, where are you seeing open models, uh, wh- when it comes to kernel development? You recently released, um, a benchmark specifically for this. So I was wondering if you could talk on that a little bit as well.

46. 6:01

    Yeah. Yeah. It's a, it's a great question. So I think, uh, we're seeing all sorts of models. So the closed frontier models and the open models, um, get increasingly better at, at writing kernels.

47. 6:11

    So we use, uh, models all the time when we are developing kernels and, and, and writing the, the optimization frameworks. Um, I think the, the interesting thing that we are starting to look at is...

48. 6:23

    So this benchmark that, that we recently released called, um, ParallelKernelBench. So it actually has a bunch of, uh, unsolved problems in it. So we went around, surveyed all the different, uh, ways they can serve model inference.

49. 6:35

    Um, and one of the interesting things that we found is that there's a lot of things that we can think of that would actually speed models up that there don't exist good kernels for.

50. 6:43

    Um, so one of the reasons that we put that benchmark out was, you know, one thing that people worry about is, like, benchmarksing or, or overfitting to particular benchmarks.

51. 6:51

    Um, one of our intentions with this benchmark was if you overfit to it, that's great, 'cause we'll go take those kernels and use them to, to, to accelerate the, um, the, the, the inference and the development.

52. 7:01

    Yeah. This is a really interesting point. A lot of people have problems with benchmarksing. But the, the way I think about it is if researchers like you put all the really useful benchmarks out and we benchmark on all of them and everything is in distribution, then that's a perfect world, right?

53. 7:14

    Exactly, yeah.

54. 7:14

    That's a very useful model that we can then use. Um-

55. 7:18

    Yeah.

56. 7:18

    Okay. Cool. So I, I wanted to touch on, um, the inference side of things now as well.

57. 7:22

    Mm.

58. 7:22

    So, like, a new model drops like this, what does it take... C- could you take me, like, behind the scenes at the inference stack, and what does it take to go from day zero launch to then optimizing it week over week, month over month?

59. 7:34

    Yeah, yeah. Great question. So when we partner with someone like MiniMax, we will get some early model details. So, uh, for M3, for example, there are things like the MiniMax Sparse Attention, um, and, and some of those choices that were a little bit different from any model that's out there.

60. 7:48

    And I think if you look at any of the open models now, they're all quite different from each other in different ways. So there's different attention, different MoE choices, um, differences in quantization and, and all these pieces.

61. 7:59

    So as soon as we get those details, we start writing kernels, benchmarking, figuring out is there existing kernels that work for it, uh, do we need to modify something, do we need to write something from scratch.

62. 8:11

    Um, and then so day zero we're d- trying to think about things like quality. So, um, when this model launches, is it gonna have the quality that we all expect?

63. 8:20

    Uh, are we gonna be able to provide the right, um, the, the right user experience? Um, and then from there, uh, as soon as it launches that day zero, we have a long list of things that we know, hey, we have to do this with the KV cache, we have to do this with the attention kernels, we

64. 8:35

    have to do... Uh, we're gonna look at this part of the quantization, um, and things like that. So we, uh, we, we have that list and then we start working on it and start, uh, optimizing over the course of weeks so that when you use these models, they actually get faster, um, between day zero and day seven

65. 8:52

    and day 14 and, um, et cetera.

66. 8:55

    Yeah. I was just talking to Ingrid actually yesterday and I, and I asked her, "Have we been improving the performance of M3?" Um, and I meant over the last month and she said, "Oh, did you mean from last night?" [laughs]

67. 9:04

    And this is the pace at which these guys work, so, um, it's very real. Um, one aspect that I wanted to touch on with this is we're seeing the workloads shift.

68. 9:13

    We're going from kind of predominantly chat workloads where you have turns, uh, coming in now to agentic workloads where you've got this thing sitting inside a harness and you're doing hundreds and hundreds of multi-turn tool calls.

69. 9:26

    Yeah.

70. 9:26

    Does that change the way you build the inference stack?

71. 9:29

    Yeah, it definitely does. So these agentic turn-based workloads, uh, they go into everything from informing your KV cache, um, your prompting, your, um, pieces like that. So it, it, it informs what part of the stack you want to go optimize, um, because now I think when we're in the chat, chat, chat world you have a system prompt

72. 9:49

    of a few thousand, um, and then you just have the, the chat logs. Now with the coding-based agentic workflows, you upload your whole code base to the model, um, and that's a very different optimization and routing and kernel challenge, um, than, than just the, the chat-based workload.

73. 10:03

    So yeah, we're, we, we follow these workloads very closely. It's really interesting to see how they evolve and, and how to adapt the inference stack and the inference engines to, to really take- uh, to really serve them well.

74. 10:14

    Not only do you have agentic workloads, but you've also got multimodal workloads in there. So what I like to do often with these coding agents is, um, get them to optimize a, a web app and then get it to use it and then do a feedback loop.

75. 10:27

    So one thing that I wanted to come to you, uh, uh, Olive, for is, uh, MiniMax M3 is multimodal, M2.7, all, all the ones before that were not multimodal.

76. 10:36

    And, uh, like, can you talk a little bit about the optimizations and, uh, how you trained it for that aspect? And then also I wanna get into the architecture of it a- afterwards as well.

77. 10:44

    Right. Uh, definitely. So what's different from before was that it was trained multimodal from scratch.

78. 10:51

    Yeah.

79. 10:52

    So from step zero we train not only text data, we also train image data. And it was normal for many other labs that the model would collapse after training a little bit and we managed to solve that problem.

80. 11:05

    And, uh, what we find was actually that with this kind of training from scratch, if you look at the attention map, it actually the visual... Uh, the, the text tokens would attend to the visual tokens so that they're naturally combined together to naturally understand each other.

81. 11:20

    So for example, we are developing, for example, uh, we- websites, right?

82. 11:25

    Yeah.

83. 11:25

    Um, it is better if we train with m- uh, both multi- modalities. Also like for example it can look at the website, it can understand how it looks and then better optimize for it, like for example during reinforcement learning.

84. 11:38

    Um, so yeah, I think that is pretty cool.

85. 11:44

    One, one thing that kind of stuck out with this model for me was the fact that it introduced a lot of new things. The multimodality, the increase of context to one million-

86. 11:53

    Yes

87. 11:53

    ... um, the fact that you have a sparse attention now.

88. 11:56

    Right.

89. 11:56

    Um, so if you go to the inference side, it's almost a, a nightmare, isn't it? You get, you get this new model, and there's so many things that you could optimize to speed up inference.

90. 12:04

    Um, practically, what are the things that you focus on? There's like 1,000 things that you could optimize, but where do you get the most bang for your buck?

91. 12:11

    I mean, you focus on 1,001 things.

92. 12:14

    Yeah.

93. 12:14

    Like, you just, you just go, and you keep, you keep doing it. You find every edge that you can, um, and, and you go and, and you push on it.

94. 12:21

    Um, so yeah, I think there, there's, there's no stir-

95. 12:23

    Yeah

96. 12:23

    ... there's no stone that you leave unturned-

97. 12:25

    Uh-huh

98. 12:25

    ... and you just, uh, ke- keep going at it. If someone tells me you can't do the 1,001st thing-

99. 12:30

    Yeah

100. 12:30

     ... like, I don't know, try harder.

101. 12:32

     Yeah.

102. 12:33

     And then up a little later.

103. 12:35

     Oh, go ahead.

104. 12:37

     No, no, that's it.

105. 12:38

     Yeah. So the other thing that I wanted to ask is there, there's a whole kind of, um, zoo of open source models.

106. 12:43

     Mm.

107. 12:44

     Uh, as you're talking about speeding up inference, are there things that ... Are there lessons that you can take from one model and apply it to MiniMax M3? Um, or, or do you have to, like, restart from scratch as you're thinking about the inference engine, the kernels?

108. 12:57

     Like, how, how does that work?

109. 12:58

     Right. Um, yeah, so there, there's definitely things that you learn from optimizing one model that you take to another. Um, so I think sparse attentions are something that have become quite popular now.

110. 13:08

     So the MiniMax Sparse Attention is a little bit different from the, uh, from the DeepSeek and the, and, and, and those, and the, and the ones that you find in GLM.

111. 13:18

     Um, but the, there's still similar lessons that you can take from that optimization process and that kernel writing process that you can then bring to, to the new sparse attentions.

112. 13:27

     Um, and you know, we've been, uh, in, in some form anoth- and, or another, have been thinking about this problem for, for many years, um, so going all the way back to my PhD.

113. 13:37

     So it's, it's, it's great to see some validation that, that folks can now train it at scale, um, and people are using it and, and it's, uh, and it's, uh, it's going pretty well.

114. 13:46

     Yeah. Yeah, one of the interesting things, uh, especially about open source is you've got all these labs that are learning from each other-

115. 13:53

     Mm-hmm. Mm-hmm

116. 13:53

     ... kind of, uh, taking the wins from each other, right? So if, if, uh, if one lab figures out that MiniMax does this really well-

117. 13:59

     Yeah

118. 13:59

     ... then that gets, becomes the golden standard. Um, o- one of the things o- on model launch that I, uh, in the blog post that you guys go into is that this model was actually able to replicate a 12-hour run where it could reproduce an ICLR paper.

119. 14:14

     Um, and so somebody who's training this model to do this thing, how do you actually go about that? 'Cause that seems like a pretty ludicrous task.

120. 14:21

     Right. So letting the model to do cool stuff like replicating papers-

121. 14:26

     Yeah

122. 14:26

     ... uh, optimizing kernel frameworks and stuff like that is always exciting for us researchers because it's, like, very related to our job.

123. 14:34

     Yeah.

124. 14:35

     But training it can be very tricky because it's very long horizon-

125. 14:38

     Mm-hmm

126. 14:38

     ... and, like, the, the task itself would require GPUs. Um, it has hardware constraints. So it's very interesting to train tasks like that. And I would say the key there is still the environments and the data and how you formulate the problem, how you formulate the rewards, how you formulate the environment, and how you change the reinforcement

127. 15:02

     learning algorithm a little bit so that it's trained more efficiently, um, so that you can see cool things emerging through the, the iterations of RL runs.

128. 15:12

     Mm-hmm. Can you talk, maybe if I keep pulling on this thread a little bit, how do you do evaluation over these longer, longer scale runs? So if you're, if you want the thing to do a 12-hour task-

129. 15:22

     Mm-hmm

130. 15:22

     ... yes, it might or might not do it at the end, but are there, like, uh, spar- intermediate things that you can also look at?

131. 15:28

     Yes, we do. For these tasks, uh, there are, there are iterations, right?

132. 15:32

     Yeah.

133. 15:32

     So the model can submit several times, and we would evaluate each of them.

134. 15:37

     Okay.

135. 15:37

     Some of them, some of the times, the models would hack, and we do, uh, do, like, validation and test split for it to test if it's really improving on the performance or it's hacking.

136. 15:47

     Uh, and also we design our internal evaluations. So for example, for the release of 2 point, m 2.7, we touched a bit on self-evolution, right?

137. 15:58

     Mm.

138. 15:58

     So we're actively using the model to improve the speed of development internally, which, um, like, out of it, out of it, we can build our own evaluations that are closely related to our own work that we can evaluate the models on.

139. 16:11

     Yeah. When, so when we start talking about these long-horizon tasks that are 12 hours long, uh, we gave an entire workshop on this on Monday, but what I wanted to come to you, Dan, for is, um, KV cache.

140. 16:23

     So if let's say you-

141. 16:24

     Mm

142. 16:24

     ... you have concurrent requests that are 500 to a million, uh, uh, thousand context length long, how do you deal with the KV cache that just keeps on growing?

143. 16:32

     And how does the inference engine deal with that?

144. 16:33

     Yeah. So there, there's a lot of different pieces that, that you put there. Um, like in, in some sense it's like recreating a distributed file system. Um, so we were in some sense building something like that, uh, or a, or a very big database.

145. 16:47

     Um, it's, it's pretty simple in theory. It's like the type of thing that you should have done in your third year of undergrad or something like that. Um, but most of us actually skipped that class, so now we're rediscovering it, um, live in, in industry.

146. 16:59

     But it's, it's all about where do you store that cache? Uh, how do you know? Have you, have you seen this before? How do you fetch it? How do, how do you send it from one place to another?

147. 17:08

     Um, so yeah, it's, it's, uh, it's, it's not that complicated. Um, but, uh, you, you do have to make sure that, that you do a good job.

148. 17:16

     Yeah. One thing I noticed, you, you gave a, a, um, a lecture at Stanford recently, and one thing that stood out was that if you fast-forward like two, three years, two, three years is, is, is a long time in AI, and if you look back, you said that we'll realize that how, um, how early we are right

149. 17:31

     now.

150. 17:32

     Yeah.

151. 17:32

     So from your vantage point, we're three years out. What do you think we'll look back on and be like, "Why were we doing it this way?"

152. 17:38

     Great question. Um, uh, some things that I hope for. So I think we underutilize our GPUs a lot right now. Um, you know, SpaceX said they would have, like, 10% FLOP utilization or something like that.

153. 17:50

     I hope in three years ... Well, they, they should already be embarrassed by it, but I hope in three years they're extra embarrassed by it. Um, so certainly training should be, should be pretty, um, should be pretty good.

154. 18:01

     I think at inference we can do a lot better with the hardware that we're using, uh, that we have today. So I hope, uh, in a few years we'll, we'll say, uh, we'll, we'll have seen the light on, on some of those pieces.

155. 18:12

     Um, and I think there will be a lot more models, there'll be a lot better ... Um, I, I hope finally by then we've put to, put to bed this question about the open models.

156. 18:22

     You know, there's ev- every few months there's someone like, "Oh, Anthropic, OpenAI, they're so ahead," yada, yada. Um, but I think, uh, we're, we're seeing with models like M3 and GLM and Kimi and, and all those models that, um, the open source frontier really can catch up.

157. 18:36

     Um, and uh, and it's, it's not even that far behind. So, so I think that that's quite exciting.

158. 18:41

     Yeah. I, I wanted to throw that same question over to you, Olive. But you mentioned that you, uh, for, for the M2 series and the M3 series, you're using this idea of self-evolution where the model is building its own harness and then it's training inside of that, and then you get the next checkpoint.

159. 18:55

     Um, if you look back three years, uh, out and then you say, like what in RL or post-training, um, do you think, uh, made the biggest difference? What do you think that is from this vantage point?

160. 19:06

     Um, great question. But three years ago I was still in school. Um [laughs] [laughs]

161. 19:12

     I actually didn't start this industry yet.

162. 19:14

     Yeah.

163. 19:15

     So I wouldn't have imagined what's happening right now today, so it's really exciting. But I can see how, uh, models that were developed are, were already, um, improving the speed of development-

164. 19:25

     Yeah

165. 19:25

     ... maybe a year ago or even further than a year ago.

166. 19:28

     Yeah.

167. 19:28

     So I could see how the speed is actually accelerating, um, how the development is accelerating. And that's how, like, open-weight models can really catch up with frontier labs.

168. 19:41

     Yeah.

169. 19:41

     And yeah, it's, that's how we think we are more missions to bring this model to everyone, um, so that everyone can use it. Yeah.

170. 19:50

     Awesome. Thank you, guys. Thank you, Dan. Thank you, D- uh, thank you, Olive. Um, thank you guys so much. Have a great day.

171. 19:55

     Very cool. Thanks so much.

172. 19:56

     Thank you. [clapping] [outro music]