From Mixture of Experts to Mixture of Agents … with Super Fast Inference

Talk transcript

1. 0:00

   [upbeat music] This is, uh, the API key, so if you haven't scanned this already, please scan it.

2. 0:21

   Um, it'll take you to our cloud, where you can sign up for a free API key. This will be the only thing you kind of need to do the workshop today.

3. 0:30

   Everyone got a picture? Everyone good? Cool.

4. 0:35

   Okay. Um, hello, everyone. Uh, we are very excited to see you here. Today, we organized a very fun workshop for all of you. Uh, we're gonna first start with explanation of what Mixture of Experts is and what...

5. 0:47

   Like, why this architecture matters, and then we're gonna use agents to create something very similar to it, but it's called Mixture of Agents, so you replace your experts with real agents.

6. 0:57

   Um, there will be a code-along session, so everyone will build their own app at the end.

7. 1:04

   And, um, for the agenda, first of all, we're gonna introduce the concept, what is the Mixture of Experts, why this architecture exists, and what kind of models are using it.

8. 1:13

   Um, and, uh, this is basically, like, the way for us to continue improving the models that we build. Uh, so, like, ChatGPT, think about that models. How do we scale them further so they become smarter, more inta- intelligent?

9. 1:25

   And there are two ways to do it. So one is to pre-train from scratch with mixture of experts architecture. Another way is building a mixture of agents, so when you combine already pre-trained model together into one complex architecture.

10. 1:37

    So we're gonna build exactly approach number two in a hands-on workshop, and, uh, at the end, we will have Q&A, so if you guys are interested, um, about these models, what we do with these models at Cerebras, or you have any other questions, we can answer that.

11. 1:52

    Um, yeah, a little bit about me. So m- I'm a head research scientist at Cerebras. I worked there for almost 4.5 years. Um, I am specifically focused at researching mixture of experts architectures and, um, just other architectures that help us improve the LLMs and make the training more hardware efficient.

12. 2:11

    Um, in the past, I spent some time working on data scaling, so I created a data set called SlimPyjama. Um, that data set was the largest, uh, the, the best in quality data set when it was released.

13. 2:23

    Um, and yeah, prior to Cerebras, I was at Google working on research and engineering projects.

14. 2:29

    Perfect. Hi, my name is Daniel. I'm the head of growth here at Cerebras. I do both developer activations, developer marketing, but I also do startup sales. So if you need tokens and you're trying to run your startup, and you wanna use Cerebras in production, after the workshop, come talk to me.

15. 2:48

    I am the token, the token arbiter for Cerebras for everyone that is not enterprise. Um, and if you have a cool research project you wanna use Cerebras for, I am also the person to talk to, so if you want tokens for your project, talk to me.

16. 3:00

    I also really like hot pot. Um, I had hot pot three times in the last week. [laughs] Um, so yeah, that's, that's a little bit about me. I'm based in San Francisco, and I do a lot of these types of workshops and events and things like that.

17. 3:12

    Um, and here's my Twitter, uh, if you guys wanna follow me. Um, and this is our intern, Kevin. Uh, he's not here with us 'cause he's taking his last high school final.

18. 3:22

    He's in high school. But I wanted to put him here because he's the, uh, person that built 99% of the workshop that you're gonna be doing here today, and I wanted to make sure we shouted him out.

19. 3:31

    Wait, I wanna take a selfie with him.

20. 3:34

    Oh. [clapping]

21. 3:35

    I want to show him 'cause he doesn't know I'm doing this. Okay. [laughs] Um, and yeah, he's, uh, going to school in UCLA coming in the fall, but he's currently in high school.

22. 3:44

    So yeah, and that's his Twitter if you guys wanna follow him.

23. 3:47

    Great. Before we get started, who here has heard of Cerebras?

24. 3:52

    Okay.

25. 3:52

    Ooh.

26. 3:53

    Pretty good. Last time I asked this question, like eight months ago, the, there was, like, two hands in a room full of like this. So it's so much progress that we've made in the couple...

27. 4:00

    last couple months. So I will dive a little bit deeper into what Cerebras and why our hardware is so much superior than our competitors in the later part of our presentation.

28. 4:11

    But quickly, Cerebras is a hardware company that makes custom silicon that, uh, runs AI models super-duper fast. And here's a side by side of the chips, uh, comparing us to an, uh, NVIDIA H100, so there's just a sizable difference here.

29. 4:25

    And, um, things that make our hardware architecture super dominant kind of, like, all boils down to the innovations we've made in the hardware itself and how we're able to linearly scale with larger models, which I'll get into later.

30. 4:38

    But currently, we hold the world record in every single model we host publicly, um, and it's not even close. So for Llama 3.3-70B, we're s- around 15.5 times faster than the fastest inference provider on a GPU.

31. 4:52

    So if you want kind of, like, something that doesn't compare to anything that's currently in the market, Cerebras is kind of your only option for fast inference. So that's what our company does, and this is what I'm bringing to startups.

32. 5:01

    So if you're a startup that wants this inference, you should come talk to me after the talk.

33. 5:06

    So what are we doing today?

34. 5:07

    Yeah. So actually, spoiler alert, we're gonna use Cerebras, uh, hardware today for our, uh, workshop. So if you, you guys can, um, actually try it. But, uh, before, uh, I wanna explain what we're actually building.

35. 5:19

    So we're gonna build, um, an application with a mixture of agents. Um, each agent will be a separate LLM. Um, and kinda like right now, I want to explain why this is beneficial, why we wanna build that, um, and why this architecture is better compared to, like, a monolithic one LLM that we use right now.

36. 5:38

    Ah, thank you. Yeah, so, um, sort of like from like pre-training perspective, how do we make, uh, large models more intelligent, better? How do we scale them faster? So all of these type of questions we ask at Cerebras, uh, when we have our hardware, we can scale, uh, models pretty fast, but like how do we make them

37. 5:56

    more efficient? What kind of architectures we need to invest in? So, um, I kinda wanted to give sort of like the evolution that happened in the LAMP space, uh, for you guys.

38. 6:06

    So we started with the GPT-3 that was released a few years ago, and the model there, um, was quite small. Um, and basically what GPT-3 paper showed is that if you continue scaling the model size, you're gonna improve the performance.

39. 6:19

    Your models will have better skill sets. That's how you're gonna scale it. Um, the, the, the next thing that we saw in the LLM evolution is you actually have to spend a lot of time improving your data that you pre-train on.

40. 6:32

    So Llama model, uh, became bigger, but it also spent a lot of time on curating the dataset and scaling the, the, the number of, uh, tokens you train for as well.

41. 6:41

    And now, uh, you guys probably heard about DeepSeek V3 that was released a few months ago. That model took some additional innovations into place. So if you wanna go even larger, you see like GPT-3 is thr- 13 billion, Llama 3 is 400 billion, and DeepSeek V3 is 600 billion.

42. 6:56

    So if you wanna continue scaling the model size, which is what gives us better models, we need to come up with not just dataset improvements, but also architecture improvements.

43. 7:07

    So how do we actually, um, improve the models and thus like serve the large models? Because as you increase the number of parameters, you have to come up with a, like a way to scale it, uh, to scale your inference infrastructure and make it more efficient.

44. 7:21

    Uh, the answer here is Mixture of Experts. Um, and these type of models, like to just give you like an overview how it works, imagine that you have a transformer architecture, which is what we use as a backbone for large language models.

45. 7:35

    It has different types of layers. So here I highlighted ... Like, there are more layers there, but like one important layer, some important layers are embedding attention at feed-forward layers.

46. 7:45

    They all have different types of purposes in the network. Um, and now we're gonna see like how, uh, we change the standard transformer bottle- transformer architecture into something called Mixture of Experts.

47. 7:57

    So if you look at different layers and you do some interpretability work, you will figure out that a feed-forward network has a specific bottleneck. It has a challenge because feed-forward network sort of like has to disentangle all the information that, um, previous layers, uh, process, like attention layer.

48. 8:14

    So you can think about it this way. Um, feed-forward network has to decide which neuron in the network to activate when it sees a Golden Gate Bridge in a text.

49. 8:23

    So it's really hard because the, the, the task that we have for LLMs, sometimes they have different languages, sometimes, sometimes they require different specializations. It could be like math domain, biology, et cetera.

50. 8:34

    So feed-forward network has the hardest job in, in the whole. Um, so how does Mixture of Experts solve this bottleneck? Um, instead of having one monolithic feed-forward network, we will create separate feed-forward networks, and we will call them experts, as you can see on the right side of the screen.

51. 8:52

    Um, each expert will be specialized in a specific task. So you can think about this way. One expert can be solving math problems. Another expert can be a biology teacher, right?

52. 9:03

    And the other thing that is crucial here, this, this type of architecture allows us to increase parameters of the model. You create multiple copies of the feed-forward network, but you don't have to activate all of them for every token you route through the network.

53. 9:18

    You can see that there is an additional network called router within our network, which basically decides which expert to select for a particular token. So you can click next.

54. 9:29

    Yeah. And that allows us to actually increase the width of the model, increase the capacity of the model, uh, and scale the parameters because we know that from parameters you're getting better skills without increasing the inference time.

55. 9:42

    So you can actually activate same number of parameters as for the monolithic model, so you will mi- map like in terms of the time, but you will be better in quality because you trained a larger model.

56. 9:54

    Um, yeah, so, um, sort of like to close on, on this, um, this is, uh, the approach that other, uh, companies are using. Like, this is the industry standard right now.

57. 10:04

    OpenAI, GPT-4 models, Anthropic Claude, all of them are using this way, um, this approach to scale the models, um, and to gain better skills, uh, for, for their models.

58. 10:16

    So yeah, MoE models are really, really cool, and they are kind of like becoming the industry standard for being able to run large, large parameter models in a efficient way, so you don't have to continuously just add hardware to be able to run better and better quality models.

59. 10:30

    But there are some other approaches that also work. So something I wanna talk a little bit about is inference time compute. Um, last year, I believe, um, uh, Ilya Sutskever, uh, gave a talk at NeurIPS around how it i- we're in the age of inference time compute, where we have just thrown as much data as possible when

60. 10:47

    we pre-train these really, really large models, and eventually we're gonna hit a data wall, right? Where we have no more additional unique data to train our models with. So now what we can do is do more compute after the model has been trained to be able to get better and better results and more and more intelligent models.

61. 11:06

    So an example of, um, problems that benchmarks test for are math problems. So I wanna first take a, um, math problem from the AMI Math Competition, um, and see how certain, um, models kind of tackle this kind of problem.

62. 11:20

    And the thing with these types of math problems is that it's harder for, like, uh, a single non-reasoning model to be able to solve these, because they require multiple steps and sequential thought, where it's really hard to do things like this without reasoning.

63. 11:33

    And when I ran this exact problem through GPT-4o, which is not a reasoning model, it took 45 seconds to come to the wrong answer, and this is like the frontier model.

64. 11:43

    But, uh, ChatGPT o3, um, uh, sorry, GPT o3, which is a reasoning model, took 293 seconds to come up with the right answer. So this is it on the right Uh, on do- doing everything correctly, but it just took 293 seconds.

65. 11:57

    So if you want something in a reasonable, within three business days kind of timeline [laughing] This is, like ... This is probably not the solution for you. Like, imagine you're, like, do- scrolling through an app, and it just l- is loading for six min- or three minutes or, uh, uh, four minutes and 53 seconds.

66. 12:11

    That's just, like, an unreasonable amount of time for a lot of, like, tasks.

67. 12:17

    So I wanna introduce something called Mixture of Agents, which is leveraging the collective intelligence of multiple LLMs to come to the right answer. And the ... I, I think, like, because Cerebras is a super-fast inference provider, I'm sure everyone can see where this is going.

68. 12:29

    So a Mixture of Agents is basically the ability to take advantage of these earth-shattering speeds from our hardware and apply them into harder problems like this. So it's not just about speed, it's about getting higher intelligence with less smart models.

69. 12:44

    And Mixture of Agents is... A lot of it is inspired by Mixture of E-Experts architecture. Because essentially you're trying to do the same thing. You're trying to squeeze out as much intelligence in an efficient way from a lot of, like, a, a lot of, like, kind of, uh, tokens, whether it's within the model or outside of the

70. 13:00

    model. So the... Basically how it works is that you send inputs to multiple LLMs with custom system prompts, like agents, and then each model gives its own response, and then basically a final model combines all of the answers from all the individual models into a single answer, and this has shown that it outperforms even frontier models on

71. 13:19

    certain, uh, benchmarks, um, a- as benchmarked by, uh, Together AI, which, who's the ones that kind of came up with this idea and this term.

72. 13:28

    Um, so I wanna show you an example of a startup that's actually building in production with Cerebras in this Mixture of Agents model.

73. 13:39

    There's a YouTube video.

74. 13:40

    Should we read it?

75. 13:40

    Let's, let's see. Worst case, I can just show the Google Drive link.

76. 13:45

    Sorry, the internet, as you all know, is not great. So I hope this loads. If not... Wait, I think I have it locally saved. One second. Uh.

77. 13:56

    Oh yeah, I do have it locally saved. Perfect. So this is, uh, NinjaTech, uh, .ai, and you can try this product in production right now, and they're basically building a smarter, uh, like, chat, uh, chatbot, and this is NinjaTech solving the same exact question in 7.4 seconds and getting the answer correct.

78. 14:16

    So people are using this type of technique and our inference together in production to solve really hard questions like this math problem. Cool.

79. 14:31

    Okay, hopefully. Please, please, tech gods. [laughing] [laughing] Okay, cool. So we, uh, basically with, uh... What this startup did was take, um, a bunch of models and a bunch of LLM calls and get the right answer that a frontier model, reasoning model, took 293 seconds all the way down to seven seconds.

80. 14:54

    And here's how their whole application works. So they have a planning agent that spits out eight potential proposals for the right answer,

81. 15:04

    and then another token, a critique agent, comes in and be like, "Hey, are these, any of them feasible answers to this particular problem that I'm trying to solve?" In this case, all of them were bad.

82. 15:14

    So then what happens is that the planning agent goes back to the drawing board and then spits out 16K context worth of thinking tokens of eight answer proposals, and then the same virtuous cycle happens where the critique agent's like, "Are any of them good?"

83. 15:27

    In this case, uh, in the example I showed before, two of them were potential answer candidates that could've been the correct answer, and then another agent, a summarization agent, takes those two top answers and then turns them in to the final answer, which is eventually the right answer.

84. 15:42

    So this whole process, even though it took seven seconds, took over 500,000 tokens to be generated and 32 LLM calls, some of them in parallel, some of them sequential.

85. 15:52

    So this type of system allows you to take advantage of non-frontier models, even open source models that may not perform as well on benchmarks, and turn them into performing better than frontier models.

86. 16:06

    So w- yeah, like I said, again, why don't people use o3 in production? It's because it's very slow. Like, what... I, I can't even think about, like, use cases where you can wait five minutes to...

87. 16:17

    if come up with an answer, unless it's, like, very asynchronous.

88. 16:21

    And Cerebras solves the speed bottleneck. Um, and obviously Cerebras is the leader in fast inference, so I wanna show you why we are so fast. So this is an architecture diagram of a GPU, and highlighted in red is the core, or the thing that does all the mathematical computations that allow LLMs to predict the next token.

89. 16:40

    In this particular GPU, which is the H100, there are around 17,000 cores on this chip.

90. 16:47

    The problem is that the memory where all the weights and all the, uh, intermediate calculations and all the other information needed to produce the next token is stored largely off the chip, and these memory channels that communicate between the cores and the o- external memory become bottlenecks as you run larger and larger models, because of course you

91. 17:07

    have to transfer in more weights, and you have to transfer in more intermediate calculations in the KV cache while you're trying to calculate the next token.

92. 17:16

    Cerebras tackles this by having a radically different memory management system. We have 900,000 individual cores on one chip, and then with those 900,000 cores, we have 900,000 individual memory stores that are distributed all across the chip, one to one with our compute cores, and each core has direct access to memory.

93. 17:38

    The core-specific memory holds the same set of weights regardless of what you're putting into the system. So basically you don't need to wait for the external weights or the intermediate calculations to load to be able to do the computations.

94. 17:51

    We can just do it in real time, because there's no kind of memory transfer time that you need to wait for. Everything is just on the chip.

95. 17:59

    And Cerebras scales linearly across larger models. The thing that makes Cerebras really fast with super large models is that the only piece of data that's being transferred from chip to chip is activations.

96. 18:11

    That's the only piece of data that travels. It can even be, like, transferred using a single ethernet cord, that amount of data. It's very small. With a DGX cluster with multiple GPUs, they have to transfer so many activations and cache computations in between layers because single GPUs cannot do multiple layers of computations via hundreds of NVLinks connectors,

97. 18:32

    switches, et cetera. And that networking piece is the reason why we're so dominant compared to NVIDIA GPUs.

98. 18:40

    So Daria, how does this all translate?

99. 18:43

    Sure, yeah. Um, I probably wanna start with, like, a problem that I have and see how many people have the same problem. So when I interact with a monolithic model, with just one model, I usually ask to solve a particular problem, and then it doesn't get the right solution right away usually if the problem is complex, right?

100. 19:01

     So I have to continue prompting it and refining it, and at some point hitting the number of tokens that the model can process or something like that, and start over the chat from scratch.

101. 19:11

     Does anyone else have the same problem? Okay, cool, so I'm not alone. [laughs] Here, uh, what we do, uh, with mixture of agents, we're gonna specialize each agent to solve a particular portion of the problem.

102. 19:26

     So imagine you have a very complex problem, um, I don't know, you need to do a surgery, let's say, right? Imagine that. So we need different types of people to help with the surgery, and we're gonna ask each expert to specialize in one specific part of the surgery so they all together can work and produce, like, a

103. 19:41

     better result compared to just one person who can do a surgery, right? So we're gonna do it through prompt engineering. Um, and the, the one nice thing about it is instead of doing multiple rounds, multiple iterations to find the best solution at the end, you're gonna find a solution at the zero shot, so you...

104. 19:59

     At the one shot. You're gonna ask one question, and because each kinda, like, agent is already specialized in solving a particular portion of the task, they will, they will combine the result together and it's gonna be the final solution without continuous prompting.

105. 20:13

     So this is what we're gonna build today.

106. 20:15

     So this is the time for the hands-on workshop. So before we move on, this is, like, the second time I'm showing this. [laughs] Everyone ha- needs an API key. This is, like, the one thing you need to do for the workshop.

107. 20:25

     So everyone, get an API key. Okay, cool. And then you... Uh, please, uh, go to this GitHub link, um, and then star and fork the repo. That's, um, the next step.

108. 20:36

     Kevin created it. Kevin.

109. 20:38

     Yeah.

110. 20:38

     Not Daniel, Kevin. [laughs]

111. 20:39

     But I don't have, uh, HD shots of Kevin going like this. [laughs] So, uh, that's me.

112. 20:45

     Yeah.

113. 20:50

     This is also in our Slack channel if you don't want a QR code scan. So if you are in the AI engineer Slack channel, feel free to go to, um-

114. 20:58

     Yeah, we also dropped the slides there, and, uh, you guys-

115. 21:01

     Yeah

116. 21:01

     ... can interact there if you want, help each other out.

117. 21:04

     It's called MO- wait, MOA-

118. 21:06

     MoE Workshop

119. 21:07

     ... dash Workshop

120. 21:07

     Does anyone, uh-

121. 21:08

     Yeah

122. 21:08

     ... have issues finding that, let us know.

123. 21:11

     Yeah. And once you guys, uh, star and fork the repo, um, we're gonna deploy this app via Streamlit, or you can run it locally, whatever you prefer. Um, I just don't wanna deal with Python installation issues, so that's why we're...

124. 21:25

     I- I suggest if you have the internet bandwidth to go through Streamlit, but if not, doing it locally via, um, Python is fine.

125. 21:33

     And just quickly, uh, Daniel was going through the slides, but if you wanna come back to some of the steps, we shared it in the Slack channel. So you can open slides 50 to 54.

126. 21:42

     They will have the same instructions.

127. 21:44

     What's the Slack channel?

128. 21:45

     Um, MoE-

129. 21:46

     It's-

130. 21:46

     ... Mixture of Agents, uh-

131. 21:49

     Dash

132. 21:50

     ... dash, uh, Workshop

133. 21:53

     Yeah, so once you star and fork the repo, uh, you basically are gonna deploy it via Streamlit, um, or locally. Um, and basically how you deploy it on Streamlit is you go to streamlit.io.

134. 22:05

     I would suggest logging in with GitHub. That way everything is set up for you. Um,

135. 22:10

     and then deploy the cerebras-moa-workshop that you have forked into your GitHub repo, changing the main file path, and then clicking advanced settings. And once you click advanced settings, um, you have...

136. 22:23

     Uh, you just plug in your Cerebras API key here, and that's how you run the app.

137. 22:28

     So I'll, uh, wait, like, uh, like two, let's say three minutes, uh, for everyone to go and clone and, um, spin up their, um, app, and then we can go from there.

138. 22:41

     Uh, raise your hand if you need help from me or Daria. We can help you get set up if you have any questions.

139. 22:47

     Okay, let me set a timer.

140. 22:49

     If you guys wanna have an access to the presentation, here is the QR code to our Slack channel called MoE Workshop, and if you want steps on how to set up the, uh, everything with the Streamlit, you can go to the slide 50 to 54.

141. 23:04

     I already shared it in a chat there.

142. 23:09

     Yeah?

143. 23:10

     I think Kevin committed the API key. Can we just use that one?

144. 23:15

     Kevin committed the API key. Goddammit. [laughs] [laughs]

145. 23:20

     Thank you so much for [laughs] for bringing that up. [laughs] [laughs]

146. 23:27

     It's okay. His API key is rate limited. It's not like he has, like, the god API key. [laughs] But it's okay. He's an intern, right? Like [laughs]

147. 23:35

     yeah, feel free to, like, use your own though, please. [laughs] That'd be great. [laughs] 'Cause that will get rate limited. If everyone uses it, it's gonna, like, obviously get rate limited.

148. 23:47

     Daniel, I think-

149. 23:47

     I'm glad that my boss is not here.

150. 23:49

     Yeah. [laughs] [laughs]

151. 23:50

     So the secret just stays between you, us, and the internet, you know? 'Cause apparently this getting in-

152. 23:56

     The door is being record... It's being recorded. [laughs]

153. 23:58

     No, I know, I know. [laughs] I'm gonna rotate it right now. [laughs]

154. 24:05

     Perfect.

155. 24:08

     Daniel, I think we are getting the prize for the funnest workshop ever, right?

156. 24:12

     Wait, really?

157. 24:12

     Yeah, probably, right? [laughs] API key is committed. [laughs]

158. 24:15

     I kind of want a win though. [laughs] So like, um, if everyone wants to vote for us as, like, the most popular/funnest workshop, you know. I wouldn't say no, you know. [laughs]

159. 24:26

     I said how many secrets are there for you to actually-

160. 24:28

     Oh, it's a bribery thing. Okay, I see. I see. [laughs] There are strings attached to this vote. I see. I see. [laughs]

161. 24:36

     Yeah, let us know if you have any trouble spinning up the workshop, and then using definitely your own API key. Um. [laughs]

162. 24:48

     Okay.

163. 24:50

     Do we have a slide how the, um, workshop page-

164. 24:53

     I can use some help

165. 24:53

     ... front page should look like?

166. 24:54

     Yeah. Um-

167. 24:55

     I think I can use some help.

168. 24:57

     Wait, what?

169. 24:58

     Sorry?

170. 24:58

     I could use some help.

171. 24:59

     Oh, okay.

172. 24:59

     You need some help? Okay.

173. 25:02

     So I downloaded it. What do I do next?

174. 25:06

     Oh, no, you, you... It's in the Read Me. It's basically you have to install, like, all the requirements file, and then you run one command.

175. 25:15

     It's very simple. We made it... We tried to make it as simple as possible.

176. 25:18

     Thank you, timer.

177. 25:19

     So... Oh, yeah. When, when you have it all run- running, it should look like something like this. I would say very sexy UI, you know. [laughs] [laughs]

178. 25:28

     So, um, we wanted you to really experience what MoA systems look like, because I feel like this is a very new concept, and it's a great way... This kind of app that Kevin built is a great way for you to get started with h- what the possibilities are with MoA systems, maybe in production.

179. 25:44

     So, um, basically how this UI works is that in the top you configure your summarization agent, basically the thing that summarizes all the things and it actually, like, aggregates all the individual results from your individual agents into one.

180. 25:57

     Um, we have a very basic system prompt here. You can change it if you want. And then under Agent Management, we have individual agents that you can create with custom prompts, and you...

181. 26:09

     Here you can adjust things like the temperature, the model, as well as the specific prompt that you wanna put in the agent, and also rename it, you know, to something fun.

182. 26:19

     Um, and here you can delete agents, or you can create new ones. So you can have more than three, and you can also have different layers. You can have multiple, like, uh, iterations of how many times you want them to go through the solution.

183. 26:32

     And basically, the first part is not competitive. It's just gonna be us all having fun, right? So it's gonna... I, I can ask a question here like, "Plan a trip to San Francisco for me and my friend Daria

184. 26:46

     from, from 3:00 PM to 9:00 PM," and then see.

185. 26:53

     And then what it's doing is behind the scenes it's basically spawning all of the individual agents, and then going through multiple layers of calculations for it to return the final answer.

186. 27:03

     Here we have three layers with multiple LLM calls, so it might take a little bit of time.

187. 27:10

     It's just the Wi-Fi.

188. 27:11

     Oh, it's probably the Wi-Fi actually.

189. 27:13

     Yeah.

190. 27:13

     Um, can, can you zoom to me the-

191. 27:15

     Oh, yeah

192. 27:16

     ...

193. 27:19

     And if anyone doesn't see a screen like this, could you guys let me know? And I can come and help you with any steps.

194. 27:25

     Wow, the Wi-Fi is really bad. Does anyone not have the screen?

195. 27:41

     I have the other thing.

196. 27:42

     You don't?

197. 27:43

     Yeah. So, um, I'm studying the cloud [microphone feedback] and, um, in regards to setting the secrets. So if I want... Well, hold on. So that's where I'm stuck at in regards to

198. 27:53

     Okay. Maybe I can just, like, walk you through what-

199. 27:55

     Oh, yeah

200. 27:55

     ... what we see right now on screen.

201. 27:57

     Just delete this, and then just put cerebras_api-

202. 27:57

     Um, there are multiple layers that we created

203. 27:59

     ... key equal-

204. 28:00

     Each layer is basically a set of models, and they are connected together. Layers are sequential, but at each layer models are processed in parallel. So once we get the output from layer one, they will...

205. 28:10

     it will be combined together as an input for layer two. And then after the layer three there is, uh, this, like, uh, final model that we call summarization model that will process the final output combining the results from, uh, previous layers.

206. 28:24

     So all of these tokens were generated, sacrificed [laughs] [laughs]

207. 28:30

     ... for, for the final answer. So all of these tokens basically allow you to cover more surface area and have a more comprehensive answer.

208. 28:38

     So all of this was generated through all the other models.

209. 28:44

     Yes?

210. 28:45

     So each layer is an agent, right?

211. 28:49

     So each layer has a set of agents-

212. 28:51

     Okay

213. 28:51

     ... that operate together.

214. 28:53

     Okay.

215. 28:53

     Their output is finalized after that layer, so you can think about it's concatenated and used as an input to the next layer.

216. 29:02

     Okay.

217. 29:03

     Was everyone able to ask a very fun personal question to the MoA chat? Anyone struggling with that? Happy to help.

218. 29:15

     Okay, great. So now comes the actual competitive fun part of the co- uh, of the workshop. This is all f- fun and games until comes the competition. [laughs] So, uh, then go and select the AI configure...

219. 29:27

     uh, AI configuration challenge. And basically, your job is to come up with the perfect MoA system to generate... Okay, perfect code is a little extra. Maybe it's, like- [laughs] ...

220. 29:39

     really good code. Really good code.

221. 29:41

     We believe in you.

222. 29:42

     Um, so instead of writing code yourself, you'll become an AI prompt engineer and system architect, and your goal is to configure your AI agent, like mixture of agents system, to automatically generate code that scores the maximum 120 points in our automated grader.

223. 29:57

     And basically, uh, to participate you just have to go into configure AI and generate. So here you can select. So I can actually just do it with you for the first one.

224. 30:06

     So here it has some very awesome preset responses. So let me... So let's see. You can, like, set some awesome presets that are in turn created And then we can generate the code.

225. 30:25

     Hopefully the Wi-Fi is good. Okay, the code has been generated. So here, the code has been generated, and we can submit the solution. And here-

226. 30:33

     Baseline

227. 30:35

     ... the baseline is already designed with the prompt, the preset prompts, to be at, like, C, and your job is to be an A student because we're all overachievers in this room.

228. 30:43

     So basically, you can now change things like the prompt itself or the model. So let's, like, set it to the Qwen model and then see what happens. So we can generate the code again

229. 30:57

     and then submit the solution for grading. And here it's scored slightly better. So we went from a C to a B. [laughs] B. It's a solid B with literally just by changing the model, model, like, the, the model that it uses for the main agent.

230. 31:12

     So your job for the rest of this workshop is to figure out first what is the right combination of the main model, the number of cycles, like how many layers you want, the temperature, as well as the system prompt or the main prompt, and as well as the individual layers that will h- uh, that will spawn for

231. 31:31

     every single iteration. So here you can, uh, select different models, different particular prompts, and then use that to come up with the right answer. And I believe Daria and our intern both got perfect scores, so you're s- uh, shooting for perfect scores for your mixture of agent system.

232. 31:47

     Do we, do we have a function somewhere that we've given them?

233. 31:50

     Wait, uh-

234. 31:50

     Like, the actual Python function that has, uh, bugs in it?

235. 31:55

     Uh, no, that gets, um, inputted through here.

236. 31:56

     Okay.

237. 31:56

     So you can just, like, change this.

238. 31:58

     But, um, how do they... where do they find the function?

239. 32:02

     Oh, they don't. They kind of optimize it. Oh, it's right here.

240. 32:05

     Uh, no, no, no.

241. 32:07

     Default challenge.

242. 32:07

     The def- a default challenge. Where is the function itself, like the Python function?

243. 32:11

     Mm. No, it's the function. This is the function we're running.

244. 32:15

     Where is the implementation of the function that we give them as a baseline?

245. 32:18

     Oh, we don't. It, it, it... The, the, the... We change this so the challenge, like, generates the Python function from scratch.

246. 32:24

     Okay, okay. Gotcha. Okay. So this is important to say. Like, let's come back to that and set up the challenge-

247. 32:29

     Mm-hmm

248. 32:29

     ... right? 'Cause we have engineers-

249. 32:30

     Yeah

250. 32:30

     ... who know how to write Python. Um, so the challenge is here. We want to create a function that's called calculate user metrics, and the purpose is basically to calculate the metrics, right?

251. 32:41

     Uh, we have some details here what the function is supposed to, um, output-

252. 32:45

     Wait, um-

253. 32:45

     ... given an input

254. 32:45

     ... yeah, so this is the, the test that is used when you click grading. This is, like, the, the, the input to your AI agent system.

255. 32:55

     This is the input that will go in. And then basically, um-

256. 32:59

     Okay, I'm gonna send them the baseline function-

257. 33:02

     Okay, yeah

258. 33:02

     ... in the chat.

259. 33:03

     Feel free.

260. 33:03

     So imagine this, like, you have an interview, um, let's say a tech company, right? And at the interview they can ask you to optimize a function. For us, today, it's Python function.

261. 33:13

     This Python function has some bugs, and it's also not optimized. So what we want from you is using LLMs, basically find the solution that fixes all the bugs and optimizes to the level that you get the maximum score.

262. 33:25

     We created the grader that basically will use, um, like, LLM-generated function, um, as an input, and it will give you a score. And an idea is to get 120 out of 120.

263. 33:37

     Um, you can see we already created some specific agents to help you solve this task in one shot. One agent is working with bugs. Um, it basically tries to find all the bugs in edge cases.

264. 33:50

     One agent is working-

265. 33:51

     This one

266. 33:51

     ... with, um-

267. 33:51

     Performance

268. 33:52

     ... yeah, performance, so it's gonna optimize it. You don't, you don't, like, recompute some stuff. Um, and the last one is com- what is it?

269. 33:59

     Just, like, overall.

270. 34:00

     Oh, just overall.

271. 34:01

     Just, like, general.

272. 34:01

     Yeah. So, like, the final, the final, like, model, uh, the way it can work, you can ask it to look at the outputs from, like, three or whatever number of agents you created at previous layer and kinda create a final function using the inputs from different, um, agents.

273. 34:16

     And I'm gonna send you the, uh, Python function in the Slack channel. So if you guys see some errors yourself, like engineers, right? You can actually configure the prompts this way.

274. 34:26

     You can ask one agent to, like, "Oh, I see that there is an empty list that we try to, like, access by index, right? Could you please fix that?"

275. 34:33

     And that agent will work on that specific problem. Um, or you can also see what the default config outputs and, and figure out, like, what are the remaining issues in the Python function.

276. 34:44

     So it's sort of like vibe coding but with, with your brains, right? Um- [laughs]

277. 34:48

     Isn't that vibe coding? Don't you vibe code with your brain? [laughs] Well, I don't know.

278. 34:51

     Well, I've seen, I've seen multiple ways. [laughs]

279. 34:52

     Do you turn off your brain when you vibe code and just like you really let the computer take over, huh? [laughs]

280. 34:58

     Sign up

281. 35:01

     Oh, it's... This is should be default. Like, this is literally... I just... All I did was spun up the Streamlit app. Like, are you not seeing it?

282. 35:08

     Oh, no, you have to go to the... Okay, you have to... Sorry, can you scroll up here?

283. 35:13

     And there will be a prize for whoever gets the highest score, actually. Have I thought about what the prize will be? [laughs] Not yet. So I'll take suggestions from the winner.

284. 35:22

     What about the lowest score? [laughs] Great.

285. 35:27

     How about you get zero?

286. 35:28

     Sorry, I don't know how to respond to that. I'm, I'm, like, erroring out right now. [laughs]

287. 35:35

     Not the lowest score. That's so funny.

288. 35:37

     It's actually quite good. If they can get zero, it means they can get maximum two. [laughs]

289. 35:41

     Oh, really?

290. 35:42

     Yeah. If you get, like, low... zero, let us know.

291. 35:45

     Wait, if... To get zero, can't you just edit the thing to be like, just return nothing? Wouldn't that, wouldn't that be zero?

292. 35:52

     Or be like write a-

293. 35:53

     Ooh

294. 35:53

     ... write a Rust function-

295. 35:54

     Try it. Try it

296. 35:54

     ... and then the grader will be like, [laughs] "It's Rust." [laughs]

297. 35:59

     Don't give them ideas.

298. 36:00

     Oh, okay. Okay, sorry. [laughs]

299. 36:08

     But yeah, if someone is really interested in getting maximum score and you need hints, try it yourself though, but, like, if you need hints, we're happy to give you some hints.

300. 36:16

     Oh, yeah, we can do hints, I guess. You know? [laughs]

301. 36:24

     ... hits the max score or zero, please let us know

302. 36:27

     Z- not the zero, Daria. Not the zero

303. 36:29

     No, I wanna see the first people who get zero [laughs]

304. 36:31

     Not the zero, Daria. Not the zero [laughs]

305. 36:38

     What is this current, uh, score to get [inaudible]

306. 36:41

     Sorry, what'd you say?

307. 36:44

     Like, is the top score, what is the existing top score?

308. 36:47

     But you- you put the correct spectrum

309. 36:48

     No, it's the perfect score. There was, like, three people that got it

310. 36:51

     No, that's my invitation

311. 36:52

     It's on.

312. 36:53

     Okay, everyone. Okay, can you raise your hand if you got the 120 perfect score? [applause]

313. 36:59

     Okay, everyone, congratulations. Wow, we got hella winners. Okay.

314. 37:04

     Um, a- after the Q&A, please come up to us so I can get your email, and then I can get you something fun. I don't know what it will be, so please, when you come up to me, come with suggestions of what you want.

315. 37:14

     But, like, let's keep it realistic. No Teslas, no, no, like, no large gifts like that, okay? Like, uh, so something fun.

316. 37:23

     How about a date with you?

317. 37:25

     Huh?

318. 37:25

     How about a date with you?

319. 37:26

     Oh. [laughs] Uh, I don't know about that one. Okay. Um, before we ke- you all keep going, 'cause I feel like a lot of people got the challenge, so great job, you guys.

320. 37:37

     Um, I want to do a quick Q&A with me and Daria. So if you don't mind, if you can come up with one of those if you have questions for us about anything Cerebras-related or Mixture of Agents or Mixture of Experts, come ask us questions right now, please.

321. 37:53

     Wait, can you come up to the mic? I'm so sorry for you to do this, but apparently, like, that's how we get content.

322. 38:00

     Um, do you guys got, like, something to approach the microphones with? [laughs]

323. 38:04

     Can you, can you come to the mic? I'm so sorry. You have to do cardio.

324. 38:08

     How do you-

325. 38:09

     Yeah, if you gu- guys can, like-

326. 38:11

     Okay

327. 38:11

     ... hit the mic.

328. 38:12

     How do you, how do you AutoML this stuff?

329. 38:15

     AutoML this stuff?

330. 38:16

     Yeah. Okay, so I mean, it's all fun and all to go manually and figure out the prompt. What if the prompt's slightly harder, where it would take you 100 hours or 1,000 hours to figure out the prompts and you said, "I wanna throw a solution to solve this"?

331. 38:34

     That's my question.

332. 38:36

     I mean, I feel like in the application-

333. 38:37

     Which is gonna happen, right? So I don't think that's

334. 38:38

     No, I mean, it's already happening. If you look at, like-

335. 38:41

     Okay

336. 38:41

     ... um, like, um, Devin, for example, right? Like, the code gen startup. You ask it to do something and, like, a couple hours later it comes back to you with, like, a proposed solution, right?

337. 38:49

     And it tackling beyond just, like, fix a snippet of code. It's just building whole new systems or whole new applications. So that kind of thing is already happening. It's all about, like, how can Cerebras as a company that builds custom hardware, how can we enable those people so instead of taking hours, it takes minutes?

338. 39:05

     So it's ... I don't think it's the technology that's not there yet. It's about, like, how do we make it usable in the current, like, landscape that doesn't take s- that's not so painful?

339. 39:16

     So I think it's, like, actually it's already happening. Like, to- to- to u- apply AI towards these really hard multi-hour problems. This is kind of a small scale simulation of what you could do with a lot faster inference to speed that up.

340. 39:30

     What regions are your, uh, is your hardware currently running in? And do you have plans for, uh, global distribution?

341. 39:37

     Yeah. So right now we opened, I think, uh, six data centers in the US in the last year, and we are opening one in France this year, and then we, uh, currently have plans for one in Canada as well.

342. 39:49

     And we only expect that to go more global as the time goes on.

343. 39:54

     And, uh, one more. How- how long does it take you to onboard a new model when a- a new model's released or a new version of a model?

344. 40:00

     That's a great question. I think it depends on the model. So the- the blocker for us to onboard a new model is if we have all the kernels that are written to make sure that it supports the new model.

345. 40:10

     And in some cases, like Qwen 32B, that you all have access to, that took very little time because that architecture was very similar to Llama architecture in terms of the kernels needed to run it, and we had all the available kernels.

346. 40:23

     So all it took was all a bunch of QA-ing and, like, implementing API-level features to get that to work on our system. On the other hand, there are other models, um, that are extremely hard because we don't have the right kernels ready yet to support that model at, uh, in an efficient way.

347. 40:38

     So that takes more time because the kernel engineering team needs to write the custom kernels needed for that model. So it really depends on the model architecture.

348. 40:46

     Thanks.

349. 40:48

     Um, got a question about, um, power consumption. So

350. 40:53

     this sort of new t- architecture's gonna take shorter amount of time, but what about power consumption? Like, are we talking about similar to what NVIDIA's doing, or is it more, is it less?

351. 41:06

     Yeah, that's a great question, and I think it's not a one-to-one, right? Because we have a just c- completely different chip architecture, so it really depends on the workload.

352. 41:14

     But, uh, we've observed, and this is what we put in our website, that it's around a third of the power consumption of NVIDIA GPUs for the equivalent workload. It's just that our chips are a lot more massive, so it has a lot more throughput and a lot more, like ...

353. 41:26

     It can just take in a lot and generate more. Um, and- and it takes a significant number of chips, uh, NVIDIA chips, to match, like, what one system can do.

354. 41:35

     So it's not an apples to apples, but, like, I think, uh, we are a lot more power efficient in most use cases.

355. 41:43

     That's a great question.

356. 41:45

     It is. I do get that question also, so it's a very commonly asked question, I think. Everyone's like, "How does this big chip take in energy?" Okay. What's up?

357. 41:54

     Uh, my question is about a mixture of, uh, agents when I compare with, um, I don't know, like, um, the SOTA models, you know? Like, um, benchmarks-wise, where I sh- I should put it.

358. 42:08

     I think it's, like, about the configuration of the MoE agent, and I think that's where things get a little bit tricky. It's like- If you have really shitty prompts for your MoA system, it's gonna perform shittily, right?

359. 42:19

     So it's about like a tu- it's like tuning all of the prompts in your MoA system, and then it will perform better in the benchmark. So you actually do need to put in...

360. 42:27

     It's not like an out of the box thing, right? Because it's not training. It's like-

361. 42:30

     Mm

362. 42:30

     ... optimizing the system for your use case. So you need to make sure that you put in the engineering work needed to optimize all of the whole system, whether it's like using the right models or is it writing the like...

363. 42:42

     All of the combination of the things needs to work out for it to be better. The whole point though is that it can be better. It's just that you have to like actually like engineer it to be better.

364. 42:51

     Well, I think like from the theory perspective, if you have like... If you have an idea of like what's the ensemble learning case, where you create like multiple models that communicate between each other, and like together, ensemble basically provides a more robust, better solution than just one model.

365. 43:07

     So this is inspired by that, by like, you know, like decision trees or like-

366. 43:10

     Yeah

367. 43:10

     ... you know, when we created like standard ML models, we always get like more... less, less I guess like, um, memorized solution, uh, more generalized from multiple models. So in theory, even if you configure each model the same way and create the same prompt, the final answer can be better than just one model.

368. 43:29

     Yeah, yeah. Like, uh, XGBoost got to my mind when I saw the, that kind of approach, right? Like, you have multiple, uh, like a swarm of decisions and then you can get like, um...

369. 43:41

     I, I guess what I'm trying to get at is maybe there is a trade-offs when you have like too many, um, trees or agents.

370. 43:49

     Yeah.

371. 43:50

     And, uh, like, um, maybe like, uh, if the question that you need is too off, uh, it, it, it-

372. 43:58

     Can it like o- like can it like create worse solution if you create too many agents?

373. 44:04

     I, I guess if there is a danger of, uh, being more like a homogeneous answer-

374. 44:10

     Mm

375. 44:10

     ... if you have, uh, too many agents.

376. 44:13

     Mm.

377. 44:13

     Because maybe you have like a single agent that gets, gets this right, but you have like-

378. 44:19

     Right

379. 44:19

     ... 2,000 agents that get it wrong.

380. 44:20

     Yeah. This is a very good question. And so for like Mixture of Experts, when we create different experts, we also think about it this way, like how many agent, how many experts you wanna have, so then they're all kind of like used in a network.

381. 44:32

     So here, what's likely gonna happen, if you create too many agents, not all of them are gonna be used. So you will create like redundancy in the network, and you will just spend more time like getting the output from like ex- like agents here, and they're not gonna be used in the final solution, if that makes sense.

382. 44:48

     Yeah, yeah. I feel like, like feature importance in XGBoost, right?

383. 44:51

     Yes, yes, yes.

384. 44:53

     Okay, cool. I got it. Thank you.

385. 44:55

     Yeah. Hi. Uh, do you all support like bringing your own fine-tune model? Uh, like let's say if it's fine-tune on like Qwen 32 billion itself, uh, which you already support.

386. 45:03

     We do for enterprise clients. I know that's not the answer anyone wants to hear, but right now we are working on supporting LoRA fine-tune models. That's in the roadmap, but it's not currently supported.

387. 45:14

     But if you're an enterprise customer that's looking to onboard, uh, custom models, uh, we do have a number of customers, uh, running fine-tune models in our cloud.

388. 45:28

     Hello. Yes, a short question. Uh, did you already tried diffusion text generation models? Like they're 10 time faster than simple LLMs, and they may have super different architecture and how it, uh, fits your approaches and course.

389. 45:46

     I'm curious, are you working on diffusion models yourself?

390. 45:49

     Uh, no. Not now. We mostly trying to use it, onboard it from pragmatic perspective, and they already really fast and probably with, uh, with your approach it will be much faster, like 10 time.

391. 46:02

     Yeah. So from like diffusion models, um, they're definitely like one of the architecture to consider after Mixture of Experts and transformer-based architectures. Um, they feel like es- exploring different types of models.

392. 46:13

     There is state-space models, you know, all of them have different like improvements on top of the existing transformer decoder. I would say right now they're still in sort of like research.

393. 46:23

     Yeah.

394. 46:23

     So in our like inference API, uh, we kind of try to put models that are proven to be the best and they are like robust. For diffusion models, I think we still trying to scale, like from the research perspective, trying to figure out what's the best recipe to train them.

395. 46:38

     But, um, I, I know that this is a very interesting research direction that, uh, labs are looking at.

396. 46:44

     Thank you.

397. 46:44

     I did see an internal demo though that was insane around diffusion models. So I, uh, like diffusion models, uh, on Cerebras hardware is gonna be insane when it, when it comes out.

398. 46:53

     Yes. [laughs]

399. 46:55

     That's not helpful to anyone, but, you know, I thought- [laughs] ... I'd just throw that in there. [laughs] I can't even talk about it 'cause it's, the, the guy was like, "Don't talk about it now." [laughs]

400. 47:02

     But I'm like... But I'm being very vague about it, you know.

401. 47:05

     So I have a question related to what the previous gentleman asked about fine-tuning model. So say for example, I have a new architecture-

402. 47:14

     Mm-hmm

403. 47:15

     ... um, which, which has got certain different layers which are not there on, uh, not supported by Cerebras or does not have those kernels defined. What would happen for those kind of situation?

404. 47:25

     Yeah, that's a great question. I think, uh, one of our customers, Mistral, um, is a great example of this, where they brought in a custom architecture model, um, and they were like, "Hey, uh, we wanna run this on Cerebras."

405. 47:35

     And basically what it happens is like a partnership and a collaboration between their engineers and our engineers to make sure that all of the kernels are in place to support their new architecture.

406. 47:43

     So it's very simple. It's just like all, all our hardware is, if you think about it, is a bunch of memory and a bunch of compute that are organized in a very efficient way in a very v- like large surface area.

407. 47:54

     That's all it is. So we can technically, in theory, support like a very diverse set of models and architectures that may not even exist yet.

408. 48:01

     Okay.

409. 48:01

     So it's all about like creating that partnership and figuring out like how do we support the models that we don't yet support.

410. 48:07

     So that partnership is more about de- defining those kernels so that the architecture can take-

411. 48:10

     Um, there's al- also other quirks about models. Like if there is like custom like RL or something, you know, like it's all about like making sure everything is supported that you wanna run in your model.

412. 48:20

     Thank you.

413. 48:24

     Okay. Um, okay, last call for questions.

414. 48:32

     Yeah. Um, so I got here late, but do you think you'll ever do, uh, real time APIs and sort of those other multimodal models as well?

415. 48:42

     So we actually support, uh, we released our first multimodal API, uh, not available publicly, but through the Mistral app. So now if you use Mistral, uh, Mistral Large 2, uh, the, the Le Chat, um, par- ma- some of the image-based queries are running on hardware.

416. 48:57

     So, uh, next comes our cloud. So once we have it running in one place, we assume, uh, we're going to scale it into our public cloud. So that will be coming pretty soon actually to our Cerebras cloud for multimodal.

417. 49:08

     Around real time, we are actually thinking about this, so like I would love to learn more about your use case. Um, so yeah, real time is definitely a very interesting, I think, idea for the company because of the speed of our inference.

418. 49:21

     I think it'll be great in like real time use cases.

419. 49:25

     I think we have time for more questions.

420. 49:27

     Yeah, we do. We have like so much time.

421. 49:28

     It's like forty, forty minutes left.

422. 49:29

     Yeah. Maybe not all forty-five minutes though, but you know. [laughs] What's up?

423. 49:34

     So I'm curious, have, has there been, uh, models, um, engineered especially for Cerebras hardware? As in like, you know, you have new types of capabilities, and I'm curious, uh, uh, how good are people at exploiting those capabilities, right?

424. 49:45

     Because people are taking existing models, and yes, it's easy to, you know, port them if you have the kernels, but what about stuff that you really would, um, encourage people to try?

425. 49:56

     Wow, this is a very good question. Um, so it really depends on what your use case is. If you are a researcher who wants to try a new architecture and train it, I would say we have specific, um, advantages for unstructured sparsity algorithms.

426. 50:11

     So you can try that. Um, I don't think it's at the speed of any other hardware. Um, so like if you try that on GPU, it's gonna be hard.

427. 50:19

     Um, that's for pre-training side. For inference side, um-

428. 50:23

     I think for inference side, not yet, because we released inference like nine months ago. So it's a very, very new product, and as we're working with, uh, frontier model companies like Mistral, we're planning on have- working with them even more closely to design models specifically to take advantage of not only our current generation of chips, but our

429. 50:41

     future generations of chips. We'll have even more, uh, optimizations.

430. 50:45

     But maybe from the inference perspective, if you have a very large model you wanna serve, then Cerebras is the best position to do that. It's gonna be s- gonna scale, you know, multiple chips.

431. 50:55

     You're gonna use multiple chips, and you won't have to like distribute the model weights and wait, you know, uh, for all this like orch- orchestration to work together. Um, so I would say like the best use case here is like if you have a very large model, then use Cerebras inference.

432. 51:12

     So I have two questions. First is, you know, like what kind of model sizes can be hosted simultaneously on, on a Cerebras instance?

433. 51:21

     On one instance or like in a s- like what, what-

434. 51:23

     Well, like when I'm using it now, I don't know how many, uh,

435. 51:28

     VMs are currently-

436. 51:30

     So it's not a VM, it's like the number of systems, but basically there's no limit to the size of the system because what we can do is we can just infinitely add more chips.

437. 51:39

     Like, so like go, go... We can go from like as small as like an eight billion parameter model, and we can go all the way up to like, like Maverick is a recent model that we onboarded.

438. 51:49

     We're planning on supporting the bigger Meta models, you know. Um, there's no limit because we can just scale linearly. Our networking architecture actually very simple, just-

439. 51:58

     But there's a cost associated with it.

440. 51:59

     Of course.

441. 52:00

     Right. So, uh, and would... I wouldn't be able to partition a, a single, uh, SOC or, or one Cerebras instance into-

442. 52:10

     Oh, we can't. We don't offer that right now in our cloud. Basically-

443. 52:13

     You can't partition.

444. 52:13

     No. We basically handle it... We handle the load... It's an a... We currently only offer the service unless it's like an on-prem client with just an API. So, um, we give you rate limits that you can hit, and we in the back end provision the number of systems needed to match your, like, workload.

445. 52:29

     For the public API that y- everyone use today, that's a shared pool where we set rate limits for each user, and then they can consume until that rate limit.

446. 52:41

     Great question. Awesome. Um, if there are no more questions, um, uh, me and, um, Daria will be up here, uh, if you have more questions to come up to us.

447. 52:51

     Or if you're a startup looking for inference, I'm the guy to talk to. And if you also got one twenty, please bring you getting one twenty. Like get the proof, and then I'll get your email and give me ideas for prizes, and then we can go from there.

448. 53:05

     Thanks so much for coming, everyone.

449. 53:06

     Thank you. [clapping] [outro music]