State of the Union: Why Local, Why Now

Talk transcript

1. 0:00

   [upbeat music] Can you guys hear us?

2. 0:13

   Pretty good time.

3. 0:14

   Soundcheck. All right. Give it up for local AI, everyone. [clapping] Woo.

4. 0:23

   I hope you guys are excited as we are. Um, this is-

5. 0:26

   Woo.

6. 0:26

   Woo. This is the Local AI Summit. So we're gonna be here all day talking about local AI. And the reason why is we hit an inflection point this year.

7. 0:35

   Not only did the models get really good, but the harnesses got really good. And this happened really fast. It's been, I think, a struggle for anyone here to keep up.

8. 0:44

   I felt that most when I saw one of Andrej Karpathy's tweets. In November, he tweeted that you can't really trust these coding agents alone yet. You have to monitor them with an eye like a hawk.

9. 0:53

   Three months later, he tweets that he's struggling to keep up with the capabilities of how good this all has gotten. And the thing is, both times he was right.

10. 1:01

    This space is progressing really quickly, and it's wild how much you can do. And honestly, the only thing to do is just try to use it a little bit more today than you did yesterday.

11. 1:10

    That's how to keep up with this space, and that's exactly what you guys are doing right here. And so I'm really excited about this panel. The way that we use AI has also changed.

12. 1:18

    I'm not just using chatbots. I'm not just asking simple questions. When we got reasoning models, the, the profile of how the AI model responded changed. Not only is it bursty and responding to me, but before the burst, it kind of plateaus for a bit.

13. 1:33

    It's reasoning. It's churning on tokens that I'm not consuming.

14. 1:38

    And then we got agents, and suddenly, I don't even want these agents to turn off. I want these always-on agents. They can always be productive if we set them up.

15. 1:47

    And so we have enterprises that want to put a lot of their IP into this because it becomes more useful. We have consumers with the same thing. I want to give it my health data, my medical records.

16. 1:58

    I want to give it footage from my home camera. And both enterprises and consumers, we don't want that stuff to leak. And as you also have a profile of co- uh, tokens continuously generating, suddenly costs matter.

17. 2:11

    So local is amazing for both of those things. You get to make sure that you are plateaued on the costs, uh, for the tokens that you're generating. And also, uh, everything sits in that room.

18. 2:22

    So we have amazing demos here after these talks where the y- everything that's being run stays on those devices. It stays in this room. And so that's a really nice guarantee.

19. 2:31

    So as we turn over the panels, um, first, do you guys want to introduce yourselves?

20. 2:36

    Yeah. Should I start?

21. 2:37

    Please. You got the mic.

22. 2:38

    Oh, yeah, your mic'd up. [laughs] [laughs] Um, so yeah, I'm Alex. I'm the co-founder and CEO of EXO Labs and also the creator of Local.ai. So we're-- we've been working on local AI for over two years, which feels like a lot longer in this space.

23. 2:54

    And, you know, from the days of, you know, running Llama Four hundred and forty B on two MacBooks to now where we have, you know, demo there of Nemotron Ultra running on four Sparks.

24. 3:04

    You know, our mission is to make AI more accessible and, you know, it's crazy to see what is possible now. Um...

25. 3:12

    Cool. Hey, everybody, I'm Matt. Um, I make content, uh, videos. We have a newsletter all about artificial intelligence. I'm an AI enthusiast. Uh, and yeah, thanks for having me.

26. 3:25

    Hey, everyone. Um, my name is Ahmad Osman. Uh, I am the founder and CEO of Osmantic. Uh, somebody jumped into my DMs, asked, "Hey, what does OS Man stand for?"

27. 3:36

    Open Source Man, so that became a company. [laughs] And, uh, uh, I also moderate Local Llama, the subreddit. I have been in this local AI space since twenty twenty-two. And, uh, yeah, let's, uh, let's make open source and local AI one.

28. 3:50

    Love it.

29. 3:51

    Woo.

30. 3:51

    Woo. [clapping] [laughs] Open Source Man gets the round of applause. [laughing]

31. 3:58

    Woo.

32. 3:58

    That's, that's appropriate. [laughing] Today is, uh, I think, a pretty momentous day for the panel. Um, you know, Fable just came back, but it's a good reminder of why we need to have access to frontier intelligence to be able to build anything.

33. 4:10

    Uh, I'm Joseph. I'm co-founder and CEO at Roboflow. We do all things vision. I kinda like to joke that vision is like the original local AI. [laughs] 'Cause, like, everything needs to run probably concurrent with, like, low latency alongside your video, uh, where the images and data is being captured.

34. 4:25

    Uh, looking forward to the discussion today and all things why local AI is the future of making AI useful for everyone. Everyone, let's give Joseph and everyone a round of applause. [clapping]

35. 4:33

    Woo. So one thing that I'm really excited about for all the panelists is you guys were all early to the space in your own way, uh, and, uh, you know, very early.

36. 4:45

    And I think as we all feel this inflection point now, uh, when do you guys feel like you felt the inflection point?

37. 4:52

    Yeah, I, I can go first, sure. I mean, look, when I first saw Llama, right, I got really excited. The idea that I can actually download this intelligence, run it from my local computer, it was very exciting.

38. 5:03

    I'm a tinkerer at heart. I'm a builder at heart. I, I've overclocked PCs for years, so this just felt like so good. It felt so interesting to be able to have this crazy alien intelligence running on, on my device in my office.

39. 5:19

    Uh, and that, that was the first time where I didn't even think it was possible, to be honest, till I saw Llama, and I was like, "Oh, wow, this is incredible."

40. 5:27

    Um, so that was really the point at which I was very turned on to it.

41. 5:30

    I, I think I, um, it's kinda the same for me. Uh, yeah, it's Llama Two, I think, that, um, finally on my forty ninety, RTX forty ninety, I'm like, "Wow, I can actually understand this black box and customize it and play with parameters and, like, you know, sampling parameters and all those, uh, configurations and see how inference

42. 5:49

    engines work." Uh, that, that, that made me feel like, you know, something clicked. It's like, oh, magic. I can be a wizard now with this thing. I can control how it, how it plays to my ways of thinking.

43. 5:59

    Um- And from there it was, you know, the rest is history. Basically, I've been very vocal about why local and open-source AI must coexist with cloud and how, how we are supposed to like, you know, push for that and try to understand it as much as possible and teach people about it.

44. 6:15

    Totally. I actually felt this as well. I was on a plane, I think it was in 2023, and I had, uh, a model running on my phone, and it was awful.

45. 6:23

    It took twenty minutes to complete a sentence. But I was-- I didn't have internet, and so I was able to like ask, uh, essentially intelligence something there in the plane.

46. 6:31

    And, and that was my first feeling of like, "Hmm, this is gonna be something really cool."

47. 6:35

    Do you know that you can run the equivalent of GPT-4o on your iPhone now?

48. 6:39

    Really?

49. 6:39

    Yeah.

50. 6:40

    Amazing.

51. 6:40

    It's Qwen 3.5, the four parameter-- four billion parameters. Y- it, it's basically the same quality as something that used to be served in data centers, and it's in a device in your pocket.

52. 6:52

    Yeah, that's crazy. [laughs]

53. 6:54

    Yeah, I think if you just think about like how crazy that GPT-4o moment was, and now you can run it on the phone, it's like... I think-

54. 7:01

    Massive

55. 7:01

    ... a lot of this stuff was just, you know, you had to see the vision of like where things were going because it was a toy, right? It was like, you know, the, the first-- I think there were a few key points for me.

56. 7:11

    One was like running Llama 45B locally, right? That was like the first really big open source model. And I remember like the gap, it closed the gap quite significantly with the frontier closed models and open.

57. 7:24

    Uh, but it ran at two tokens a second. [laughs]

58. 7:27

    So it wasn't useful, right? Um, and then, you know, another big moment was DeepSeek, um, both V3 and R1. De- DeepSeek V3 was like a massive MoE. So like, you know, Llama 45B was dense, so super slow.

59. 7:39

    MoE seemed to unlock the performance. So it was like, "Oh wow," you know, with the devices I already have, like with a, you know, a Mac, a Mac Studio or Spark, I can run, you know, this massive model at actually like decent performance, which is comparable with, you know, what you can run in the cloud.

60. 7:57

    And then, uh, I think j- just recently, to me, GLM 5.2 is a big moment as well. Uh, 'cause again, it's closing the gap and it's like, you know, Opus level.

61. 8:06

    Um, and, you know, we have it running over there on a device that like literally can fit under your desk, uh, the DGX Station. So this is, to me, just like, you know, a, a trend and it's gonna keep increasing, right?

62. 8:20

    Like there's gonna be smaller and smaller devices with less memory, better compression. We'll be able to run more capable models locally and, you know, soon it'll be the default.

63. 8:32

    I'll keep with the, uh, the theme of, of airplane stories. [laughs] [laughs]

64. 8:36

    So I, I agree that there's been like multiple moments over, over time of what's been, what's been going on, but one personal-

65. 8:41

    I have a presentation for that after this, so stay around please. [laughs]

66. 8:44

    All right. Uh, one example moment where this like came up for me was, uh, I was on a plane. I was sitting next to someone, man, this is maybe two, three years ago, and they were hard of sight and they were using-- if you, on Apple, you know, if you take a photo of something, you can use

67. 8:58

    the accessibility settings, and it'll describe the photo to you.

68. 9:01

    Mm-hmm. Mm.

69. 9:01

    And we were seated on the plane together, and they were taking constant photos to understand where were we seated and how to get buckled in and these sorts of things.

70. 9:08

    And I remember they took a photo of the seat back in front of them, and the Apple accessibility described the photo, described it as like a printer or something. [laughs]

71. 9:16

    It was like, "Oh yeah, like you're seated like by a printer." And of course, the individual knew from the context that couldn't be right. Uh, and I was like, you know, I wonder.

72. 9:22

    LLaVA had just come out, a multimodal model- [laughs] ... that like kinda just described things with like the pivot to text base rather than like natively multimodal, but still kind of a useful example.

73. 9:31

    And I was curious, I was like, "Well, let's see how LLaVA would do on the same thing." So I took a photo of like the seat tray in front of me, and it aptly described it as like, "Hey, you're on an airplane.

74. 9:37

    That's a seat tray." And I showed the person sitting next to me, who had never seen local models and never experienced local AI, and what that stood out to me is I was like, man, a company that's a trillion-dollar-plus market cap business shipping the latest intelligence on their phones for describing visual settings is inferior to something that's

75. 9:54

    broadly accessible and available to anyone. And that was such like a watershed clear moment that like even the largest companies don't have a monopoly on the frontier of intelligence.

76. 10:03

    And so increasingly making that be accessible to others, um, is, I think, gonna be critical for un- understanding all the massive impacts that tech will have. And that was LLaVA. [laughs]

77. 10:15

    Yeah, it's interesting to hear you-

78. 10:16

    That's like, that's like twenty twenty-two, twenty twenty-three, right?

79. 10:18

    Around there.

80. 10:18

    Yeah.

81. 10:19

    Yeah.

82. 10:19

    Yeah, that was a while ago. It's interesting to hear you say about the, the accessibility of that, right? The, uh, taking this frontier intelligence, what made it useful was giving it peripherals, in this case, a camera, so that it could access the data in front of you.

83. 10:31

    And I think that's where the inflection point this year was so much more than just models, but also these harnesses and what you can give it access to. It was CLIs so that you can give business systems, uh, and plug those right into your agent.

84. 10:43

    It was, uh, you know, 4o was really exciting because I was coding with it, right? I was taking code snippets from my code base, and then I'd go to ChatGPT, and I'd paste it.

85. 10:51

    I'd give it as much context as I thought I should, and then I would take the result, and I'd go back into my code base and paste it. And what was really great about tools like Cursor was it essentially was this harness that said, "What if I just had the full file system?"

86. 11:03

    And I can let these agents reason on what files they needed and so on. And so the way that, you know, you get more use out of all of this local, uh, out of all of these local s- models and systems is how does it, how can it interact with the real world?

87. 11:17

    I'm curious, um, Joseph, so you guys got the start in vision, and it seems like vision had to learn the hard way what a lot of LLMs are dealing w- uh, discovering right now.

88. 11:28

    What do you think is like a big lesson that the language world is discovering right now that, uh, we could look to vision for?

89. 11:34

    I think, um, one of the things that language has an advantage of is it's inherently a human construct. So language generally exists where people exist, and that also means usually you can use, uh, infinite amounts of, of computer, what is available in a, in a data center.

90. 11:49

    Whereas a lot of vision, uh, you're compute constrained. You're running maybe where there's low internet connectivity, you're running on a device, you're running on a robot, and the amount of compute you have available to you is, is what, um-

91. 12:01

    And so what did that do? That created, I think, an emphasis on specialized learners faster.

92. 12:05

    Mm-hmm.

93. 12:06

    Because it's like, okay, I need this model to run in a limited context. Um, and I'm not gonna prioritize full world-scale generalizability. Instead, I'm gonna have specialized in-domain context work.

94. 12:18

    And what's interesting is I think you're actually seeing language follow a similar thing. You're describing it in the context of, like, harnesses for a given context. Um, but, you know, tools like, uh, any sort of coding agent where you have a really good harness and is specialized for doing those sorts of tasks.

95. 12:31

    But you're also seeing that even in like, you know, tax preparation or legal preparation of last-mile fine-tuning adaptation specialized learners. And so in some ways, I feel like the pendulum's swinging back to actually having specialized models, even in language just as much as vision, I think is one thing.

96. 12:47

    Um, and then, you know, there's a whole different thing around how you get the most out of compute optimization-

97. 12:52

    Yeah

98. 12:52

    ... when running on, on an edge device. Um, but I know that probably the Exo folks can talk, speak more to that.

99. 12:57

    Absolutely.

100. 12:59

     Um, but yeah, I think specialized models being... You remember there was like one model will rule them all world-

101. 13:04

     Yeah

102. 13:04

     ... was what everyone kind of thought, and I feel like the pendulum has swung back to people realizing specialized-

103. 13:07

     Specialized models. I think we definitely-- At NVIDIA, we see that it's gonna be a multi-model world.

104. 13:11

     Yeah.

105. 13:12

     We, we definitely agree with that. Um, there's actually one of the panels that's coming on later, I think at 3:20 PM, is all about model routing, and so that's, that's really exciting.

106. 13:19

     I guess as an enthusiast, uh, and as you use some of this, is that, uh, how your usage pattern looks? Are you using many models? What does that kinda look like?

107. 13:27

     Yeah, absolutely. Everything from obviously, like the top models, Fable, uh, uh, to kind of the more workhorse models with Sonnet, local models for things that maybe don't depend as much on low latency.

108. 13:40

     Um, y-yeah, like the multi-model world s-seems like a no-brainer at this point, uh, especially as you're seeing all of these enterprise companies come out and look at their budgets and, and think, "Okay, well, I, I need to continue to increase the, the total number of tokens that I'm consuming as a company, but I also don't wanna just

109. 13:59

     completely blow out my budget." I know Coinbase and Brian Armstrong just came out with that just great post the other day talking about, uh, how their tokens are, are exploding, yet their costs are staying flat and, and that is because they're using a mixture of different models.

110. 14:14

     You don't need the top model for every single use case, and in fact, most use cases you don't. Uh, I think the most obvious application is let the top model plan, uh, the, the, the architecture, whatever the kinda the top-level plan is, and then the actual execution of the code can go to, uh, a, a more reasonably

111. 14:30

     priced smaller model.

112. 14:31

     Your most intelligent should provide you with the overall plan and then sub-tasks for your smaller execution, like a executioner models, and that's exactly the future.

113. 14:43

     Yeah, uh, especially like, you know, we're talking about local. Um,

114. 14:47

     local models are great at writing code, but maybe we offload the actual top-level planning to one of the frontier models, and you, you save a bunch, you control more of the workflow.

115. 14:56

     It's, it's a, it's a nice pattern.

116. 14:59

     Yeah. I, I think the market wants this. So it wasn't clear like a few years ago when this stuff was just starting to play out, you know, would there just be one big model that everyone is using?

117. 15:10

     But clearly that is not what people want. That is not what enterprises want. They don't want to be told what they can do by Dario. They don't wanna be paying for the same model for all their workloads when some mo- some workloads don't actually need, you know, a giant, gigantic model that costs fifty dollars per million tokens.

118. 15:28

     Um, they want control, they want sovereignty, they want the ability to switch out models. They don't want to get rug pulled, you know, uh, from one day to the other, uh, because of some safety risk or whatever.

119. 15:41

     Um, and, you know, that's really what's driving a lot of this progress. So, you know, I'm more optimistic than ever, I think, about sort of, you know, where things are going with Local.

120. 15:51

     I think that, you know, the market is basically pulling a lot of this stuff, um, out of, you know, startups, out of enterprise are building solutions around this and, you know, it, it's, it's, yeah, it's, it's happening, right?

121. 16:03

     Totally. I feel like this is a new frontier where as we go multi-model, that becomes really difficult. Of, of course, you need to route between the models, but then you have to figure out how to provide necessary context to whichever model you're routing to.

122. 16:14

     If you are making a plan and then breaking it off into sub-agents, what is the framework to do so? How should you do this? These are all open problems and, you know, I think just being playful is, uh, kinda the best way to, to approach this.

123. 16:25

     But I like the way that you worded that of like, uh, the market's kind of pulling for these answers. It's reaching in this direction. We're looking for startups and companies to fill this gap.

124. 16:32

     Um, makes a ton of sense. You know what's funny? Um, as any, any enterprise consumes any s- any piece of software, it matters a lot. If you're gonna build a foundation, you wanna know what the versions are.

125. 16:43

     You wanna know what your model is, which-- whether that version changed in order to see changed behavior later. So having, uh, also more control over all of this so that you know exactly what version it is, it can't be changed, uh, n-not even because of any sort of, you know, re-regulation, but just simply if there are updates,

126. 16:59

     you choose when you're opting into whatever it is on your entire stack. Yeah. No, so [clears throat]

127. 17:06

     when you think about, um, this wave that we're seeing here of like this civilizational infrastructure that is called AI, you have to consider the, the potential of, you know, things being taken away from you and your sovereignty and, uh, how can you be in charge of, you know, this thing full stack end-to-end.

128. 17:31

     That's hardware, software, and everything in between. That's the model weights. That's the specific version of the model, as you were saying. That's how you fine-tune it. And picking up of, mm, Joseph's, you know, thing, um, [clears throat] statement here about small and specialized models, I have been v- like, you know, I have been a proponent of that for quite

129. 17:51

     some time. I have tweets about that in twenty twenty-four saying small and specialized models are the future. And it is gonna be per user, by u- per use case, per workflow.

130. 18:02

     For businesses, you know, to, to decide on their niche domains, and you need to start onboarding them now because you wanna collect the data points, the traces. That's what you're asking about, Nader.

131. 18:11

     You were asking about how do we get there? How do we decide which mo- model gets routed to? How do we decide which use case gets... goes to which model or which, which endpoint?

132. 18:21

     You need to collect data, and you need to be ready to basically break down, um, the, the traces that you've collected from your employees, from everybody in your organization, and decide how can we make the most optimal use case of these data points to which models, and collit- collect feedback as well.

133. 18:39

     And you can actually automate that with agents as well. So like, this is like something that it's re- you know, the, the whole thing about RSI right now, recursive self-improvement.

134. 18:48

     Mm-hmm.

135. 18:48

     It also applies to even agents and harnesses and to, to workflows and to use cases and to enterprises very well.

136. 18:55

     Do you wanna describe that for everyone too? Maybe for folks who don't know RSI.

137. 18:59

     Uh, recursive self-improvement. It means that, uh, a model would basically rent its own compute and then start training its own checkpoints and then, y- you know, uh, deploy, uh, its next version so that it, uh, gets updated in certain ways, change its behavior in certain ways.

138. 19:15

     So basically, a model is training itself.

139. 19:17

     Totally. For our product Brev, which just makes it really easy to get a GPU-

140. 19:20

     Mm-hmm

141. 19:20

     ... we've been focusing more on agents as a first-class audience because we're seeing this. So we're seeing growing usage from agents that just want to go grab a GPU, uh, directly.

142. 19:29

     Exactly.

143. 19:29

     So, yeah. You know what? It's funny. So the second thing you talked about was, uh, not just multi-model, but then all these optimizations to actually run it performant when you have less compute than in a, say, data center.

144. 19:39

     I think, Alex, this is some fun work that we did. So, um, Alex has actually set up, uh, a second headquarters inside of NVIDIA. [laughs] We got a conference room, and, uh, we were, we were having dinner, and he mentioned that he really was motivated to get to squeeze out every drop of performance we could on the DGX

145. 19:56

     Spark. And so we said, "Hey, let's get a conference room at NVIDIA. Come bring your team. And any time you need an expert across any pillar, let's just go and pull that person into the room."

146. 20:06

     And then, yeah.

147. 20:06

     I, I actually have a follow-up here. Um, so since I have been part of LocalLlama for quite some time, um, when I think about home labbers, I think about the b- individuals that are actually, uh, are within enterprises, on boardrooms, making decisions about, you know, the models that we're gonna host or w- you know, where we're gonna

148. 20:24

     get our AI from. Um, the things that these home labbers focus on are optimizations. It's basically how can I extract the most economical value out of the hardware, the constraints that I have, the software that I have.

149. 20:36

     You know, quantizations came to be because of that. When we're thinking about enterprises, it's the same thing. It's constraints. It's budget limitations.

150. 20:43

     Mm-hmm.

151. 20:43

     It's how can I make the most use of the number of tokens that I can get under the hardware that I have, whether that's a DGX station, you know, um, centrally in an, a small, middle-sized business or, you know, a DGX P300 cluster.

152. 20:58

     Yeah.

153. 20:58

     It's all about how can I optimize the software for the latest and greatest frontier open source model out there and get as much value as possible out of it under the economical constraints that we have.

154. 21:08

     100%.

155. 21:08

     I think it's, uh, it's crucial to see home labbers, and that's why, you know, this is local AI. It's local AI, but it can, it could be on premises.

156. 21:16

     It could be co-located hardware. It could be rented clusters. It could be from Brev-

157. 21:21

     Yeah

158. 21:21

     ... basically. It's just the idea about controlling this thing end to end, from hardware to software to model endpoints-

159. 21:27

     Totally

160. 21:27

     ... to model weights and, you know, collecting all of that data to train your specialized and smaller models that will be more efficient as, as you go. Um, yeah, the future is great, and the future is local. [laughs]

161. 21:39

     Yeah. Yeah, absolutely. Um, you know, making sure it's your dates, your, your weights, your compute. I think one thing that was really interesting, we did end up getting 10X performant improvements on the DGX Spark, and we sent an update to Jensen, to the executive staff, to the teams that we're helping.

162. 21:54

     And one line I really liked in that email was that we didn't solve any new computer science to do this. We actually took things that the experts at NVIDIA had already solved and was out there, and I think what we worked together to do really nicely was assemble it in a bouquet.

163. 22:08

     And I think that speaks to some of the, the usability here, right? We're talking about what is capable, where, what are the capabilities, but do you feel like it's just capabilities that are holding people back from adopting this?

164. 22:18

     Yeah. So just to tell the story a little bit what happened here. So I think we had dinner on Thursday of the week, and then, you know, an email got sent out on Friday.

165. 22:29

     So this idea came up, like, "Why don't we do a lab," right? Why don't we like... You know, this to me sounded crazy at the time. I didn't think this would be possible, but it was like, you know, let...

166. 22:37

     Like, Nader was like, "Oh, let's, like, get a bunch of people from NVIDIA to help you guys and work with you guys to improve the performance on the Spark."

167. 22:45

     Right? And I was like, "Okay," like let, yeah, let's... I mean, if we can, that'd be great. Email got sent out, I think, pretty much that night, and then on Friday, Nader told me, "Be here on Monday, um, at NVIDIA HQ.

168. 22:59

     Uh, we're gonna have, you know, teams of people here that are gonna work together with you on this." And, you know, turned up on Monday, and, uh, Jensen talks about this concept of swarming.

169. 23:11

     Uh, it's, you know, basically the idea that, uh, the, the whole company will, like, mobilize around something.

170. 23:18

     Have you ever seen little kids play soccer? There's a ball, and everyone just attacks it. [laughs] It's like a tackle.

171. 23:22

     Literally that.

172. 23:22

     It's just the mosh pit of the soccer ball.

173. 23:24

     Yeah. And I had heard about this idea, but, like, I'd never experienced it. And I can tell you, like, we were just back to back that whole day, people coming in and out of the room, you know, from all various teams, like the Nemotron team, um, you know, people working in, in, on data center stuff, people working,

174. 23:39

     um, on vLLM. I, I realized, you know, NVIDIA has a team for everything, so there's like a vLLM for Spark team, which is oddly specific. [laughs] But like, you know, they have all these teams, and, you know, we basically got to pull those resources in.

175. 23:54

     And, you know, you know, in those three weeks or so, as you mentioned, like, we basically got 10X performance versus, you know, what, um, NVIDIA had running on the Spark- Um, in their existing playbook, which was using Hermes Agent.

176. 24:09

     So we did a bunch of optimizations there, um, using VLLM, um, as the, the sort of inference backend there. Uh, doing a lot of work with the, like tuning the models, like quantizing the models, uh, to, you know, be fit for local.

177. 24:25

     So what you'll find is, I think one of the great things NVIDIA has done with the local hardware is it's the same architecture that's running in the data center as running on the Spark.

178. 24:36

     So it's Grace Blackwell. Um, so the hardware is like fundamentally the same, meaning you actually get a lot of things for free. So for example, like the kernels are already really good.

179. 24:45

     However, there is like a lot of tuning and a lot of like configuration that is right now designed specifically for the data center. So a lot of the work that we did was not like inventing anything new, but it was literally just tweaking things to work, uh, more performantly on the Spark.

180. 25:03

     And, you know, the hardware is extremely capable. It's like, you know, it is data center level hardware that literally can sit on your desk. Um, so it's just about like, you know, how do we activate that?

181. 25:13

     And I think this has been one of my learnings from the last few years, is like we already have the hardware. Like the hardware is already really good and, you know, the models are getting better.

182. 25:24

     Um, they're getting a lot better at compression, so you can fit more on a smaller device. So like, you know, I think it really is just a matter of more people looking at the space and working in it and, you know, we will be able to do amazing things.

183. 25:36

     Like we have, you know, Nemotron 3 Ultra. It's a 550 billion parameter model running on four Sparks over that, 30 tokens per second. So like, that's-

184. 25:45

     That's huge

185. 25:46

     ... pretty easy.

186. 25:46

     I want to say something here. Like it's, it's so great to see the community being proactive. I think of Alex Cheema right here with Exo as somebody who's a member of the community who's trying to push the frontier of open source intelligence to the next level.

187. 26:00

     I love how NVIDIA is working with them on that. When I think about, you know, this space, I think, uh, we're in the '90s of the Linux operating system-

188. 26:08

     Totally

189. 26:08

     ... and we are like just starting. The infrastructure is not there yet. We need so much more. Um, I was even telling Alex I want to see how we can collaborate.

190. 26:16

     We have like an open source, uh, uh, sys- deployment system called ODS that is a bunch of open source tools that we deploy in each hardware. It fig- configures stuff, configures agents, and basically, um, gets you set like, you know, get, gets you set up with the entire infrastructure end to end that you need for your agents

191. 26:33

     locally. This kind of stuff, we need that optimized for every piece of hardware out there so that we can get more and more people onboarded, have the open source adoption, you know, just go to the moon.

192. 26:47

     That's what I want right now. This is why we're here. We want everybody to know that local, local and open source AI can run on anything, starting from your phone to your DGX Spark, as you were saying, Alex, to DGX stations, to the next level-

193. 27:01

     Right

194. 27:01

     ... in the data centers, and it will deliver you frontier level intelligence.

195. 27:05

     Yeah, absolutely. I think this is a really good question for you, Matt. You know, you test everything for a mainstream audience. You're hearing about all of this technology, you're using this aggressively.

196. 27:14

     Where does it still fall short for the, uh, average user for AI?

197. 27:18

     So I, [clears throat] I think about two things, the average kind of personal user, then the average enterprise using, uh, open source. I mean, it, it, it, it needs to basically be as simple as opening Cursor.

198. 27:31

     Yep.

199. 27:31

     It needs to be maybe slightly more complicated than that, or slightly more complicated than just installing Codex and opening it up. Right now, it, to be fair, it is quite far from that.

200. 27:41

     It, like the, the stuff these guys are doing is incredible, but it is more sophisticated than what most people, including myself, are gonna be capable of. [laughs] Um, y- you know, let alone a, a, a business.

201. 27:54

     Uh, I, I think-

202. 27:54

     Need a full-time job to just do this kind of stuff.

203. 27:57

     Totally. Yeah.

204. 27:57

     That's why we need to automate it. [laughs]

205. 27:58

     And it, it really does need to be point and click.

206. 28:01

     Yeah.

207. 28:01

     And once it gets there... And there's, there's a lot of great open source projects, there's a lot of great projects in general that are getting there, but we're still not quite there.

208. 28:09

     Uh, the other thing to make it widely adopted is to allow people to better understand what use cases are appropriate for whit- what type of model, for what type of harness, what type of hardware.

209. 28:21

     Knowing exactly the use case that I can get out of my home system or something that I'm, I'm, you know, renting from a, a service center, uh, that, that is inc- incredibly important as well.

210. 28:34

     And I, I think it shouldn't be just in documentation. I think that's where open source becomes so difficult for people. I think it needs to be a point and click, and it figures it out on its own.

211. 28:43

     Yes.

212. 28:44

     That's what ODS is about. That's what I think Exo is about. That's what, you know, as we grow more and more and, and building this infrastructure, we need to be thinking about the user experience for your average everyday user, not us as technical folks here.

213. 28:57

     Because this is AI engineering, um, you know, summit, and I'm pretty sure that we all can manage our way around this stuff. But your ChatGPT users, your Claude users, whatever out there, they want this to be an alternative that is just click, play, send a message, use an agent, and done.

214. 29:15

     Yeah. Most, most people really don't wanna know about the details. [laughs] They, they just want it to work. And even if it worked seamlessly within Cursor or Codex or any of these other things, and it just worked and they didn't have to think about it, th- that is number one-

215. 29:28

     Yeah

216. 29:28

     ... uh, for, for the vast majority of people.

217. 29:30

     So interesting on that, in a multi-model world where we have to pick a model, do you think that that's something that is a product, um, as we're... or is that, is that plumbing?

218. 29:39

     Is that something-

219. 29:39

     Yeah. Nader, you talked about the harness.

220. 29:41

     Yeah.

221. 29:41

     It, it's exactly that. It should be some, like whatever the kind of the front entry point to somebody's AI experience is, that should be what is choosing w- which model at the right time for the right use case.

222. 29:52

     This is a very difficult problem. There are, uh, you know, open source projects, closed source, uh, companies. Building routing, that's only one piece of it.

223. 30:00

     Yeah.

224. 30:00

     Right? So, go ahead.

225. 30:02

     No, like, uh, Carter from NVIDIA, he's, he's gonna be a moderator here.

226. 30:05

     Hey. [laughs]

227. 30:05

     Hey, Carter. Uh, looking good, man. So, um, he, uh, I, I, I shared ODS with him, and he told me, "I love how, like, it immediately downloaded that two billion parameter models, allowed me to start playing with it, and then it started downloading the next model that would work perfectly on, on my device."

228. 30:20

     That's the kind of experience we need to be giving these users when they-- we're onboarding them. Don't make them sit down and have to think about all these quantizations and all these extensions and all these weird things.

229. 30:30

     That is too much work for your average user. That's how we lose them.

230. 30:34

     Mm-hmm. Yeah, it's interesting, right? Like, the-- When you're not having to deal with multimodal and you have just this generally good but big model, you can kind of ask it anything.

231. 30:43

     You don't have to be as sophisticated, or you don't have to understand your use case too well. But as we talk about specialized models, if you're gonna specialize in something, you need to know what that something is.

232. 30:52

     Yeah.

233. 30:52

     And so the amount that... I mean, can you guys speak to maybe what it's like to move into a specialized model or to build one? Or have you, have you-

234. 31:00

     I, I can start with that. Uh, on the cloud, uh, you're basically getting the normal distribution. So every time that they are training something, they are taking the average feedback from everybody when they are being happy or sad or like, you know, um, they are okay with the answer or, "Hey, no, this is not what I meant.

235. 31:15

     Go back or change the model, and answer with a different model." That's the kind of feedback that allows you, you know, cloud providers to fine-tune the next model. But if we're talking about small and specialized models for use cases, that's a lot of compute to train models.

236. 31:28

     That's a lot of, um, storage for these weights. Uh, that does not happen without each use case or each business entity, et cetera, focusing on their own, um, patterns and use cases and, like, how they handle agents and how their employees, uh, message these agents and what workflows they are interested in.

237. 31:47

     So it's really, you know, it's per c- use case, it's per entity. It, it's not something that we can just generalize, and that's why, you know, continual learning is not being talked about by the frontier model, uh, as much.

238. 31:59

     It's coming though. It will happen, and it needs to be running on local hardware for it to happen. That's how you can have something that is so optimized that is not just plotted markdown files sitting in one REPL and, you know, you think that that agent is not gonna lose track of which scale to update or what

239. 32:16

     to edit or which memories to update. Because at some point, context lens becomes inefficient, and that's one issue. That's the current paradigm of agents is basically just saving to markdowns.

240. 32:27

     The next one would be updating the weights, and that needs to happen locally.

241. 32:31

     Yeah. There's-- It's funny too 'cause, you know, uh, there was a tweet I think by [REDACTED:username], uh, that was like, "Why has fine-tuning as a service not taken off?"

242. 32:37

     It was, like, a couple months ago. Um, and I think a big reason for that is that model customization itself is also a very hard problem. And then what models can you hack on?

243. 32:46

     Uh, that's-- I mean, that's why NVIDIA releases Nemotron. It's a open source model, but everything from the dates-- Sorry, the data to the weights to the, uh, you know, uh, the recipes for how to do so and then, of course, you know, the final model, everything is open sourced, um, just so that it is a model that

244. 33:00

     you know you can safely, uh, use and customize. You know, it's interesting. I keep kind of flip-flopping on this point. I, I, I think, like, people look at how good the generalized models are, and you give it the right context.

245. 33:12

     Is that gonna be better than, than having a fine-tuned model and all the work that comes with that? And I, I keep going back and forth 'cause there is a lot of value in being able to have kind of a smaller, very specialized model and, and maybe a bunch of them working in unison to accomplish whatever task

246. 33:28

     you have. Um, but I, I-- Yeah, I'm not sure. What, what do you, what do you guys think about that?

247. 33:32

     Yeah.

248. 33:32

     I think, um, if I can chime in on this one, like, I think the, the beauty of, like, the open source ecosystem is that, like, all of these paths kind of get to be explored.

249. 33:40

     Yeah.

250. 33:40

     And then whatever wins, like, you know, it's-- it-- it's like what happened with speculative decoding. Um, you know, there's all these, like, different ways of doing speculative decoding, which is this idea that you can use a smaller model to basically approximate a larger model to speed it up.

251. 33:59

     Um, and, you know, you've seen, like, recently, literally I think in the last week, there's been, like, three different, like, quite, you know, seem to be breakthroughs in, like, speculative decoding that have just come from, like, different places, one from DeepSeek.

252. 34:14

     Um, there was some work that was done by, like, Modal and, uh, the SGLang team to, like, build, uh, DFlash-

253. 34:21

     Yeah

254. 34:21

     ... draft models for, like, various Qwen models that are, are like a big improvement on the previous ones. Um, there's, like, all this work being done. And so, like, we kind, kinda get to see, like, all of it and, you know, basically, like, whatever ends up being the best thing will just be the thing that kind of

255. 34:35

     wins, right?

256. 34:36

     That's kind of-- That's a great point where maybe it's not users and consumers actually customizing their own models and using the specialized models, but someone who has a need does so and does so in an open source fashion so that someone else can just adopt it.

257. 34:46

     Yeah.

258. 34:47

     Right? If I'm gonna start to do image gen for a particular use case, I can just go find a model. Actually, we saw this a lot also with, like, early LoRAs on, like, a lot of image gen models.

259. 34:56

     I feel like that was really big as, you know, in the 4.0 era as well. Um- I think ultimately what the end user cares about is, does it solve my use case, and can it do so within my budget?

260. 35:07

     Yeah.

261. 35:07

     And if there's another service that can manage the entire fine-tuning process, and, and I don't have to think... I know I keep saying I don't have to think about it.

262. 35:13

     Sometimes I do think about things. But, like, in this specific use case, um-

263. 35:17

     I was thinking about too many things already to add one more thing to it.

264. 35:20

     Yeah. There's-- Like, I, I care about my business. I don't wanna think about the, uh... I want it to be abstracted away from what I'm worrying about day to day.

265. 35:28

     Totally.

266. 35:29

     I could, uh, describe maybe a common flow that we see that's, like, distillation or, uh, general model, specific model. So if you think about it, um, in a lot of cases, it's like if the, the challenge I like to pose that's, like, a thought experiment is, like, if you po- tell someone, "Think of any object," and it's

267. 35:45

     like, "Think of an object in your fridge," the latter of those is actually easier, and the latter of those is actually where a lot of models kinda get deployed into a real world setting.

268. 35:52

     So, so for example, um, you know the Monterey Bay and the Monterey Bay Aquarium Research Institute, MBARI they're called? They discovered a new fish species recently, and they built with Roboflow to process all of the underwater, um, data ...

269. 36:05

     footage that they capture from their deep-sea, um, submarines. And they use large models like Segment Anything 3 and LMs as Judge to basically say, "Hey, let's take all this video footage, and let's build a auto label pipeline to understand as many things as we can about all the video footage that we've collected."

270. 36:21

     But then ultimately, you know, Sam knows everything about, uh, you know, things from fish to maybe architectural diagrams to, um, items in your fridge and everything in between. But they only care about things that are, you know, in this case, underwater deep-sea exploration.

271. 36:35

     And so a very common flow that we see for, like, the fine-tune last mile distillation is, okay, let's take the large context of an array of models, have those maybe all have a pass at understanding something, use LMs as Judge to say, "Hey, we agree with some amount of consensus," and now we have a specialized dataset.

272. 36:51

     Then we can use that specific model and actually run that on the submarine in real time and also post-processing for faster video. And if you think about it, like, a lot of problems are of that shape, where it's like, yeah, I want, like, general as much intelligence to understand the thing, and then ultimately the problem I'm solving

273. 37:05

     is specific enough, even if not fully unbounded. And a mistake I see pretty frequently is people thinking, like, Sam 3, which is an awesome model and a great family of models.

274. 37:14

     It's like, okay, well, I should take Sam 3 and then maybe just fine-tune Sam 3. And in some ways that actually doesn't make as much sense because you lose the thing that makes Sam 3 awesome, which is the open vocabulary capabilities.

275. 37:24

     What might be better is, like, if you know you're distilling down to a specific fixed class list, then you can actually drop the large, expensive autoencoder portion of Sam and use a specific maybe, like, debtor or more specialized model, depending on the task that you're solving, and get all the benefits of, you know, speed up and accuracy,

276. 37:41

     uh, while still having the general knowledge of preparing and curating your problem. And I think a lot of problems are of that shape.

277. 37:47

     Are you, are you managing that pipeline for your customer?

278. 37:51

     The-- Yeah, the tooling makes it so they can do that. Um-

279. 37:55

     Do you do it on their behalf?

280. 37:57

     Uh-

281. 37:57

     Or do you give them the tooling to do it?

282. 37:59

     Give them the platform to do that, and then there's, like, recipes-

283. 38:02

     Yeah

284. 38:02

     ... where someone can go and do that. And then like any good AI company, there's an FDE that if you want, I can sell you. [laughs] We would get around back here, and they can do it for you.

285. 38:09

     Uh-

286. 38:10

     That, that's a great example basically of the use cases and workflows that I was talking about when you're trying to find like, you know, collect the data as you go for your business, for your enterprise, and then decide how you're gonna fine-tune a model, a small specialized model on those use cases as you grow.

287. 38:28

     That's how you become more token efficient.

288. 38:30

     Yeah.

289. 38:31

     Yeah. I, I, I think th-this year and, and next year you're gonna see a lot of using these, like, monster frontier models to bootstrap, you know, like a more efficient setup that runs on open source.

290. 38:42

     Yep.

291. 38:42

     And I think, you know, that's great. Like, I think this is how a lot of the open source models have been built right now, and it's, it's, it's proving to be, like, quite hard to, you know, stop that.

292. 38:53

     And I would just enc- [chuckles] you know, encourage people to just, you know, move away potentially from like, you know, these, uh, frontier models, but like, you know, use them.

293. 39:01

     Use them for what-- for, for bootstrapping that, right? So like, um-

294. 39:05

     Love that you said that actually because that's how the word AI engineer got coined, right? When Swyx released that blog post that coined it, the way that we used to build AI products was we would start with the m- with the machine learning.

295. 39:16

     We would start by training a model, then we would go try to discover a use case.

296. 39:19

     Yeah.

297. 39:19

     And what these big models allowed us to do is actually flip it. We said we could start with discovering a use case and then get into ML if it makes sense.

298. 39:28

     And that was actually, uh, honestly an amazing foresight from Swyx because he kind of defined that pattern, uh, for us a few years ago and gave this amazing conference as well.

299. 39:37

     Yeah, it's crazy how far this conference has come in just three years.

300. 39:40

     Yeah, totally. I have a question. Do you guys have questions in the, in the crowd? We have a few more minutes, and I know we were talking about opening this up if you guys wanna ask the panelists directly.

301. 39:49

     Just, just shout.

302. 39:51

     Yeah. [laughs]

303. 39:52

     Please go ahead.

304. 39:52

     Um, what are the big open problems in local AI? I feel like you're the luminaries of the field. You know more, you have a line of sight on, like, the biggest open problems.

305. 40:01

     You wanna repeat the question?

306. 40:02

     I'll repeat that one.

307. 40:03

     He just asked, what are the open problems in local AI?

308. 40:05

     I, I can repeat it.

309. 40:06

     Yeah.

310. 40:06

     It's what are the biggest open problems in local AI? Um-

311. 40:10

     Um, it continues to be optimizations and, uh, for inference. It continues to be getting things easily kickstarted, which is-

312. 40:19

     Yeah

313. 40:19

     ... you know, what ODS is, XOS for specific hardware. It continues to be how can I make the most out of my budget constraints and hardware constraints, and how can I squeeze the most performance out of that?

314. 40:33

     Uh, you know, the, the models that-- You know, I still have the same thirty-nineties that I used to run Llama two and like Dolphin fine-tunes on, and now they're running Qwen 3.5, Qwen 3.6, twenty-seven billion parameters with excellent performance.

315. 40:46

     You know, more on that in my next presentation.

316. 40:48

     Yeah.

317. 40:48

     So, you know, um, it, it comes down to the optimization, and we are still very early that there is space for so many players, for so many contributors. We need all the help we can get to make this thing the success that it needs to be, that we need to make local AI the default.

318. 41:04

     This has been my stance for years now. I have been saying open source AI must win for s- like since forever, and the way we do that is by giving the people, whether that's individuals at home, middle-sized businesses or enterprises, an easy way to use these models in a very efficient way that doesn't give them headaches more

319. 41:24

     than solve their problems.

320. 41:25

     Totally. And if you look at the panels that are, that are happening today, those are what we believe to be the biggest open-ended questions, which is why we tried to assemble this panel.

321. 41:32

     And so the-- you know, we have quantization, so all about talking about the footprint so that the models do fit on these, uh, uh, on these smaller hardware footprints.

322. 41:40

     Uh, there's model routing, there's models generally. Uh, so those are kind of the-- what we feel like are the big problems now. And as we do future local AI summits, um, I-- you know, every time the panels should change to be the topics du jour that are kind of holding the...

323. 41:54

     Or, or we're gonna be-- gonna usher the next chapter in.

324. 41:57

     Can I add one thing to that? One of the biggest challenge-- So I think there's two, two big challenges. One is what we've been talking about of basically the trade-off of simplicity versus customizability.

325. 42:07

     It's local, it's yours. You can do different things, optimization. If it's hosted, it's built out of the box the way. That trade-off is, is always difficult. The second, which I actually encourage people in this room to help solve, is the importance of open models is becoming increasingly in question.

326. 42:22

     And I actually think that, like, if you think Local AI is important, then you think open-source AI is important, and it's actually really important to be an advocate for being able to use, change, adapt, and toy with models.

327. 42:32

     And so I think that that's a problem that, um, could increasingly be something that we feel less control over absent advocating.

328. 42:41

     That's a great point. I think everyone here feels very passionate about open source. That is why we actually have access to the space at all. That's why the space has, has progressed.

329. 42:49

     It's a, it's a necessary competitive environment. It allows for the best ideas to make their way to everybody. Um, so definitely when there's talks about that being a threat, you know, we need to invest and advocate for open source.

330. 43:00

     And open source is so much bigger than AI, right? Like, there's a reason why computation-- like compute was invented in, on the East Coast, but Silicon Valley happened here, and it's because hippies realized that they could share ideas for free and software. [laughs]

331. 43:13

     I think Silicon Valley is this kind of tension between the capitalists that wanna make money and hippies that wanna give these ideas away for free, and that tension is what creates such an incredible environment here.

332. 43:22

     They can coexist, by the way. [laughs]

333. 43:23

     And they, they do.

334. 43:24

     You can have consumers, and you can sell to businesses.

335. 43:26

     And that's when the space works, works best, and so-

336. 43:28

     Ca- can I just say, like, if you care about this, and you wanna be more of an active participant, but maybe, you know, you don't wanna get involved in the technical side or you're not technical, then there are ways to get involved.

337. 43:36

     So there's a website that just came out called RightToIntelligence.org, and this is a way for you to, like, get involved and actually, like, advocate for open source and to ensure that, you know, we maintain freedom of intelligence.

338. 43:49

     Awesome. So we're out of time. I wanna thank you guys so much. Local AI Summit's gonna be a ton of fun. Thank you guys for this incredible way to kick this off.

339. 43:56

     We're gonna have amazing demos. We're running foundational intelligence models here inside of this room. Uh, we're gonna have a couple of amazing-- uh, a, a few more panels. It's gonna be an exciting day, um, and we're all lingering here, so ask questions and please participate.

340. 44:09

     Thank you. [audience applauding] [upbeat music]