Why MLX

Talk transcript

1. 0:00

   [upbeat music] Good morning, everyone.

2. 0:17

   Uh, how many of you here have ever tried to run AI on your phone, on your MacBook?

3. 0:26

   How was that experience? Was good? More or less, yes. More or less? Yeah. Okay. So this talk today is for you. I'm going to show you how you can deploy and manage AI agents, or even voice agents if you will, completely on-device using MLX.

4. 0:45

   Uh, today's agenda, of course, we're gonna start with why on-device. A lot of you use cloud code, uh, subscriptions, and many other subscriptions. I want to convince you today to offload some of that subscription completely on-device, and then all you need to pay is your energy bill.

5. 1:01

   Then I'm gonna talk about MLX, and I'll give you a small demo, and I will show you some of the amazing community projects that I've seen are built by the community using the projects that I will demonstrate today.

6. 1:13

   In 2020, something very magical and also weird happened. It was the best, yet also worst year of my life. It was a weird one. One-- on one side, my dad became blind, and on the other side, Apple released one of the most powerful, um, chips for on-device intelligence ever.

7. 1:38

   And I remember I was talking to my dad, and I said, "Hey,

8. 1:44

   I promise you that I'll get you back to reading." He's one of the most voracious readers I know, and losing his sight was one of the main things that kind of bummed him out.

9. 1:53

   He couldn't no, no longer consume information. And I made this really weird promise. I said, "Hey, I'm going, I'm going to f-fix this some way somehow." And at the same time, this happened.

10. 2:06

    And I thought, "Hmm, compute on the cloud doesn't necessarily solve all of these use cases," because my dad lives in Africa, and there we don't have internet as easy as we have here, or the subscription plan, plans there are really, really bad.

11. 2:21

    So I thought on-device is the future. And then 2023,

12. 2:27

    I was investigating on GitHub some really cool projects, and I saw this project here called MLX. It's an array framework for Apple silicon. You can imagine PyTorch or TensorFlow for Apple silicon.

13. 2:41

    And I tried out the, the initial example, and I thought, "There's a future here." There's a future that was promised for all of us that all of the big companies like Meta and, um, Google could not really deliver because they were trying to optimize for scale for the cloud, and Apple did something different.

14. 3:00

    So then I started contributing to MLX. Three years later, we have over one point five million downloads, over four thousand models ported, and we work with some of the best frontier labs to deliver to you day zero support for all of your open source models.

15. 3:15

    You can e-imagine Gemma four, the latest Gemma four, we had day zero support for that on MLX, meaning you can run all of the best frontier open source models completely on your MacBook, on your iPhone or your iPad.

16. 3:30

    So let's start with vision, which for him was the-- kind of the biggest sensory deprivation. He cannot see. He cannot navigate the world. And I thought, "Well, the easiest way to do that is by giving him his vision back or giving him a system that can help him navigate the world."

17. 3:49

    And I think in, in 2021, I went into this hackathon, and I built these goggles with my team that could, like, tell you what's in front of you. But then MLX VLM became the second iteration of that, that allows you to do this not only via some weird, uh, glasses, but even on your iPhone.

18. 4:07

    You can now just pick up your phone, point it at something, and you'll be able to, you know, understand what's in front of you. And then you have omni models nowadays.

19. 4:16

    Uh, beyond just vision models, these are models that can also take in what? Audio. For my dad, in particular, typing is not really a reality, but he can speak, and with his speech, he can control the camera and understand what's in front of him, what he needs to do, and navigate the world.

20. 4:34

    So this is MLX VLM. Right now, if you use LM Studio, it's one of the main, uh, engines that powers LM Studio, powers Liquid AI models, and many other models out there.

21. 4:45

    But when you think on-device, you might think, "Well, that's weird. Uh, I cannot run really large models." Well, that's not true anymore. You can now run models of hundreds of billions of parameters, even on your, on your initial M1 MacBook.

22. 5:01

    There's a lot of, uh, improvements that the community has made that allows you to run even the largest and most, uh, abnormal models completely on-device. I have, uh, some examples that show that you can run models like Gemma 426B on an iPhone using your storage, and you can still get reasonable speeds.

23. 5:23

    Then after a year or [REDACTED:generic_id], I was also experimenting with, you know, how can we enable humans more and give them more accessibility, especially the ones that don't have all the sensories.

24. 5:37

    Um, and I thought audio is the next iteration. But it was for that and also for a very selfish reason. I wanted to be able to control my computer without being in front of my computer all the time.

25. 5:48

    What if I could just blurt a command and have my computer do it? This is more of like the Jarvis vision of the world, where you can just speak to your computer and have actions done for you.

26. 5:58

    We started off with text-to-speech, and then that on became Marvis, uh, one of our custom models that can generate audio in less than a hundred milliseconds. Um, then you, uh, we have, uh, speech-to-text, which allows you to speak to your computer and have it transcribed in real time.

27. 6:14

    So, for example, if you ever-- how many of you here use U- WhisperFlow? Or Super Whisper. Yeah, you can now vibe code that application. Just point Cloud Code or Codex into MLX Audio, ask, ask it to build it for you, and you'll have it in like ten minutes.

28. 6:31

    And then you also have speech-to-speech. So beyond the first [REDACTED:generic_id] capabilities, a big unlock is to have the computer speak back to you. So speech-to-speech is one of the core capabilities that we recently added, and we also support both Python and Swift.

29. 6:46

    Uh, we started off with Python because it's just much easier, it scales faster. But we also understand that native experiences matter, so with Swift, you can now build fully native applications enabled by audio intelligence as well as vision intelligence.

30. 7:00

    And on the, on the right side, your right side there, you have, uh, our modular pipeline that beyond just models that are speech-to-speech natively, you can actually chain a series of different capabilities to create a modular speech pipeline.

31. 7:15

    For, for instance, you can use... You can choose which a-automatic speech recognition model you want, you can choose which language model you want, and you can also choose which text-to-speech model you want.

32. 7:26

    And this way, you can create, uh, really custom and modular experiences that fit on every single hardware budget. So if you have a, a very simple M1 first generation Apple silicon, or even the latest, you can adjust that to your hardware.

33. 7:42

    But then you might think, "Well, speech-to-speech or text-to-speech is not that good. I've seen some videos, audios of, of speech-to-speech. That may be... I, I don't know. Is it-- Does it sound good?"

34. 7:53

    I can promise you that it does, and I'll show you a demo in a bit. So let's start with vision.

35. 8:00

    And, um, with vision, I have a couple of examples. The first one here is real-time, um, image analysis, so you can understand what's happening. It's a very simple command, and we'll make this even simpler.

36. 8:14

    Right now, it's in Python. It will come to Swift very soon. But if I run this command, it's gonna to run the RF detector model by, um, Roboflow. And as you can see, this is completely real time.

37. 8:26

    It's understanding where... And this is all running on my computer. I can actually just... To make sure that this is very clear. I don't know if I turn off the internet what's gonna happen.

38. 8:36

    But here it is continuously running. I can grab a glass, and it will also detect that. Of course, it's thinking it's wine, but, um, I'm sober. [laughs]

39. 8:48

    So this is running real time, completely on device, on my Mac. It can also run on your phone. And, uh, this is one example. I wanna show you another really cool example that you can do with this particular use case.

40. 9:00

    Have you ever tried, have you ever tried the, the... When you're in a meeting, you, you wanna blur the background. You, you know that Google does this and et cetera.

41. 9:10

    So now you can actually do this natively, and you can build this kind of experiences into your products. I'm not sure you can see that it's blurring the background, but it actually is blurring the background, and it's detecting my mask in real time and will detect other objects as well.

42. 9:27

    All right, so that's example number one. Number [REDACTED:generic_id] is you can run really, um, large models completely on device. Here's Gemma 4, was released, I think, a week ago, a couple...

43. 9:41

    Yeah, last week. It was released last week. And with MLX VLM, you just run mlx_vlm.chat_ui, and you pass in the model you want, and it should load that model and give you a simple interface for you to get started with Gradio.

44. 9:55

    Wait. Not sure what's happening. This is the problem with demo. Sometimes the gods don't want it to [laughs] to, to do the demo. Give me a second.

45. 10:06

    Okay, not sure what's happening. Mm. Okay. Well,

46. 10:14

    I need to quickly get something to close off.

47. 10:20

    Um. Mm, mm, mm. So okay, it stopped.

48. 10:28

    Let's start-- Let's try that again. Okay. It's trying to download some model files, but now I'll turn off the internet again just to show you that this is now running completely on device.

49. 10:41

    So now we have a chat window. I don't know if you can see this, or I have to zoom in more. And we can choose any image to analyze.

50. 10:51

    Let me see a very simple one. Okay. Okay, here. Describe this image in detail.

51. 11:05

    So here it is. It's saying that it's a profile of a name, man named [REDACTED:username], and it's picking up all the different details like my bio and et cetera.

52. 11:17

    And all of this is running on device. Um, it's using the GPU in this, uh, particular device. So this particular machine has like ninety-six gigabytes of VRAM, so I can run actually all of the models I showcased so far in real time, all of them at the same time.

53. 11:35

    So this is demo number [REDACTED:generic_id]. Let's now get back. I wanted to show you audio, but it seems like the Swift branch, there was something there I couldn't really figure out this morning.

54. 11:49

    But for the sake of examples, I will show you some of the really cool community use cases. Um, we have one here, one of the creators, uh, I didn't include his, his particular use case, but I'll show you a video in just a bit.

55. 12:02

    So let me try and get this in full screen

56. 12:07

    so that you can see better. One second. Here. Okay, so we are back. Um, the first example is grounded visual reasoning. You can use Gemma 4. You can use the model I just showcased, the RF detector, or any of our, um, um- ...

57. 12:27

    perception models, and you can create really cool experiences like

58. 12:32

    here. You can detect all the fires, you can ask it to detect particular items in the video, and this is all gonna happen on, completely on device without the use of internet.

59. 12:42

    And now you can have, for example, security systems that run completely on a MacBook, um, on your house, and you can analyze even your dash cam. I have a dash cam video, and I've had some really cool, or not cool, but crazy experiences.

60. 12:57

    And I usually use this kind of system here that I built to kind of analyze the video footage afterwards. And then you have example number [REDACTED:generic_id]. This is more of like a honorable mention, which is cartoons generated completely on device, um, using MLX Video, one of the most recent projects I, I'm running.

61. 13:16

    And here's one of the, I think, coolest, uh, video generated by one of our users. [laughs]

62. 13:23

    This is all generated on device with a simple text prompt. [whimsical music] [screaming] [laughs]

63. 13:41

    So... [upbeat music] [laughing] [screaming] And he did something very interesting, which is he chained... This, this is not like one video that he generated all at once.

64. 13:57

    What he did is, like, he chained a system - [laughs] ... that can continuously generate from the video. [laughs] So you can create a cohesive story, even though it was not, like, one-shotted.

65. 14:08

    And, uh, this particular, uh, system can run even on a MacBook with 16 gigabytes of, of VRAM. [screaming]

66. 14:16

    So yeah, pretty cool. He has a lot more, uh, on Twitter. I will put his, uh... Uh, like, I'll share, I will reshare on my Twitter, so if you, uh, check my Twitter, I'll reshare all of these videos.

67. 14:27

    And then I think one of the latest ones that I will show... But before that, let me show you one, one here from one of our participants in, in the...

68. 14:39

    on, on the floor. So if I go to Drive...

69. 14:45

    Let me see. In my Drive. Okay. Neywa Labs. You have...

70. 14:59

    Here. This is actually- It was a small voice and a small gesture ... by this gentleman here in front, Adrian. He builds this application called Locally, and using MLX Audio and Marvis TTS, he now gave that particular application the capability of speaking back to its users.

71. 15:17

    It was a small voice and a small gesture. Can you increase the volume? The [REDACTED:location] was a bar for professional expatriates. You could drink there for a week and never hear [REDACTED:generic_id] words in Japanese.

72. 15:29

    All right. So the, that is one of the examples where you can actually build really beautiful native experiences, and if you have a, a good touch of design, a killer application as well.

73. 15:39

    And finally, I think this is one of the most exciting parts of what I, where, where I think we are going next with on-device AI, which is

74. 15:48

    robotics. So last year I acquired a robot called, um, Ricci Mini, and the way that I power my Ricci Mini in particular is that I use MLX Audio, MLX Vision to give it all the capabilities or perception capabilities using its camera and audio, uh, input.

75. 16:07

    So here it is. It's also doing voice, real-time voice cloning of the original, uh, how do you call it? Iron Man JARVIS voice. So I hope you can hear this.

76. 16:18

    Hey, JARVIS.

77. 16:26

    Hey there. Great to see you. How's it going?

78. 16:33

    So you can chain and, and build a lot of really cool applications. This is just a start, and I hope that in the future you can understand, or today, from today, you can understand that you can build agents that can hear, see, and sound just like you or y- one of your loved ones today running on your

79. 16:54

    iPhone, iPad, Mac, or even your robot. Thank you. [clapping]

80. 17:03

    Questions?

81. 17:07

    So you already, like... Apple is like, uh, uh, promoting their neural engine a lot.

82. 17:14

    Yes.

83. 17:14

    And I've also tried it MLX.

84. 17:16

    Yeah.

85. 17:16

    But if you look at the, the usage of the neural engine, it's always like-

86. 17:21

    Yeah

87. 17:21

    ... it's at zero.

88. 17:22

    That's a great question. So MLX uses the GPU, not the neural engine. For you to enable the neural engine, you need Core ML, and right now, uh, Core ML do- it does not really run well, like it's not an easy experience for developers.

89. 17:37

    I hope, uh, by WWDC Apple solves the private API issues, and when they do, we have some internal projects that can allow you to run a hybrid inference across both.

90. 17:49

    Yeah.

91. 17:51

    Okay.

92. 17:51

    It, it, it will be.

93. 17:52

    Yeah.

94. 17:52

    It will be. But we also think that they might be changing the n- the, the neural, uh, engine and putting some components into the GPU with, for example, the M5 series you can see that it already has some components of it, and w- we just don't know where d- what direction they are h- they are heading.

95. 18:09

    But it's exciting. Let's wait for WWDC. Questions? Yeah.

96. 18:14

    Is there a tool to see your GPU usage in real time?

97. 18:18

    Yes. So the easiest tool for you to get started is, um, mactop. If you run mactop It's going to pretty much show you all your usage. This is by, uh, Carson.

98. 18:33

    Um, he's a really cool dude. So you can see here pretty much what's happening, the GPU, CPU, um, and you can have this overlay across your, your, your device.

99. 18:44

    And if we do run inference, so let's say I start a new window, and if I can find that command.

100. 18:52

     Okay. So if I do start running inference on this... Let me put here for a bit,

101. 19:00

     and... Okay. You will see that the GPU now is, is gonna start moving up.

102. 19:10

     And if I say, "Hi, are you..." You see? The GPU is already moving up, and th- and this is one of the easiest ways that I found to, uh, track the performance.

103. 19:29

     Uh, any other questions? Yep.

104. 19:32

     Like, uh, you als- also mentioned omni models.

105. 19:34

     Yes.

106. 19:35

     Like, what's the one that you would recommend? Like, what is state-of-the-art at this point?

107. 19:40

     So you have a couple options. The first one is Gemma 4, the E version. They have this n- uh, Gemma 4 with the number E... Or the letter E, and then a number, E4, E2.

108. 19:52

     Those are omni models. They, uh, they take, um, image, audio, and text as input or any of the variation of the three. Um, and then you also have Qwen3-Omni, which is a much larger model, around thirty billion parameters, but you can also run on device.

109. 20:08

     Those are the top ones that I know.

110. 20:10

     What are your limit-- What-- I'm sorry. What are the key limitations of those models at the moment?

111. 20:17

     What are the key limitations? I think it depends on, on your particular use case. So there are certain things the models just cannot do. Th- you're not gonna get the performance of Claude 3 or, or 4 point, 4.6 Opus today.

112. 20:29

     But maybe in six months, these open source models will have that performance. So the experience should be kind of adjusted to your expe- the expectations of performance. That's the only thing I would say.

113. 20:40

     Outside of that, I don't see any limitations. You can run, uh, inference on hundreds of images in parallel. You can run inference on, uh, many, many documents, and you c- you now have context of up to a million thanks to a, a recent breakthrough that I made with TurboQuant.

114. 20:56

     So now you can actually serve, uh, one million contexts completely on device, depending on the size of the model and your hardware. But you can do that today.

115. 21:06

     Um, something else. I don't know the specifics about it. Like you had the, the TurboQuant paper that was released-

116. 21:12

     Yes

117. 21:12

     ... recently by Google. And, and, and how far as I understood it, it was quite a general approach.

118. 21:18

     Mm-hmm.

119. 21:18

     Like, is it, like, are there efforts to, uh, like, generally implement it in this kind of- [laughs]

120. 21:25

     So I was one of the first people on the world to implement TurboQuant publicly.

121. 21:30

     Oh, interesting.

122. 21:30

     So, like, thirty minutes after the paper was out, I already ha- had implemented it, and I made this tweet at, like, three AM. I didn't know that it would go s- like, this viral.

123. 21:40

     But if you write... If you go to my profile and you write TurboQuant,

124. 21:48

     you'll be able to see there's this particular post here. So this was, like, twenty-fifth March, and pretty much the same day, but just was, like, midnight, and it got, like, seven hundred thousand views because of that.

125. 22:00

     So TurboQuant does work. Um, example is, like, the full model takes, uh, almost one gigabyte of, uh, uh, KV cache or RAM, and by using TurboQuant, you can reduce that by four X.

126. 22:15

     But is it, like-

127. 22:16

     Yeah

128. 22:16

     ... the same quality or-

129. 22:16

     Uh, si- yeah, similar quality. A- as you can see, when it, when, when you see exact match, it means that it matches the performance of the, the responses of the full model.

130. 22:24

     And I also publi- publicize the, uh, full, um, results and performance. For example, here, uh, when you get to, like, three hundred thousand context, the performance almost doubles in terms of, like, throughput.

131. 22:37

     So yeah.

132. 22:39

     Super.

133. 22:39

     Yeah. Uh, there-- This is one of the many things that I try to do to enable on-device to go even further.

134. 22:47

     Yep. Any other questions? All right. Thank you. [audience clapping] [outro music]