AI Engineer Europe 2026
Why MLX
Read the talk
Why MLX: Building Agents That See, Hear, and Speak Locally
Local inference turns accessibility, voice control, and visual perception into applications that can work without a reliable connection, with MLX providing the building blocks on Apple silicon.
Before you start: Basic familiarity with model inference and Python command-line tools will help with the demonstrations; no prior MLX experience is required.
What happens when an agent cannot depend on the internet?
Have you tried running AI on your phone or MacBook—and how good was the experience? The practical goal here is to deploy AI agents, including voice agents, entirely on those devices. Moving some workloads off subscriptions shifts their inference cost toward the hardware and electricity already available to the user. The route runs from the reasons for local execution through MLX, live demonstrations, and applications built by its community.
The need became personal in 2020. The speaker’s father lost his sight that year, while Apple introduced its first M1 hardware. A voracious reader could no longer access information as before, and his son promised to help him read again. His father lives in Africa, where unreliable connectivity and poor internet subscription options made a cloud-dependent solution a difficult fit. On-device inference was an accessibility requirement: assistance needed to remain available when a connection was not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an array framework to accessible perception
In 2023, a GitHub search led the speaker to MLX, an array framework for Apple silicon. Think of the role PyTorch or TensorFlow plays in a machine-learning application: arrays, operations, and the machinery for running models. The initial examples suggested a practical foundation for local intelligence, and he began contributing. The speaker reports more than 1.5 million downloads and more than 4,000 ported models three years later. He also describes collaboration with frontier labs and day-zero support for Gemma 4, bringing supported open models to MacBooks, iPhones, and iPads.
Vision was the first application: help someone understand what is in front of them and navigate their surroundings. An earlier attempt, built with a team at a 2021 hackathon, used goggles that described nearby objects. MLX-VLM became a second iteration of that idea, making the same kind of assistance possible through a phone camera. Point the phone at something and ask what it sees.
Omni models add audio input to that interaction. For a user who cannot readily type, speech can control the camera and request information about the scene. The speaker identifies MLX-VLM as an engine used by LM Studio and for Liquid AI models, then challenges the assumption that local inference necessarily means small models. He reports examples of models with hundreds of billions of parameters running on an early M1 MacBook, and Gemma 4 26B running on an iPhone using storage. These capacity claims come without a specified throughput or configuration; fitting a model and achieving an acceptable interactive experience remain separate questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Assembling a voice agent for the available hardware
Audio followed vision, both for accessibility and for a more everyday desire: controlling a computer without sitting in front of it. Speak a command and have the computer act—the JARVIS-style interaction. Work began with text-to-speech and Marvis. The speaker reports audio generation in less than 100 milliseconds, without specifying the model version, hardware, or latency measurement boundary. Speech-to-text supplies the other direction, transcribing the user’s speech in real time.
Wispr Flow and superwhisper illustrate the kind of dictation product these components can support. The speaker suggests pointing Claude Code or Codex at MLX Audio to build a comparable application in about ten minutes; that is an estimate, not a demonstrated build. Speech-to-speech then closes the conversational loop, allowing the computer to respond audibly. Python came first for development speed, followed by Swift support for native applications combining audio and vision.
There are two ways to assemble the conversation:
| Approach | How it works | Main choice |
|---|---|---|
| Native speech-to-speech | A model handles the speech interaction | Select the integrated model |
| Modular pipeline | Speech recognition → language model → speech synthesis | Select each component separately |
In the modular version, automatic speech recognition, the language model, and text-to-speech are independent choices. That lets the application adapt to its hardware budget, from a first-generation M1 to a newer machine, without requiring the same model combination everywhere. The remaining question is whether the resulting speech sounds good enough for the experience being built.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Detecting objects and masking a background
The first live demonstration starts a camera-analysis application with a Python command. It runs Roboflow’s RF-DETR locally, detecting objects as the scene changes. Swift support for this particular demonstration is described as forthcoming. During an attempt to show operation without an internet connection, processing continues. When the speaker lifts a glass, the detector identifies it as wine, prompting a joke about being sober. The visible demonstration runs on a Mac; phone execution is also described as possible.
The next variation applies perception to a familiar meeting feature: background blur. A real-time person mask separates the speaker from the surrounding scene so the background can be blurred while the foreground remains visible. The speaker notes that the blur may be difficult to see in the presentation, but the mechanism is the useful part: a local segmentation result becomes an input to a product’s rendering behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Opening a multimodal chat interface
The next demonstration moves from live detection to a larger multimodal model. Gemma 4 is described as having arrived the previous week; its published release date is April 2, 2026. MLX-VLM’s mlx_vlm.chat_ui entry point accepts a model selection and opens a Gradio interface. With MLX-VLM’s chat dependencies installed, a small Python launcher can pass the chosen model through to that module:
python
import subprocess
import sys
model = input("MLX-VLM model ID or local path: ").strip()
if not model:
raise SystemExit("A model ID or local path is required.")
subprocess.run(
[sys.executable, "-m", "mlx_vlm.chat_ui", "--model", model],
check=True,
)
The interface provides a quick way to test a model before building a custom application around it.
The launch initially stalls. On retry, it attempts to download model files before the speaker switches the internet off again. That sequence exposes an operational distinction: local inference still requires the model files to be available locally. Once the chat window opens, he selects an image and enters “Describe this image in detail.” The response identifies a profile for Prince Canuma and describes details including the biography.
The speaker describes the demonstration machine as having 96 GB of memory and says it can run all the models shown so far simultaneously in real time. On Apple silicon, this is unified memory shared by CPU and GPU, rather than a separate dedicated VRAM pool; the demonstrated inference uses the GPU. The planned live audio demonstration does not follow: an unresolved problem in the Swift branch that morning prevents it, so community recordings provide the audio examples instead.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspecting footage and continuing a generated story
Grounded visual reasoning turns model output into a way to inspect video. Gemma 4, RF-DETR, or another perception model can look for fires or particular requested objects without sending the footage to a remote service. The displayed example asks to segment every jet and shows colored detection overlays on aircraft and their contrails. The proposed applications include home security running on a MacBook and analysis of dashcam footage.
The speaker already uses a system like this to review dashcam footage after incidents. He then switches from analyzing video to generating it with MLX Video, showing a community-created cartoon that starts from a text prompt. The sequence is not generated as one long clip: successive generations continue from previous video, allowing a story to develop across multiple steps.
The speaker says this particular chained-generation system can run on a MacBook with 16 GB of unified memory. The model, settings, and generation time are not given. He promises to reshare more of the creator’s videos through his social profile; the demonstrated contribution is the chaining strategy that turns individual generations into a continuing sequence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Giving applications—and a robot—a voice
The next recording, located in Drive under Neywa Labs, comes from Adrian, a participant in the audience who built an application called Locally. MLX Audio and Marvis TTS give it spoken output. The sample reads prose aloud, beginning with a small voice and gesture and continuing into a passage about the Chatsubo bar. It connects the speech components to a recognizable native reading experience. The speaker emphasizes that product design matters alongside the model integration.
Robotics extends the same combination of perception and speech into a physical device. The speaker acquired a Reachy Mini the previous year and describes using MLX Audio and MLX Vision to supply capabilities based on its camera and audio input. A recording demonstrates what he describes as real-time cloning of the Iron Man JARVIS voice. After a greeting, the robot responds: “Hey there. Great to see you. How’s it going?”
That is the prepared talk’s destination: agents that can hear, see, and speak in a familiar voice, running on an iPhone, iPad, Mac, or robot. The individual libraries become useful when they are composed into an interaction that serves the person using it. The questions that follow turn to the execution backend, observability, model limits, and the memory costs of longer contexts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where inference runs, and how to observe it
An audience member has noticed that Neural Engine utilization stays at zero while MLX is running. The explanation is that the demonstrated inference path uses the GPU, not the Neural Engine. MLX also supports CPU execution; it should not be understood as a GPU-only framework. For Neural Engine access, the speaker points to Core ML, while describing its developer experience and private-API issues as obstacles.
He describes internal work toward hybrid GPU and Neural Engine inference, contingent on Apple resolving API issues, and hopes for progress at WWDC. He also speculates about Neural Engine components moving into the GPU, citing the M5 generation, while acknowledging that Apple’s direction is unknown. These are prospective paths, not capabilities demonstrated in the session.
For observing the current path, he recommends mactop, whose author is Carsen Klock. It displays CPU and GPU activity, and the demonstration uses a monitoring view alongside the application. To reproduce the observation:
- Open
mactopand watch the utilization display. - Start the local inference application.
- Submit a prompt and observe the GPU activity as the model responds.
In the demonstration, GPU usage rises after inference starts and a greeting is entered. This connects the application’s response to visible local compute activity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing an omni model without overpromising
Asked which omni models to try, the speaker names two families:
- Gemma 4 E2B and E4B: The edge variants accept image, audio, and text inputs in combinations. These are the published names of the variants referred to aloud as E2 and E4.
- Qwen3-Omni: A larger local option, described by the speaker as approximately 30 billion parameters, without identifying the precise variant or parameter accounting.
The distinction is useful for choosing a starting point: the input modalities must fit the application, and the model must fit the device.
The main limitation he identifies is capability relative to the task. Do not expect these local models to match the contemporary Claude/Opus models he invokes as a comparison. His suggestion that open models might reach that level within six months is a forecast. The application’s promises need to match the model’s current behavior.
He also describes inference over hundreds of images in parallel and many documents, then credits TurboQuant with making much longer contexts feasible. The speaker claims local operation with context lengths up to one million, depending on model size and hardware. Cache compression reduces the memory required to retain context; it does not automatically extend a model’s supported context window. For example, Google specifies 128K for Gemma 4’s edge variants and up to 256K for its larger variants, so the million-context claim needs a separate model and workload configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reducing the cache cost of long contexts
A final question asks whether Google’s general TurboQuant approach is being implemented. The speaker says he produced one of the earliest public implementations, about thirty minutes after encountering what he calls the paper’s release. Searching his profile brings up a March 25 post that he says attracted roughly 700,000 views after being posted late at night. The publication chronology needs one distinction: the paper was submitted to arXiv on April 28, 2025, while Google’s research announcement followed on March 24, 2026. The implementation account belongs to the speaker’s recollection of that publicity cycle.
The result he highlights is a reduction in KV-cache memory, the stored attention keys and values retained during inference. He describes a cache approaching 1 GB becoming roughly four times smaller. The inspected table gives a more precise view of the displayed comparison:
| Configuration | Exact match | Average cache, GB | Displayed reduction |
|---|---|---|---|
| Full | 6/6 | 0.703 | — |
| TurboQuant 2.5 | 6/6 | 0.143 | 4.9× smaller |
| TurboQuant 3.5 | 6/6 | 0.185 | 3.8× smaller |
The speaker interprets the exact matches as agreement with full-model responses and describes the quality as similar. The model, hardware, and six evaluation cases are not specified here, so those matches establish agreement for the displayed test, not general quality equivalence. A smaller KV cache also does not mean total application memory falls by the same factor: model weights and other allocations remain.
The speaker reports almost doubled throughput at around 300,000 context in his published results. The workload and hardware behind that comparison are not specified in the discussion. It closes the session on a concrete engineering constraint: as context grows, cache memory and the cost of accessing it matter alongside model size. Reducing that burden is one way to extend what an on-device application can do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Apple's machine-learning array framework with Python and Swift interfaces and shared CPU/GPU memory.
Local multimodal inference and fine-tuning, with a Gradio chat interface and KV-cache quantization.
Google introduces Gemma 4's model sizes, input modalities and context windows.
Speech synthesis, transcription and speech-to-speech tools for Apple silicon.
Streaming speech model with MLX Audio examples for synthesis and reference-audio voice cloning.
Roboflow's object detection and segmentation model project.
Image, video and audio generation models running on Mac through MLX.
Terminal monitoring of real-time Apple-silicon system activity.
Original research on online vector quantization, including KV-cache compression.
Further reading
An explanation of TurboQuant, PolarQuant and QJL for memory-efficient vector processing.
Read the complete timestamped transcript
- 0:00
[upbeat music] Good morning, everyone.
- 0:17
Uh, how many of you here have ever tried to run AI on your phone, on your MacBook?
- 0:26
How was that experience? Was good? More or less, yes. More or less? Yeah. Okay. So this talk today is for you. I'm going to show you how you can deploy and manage AI agents, or even voice agents if you will, completely on-device using MLX.
- 0:45
Uh, today's agenda, of course, we're gonna start with why on-device. A lot of you use cloud code, uh, subscriptions, and many other subscriptions. I want to convince you today to offload some of that subscription completely on-device, and then all you need to pay is your energy bill.
- 1:01
Then I'm gonna talk about MLX, and I'll give you a small demo, and I will show you some of the amazing community projects that I've seen are built by the community using the projects that I will demonstrate today.
- 1:13
In 2020, something very magical and also weird happened. It was the best, yet also worst year of my life. It was a weird one. One-- on one side, my dad became blind, and on the other side, Apple released one of the most powerful, um, chips for on-device intelligence ever.
- 1:38
And I remember I was talking to my dad, and I said, "Hey,
- 1:44
I promise you that I'll get you back to reading." He's one of the most voracious readers I know, and losing his sight was one of the main things that kind of bummed him out.
- 1:53
He couldn't no, no longer consume information. And I made this really weird promise. I said, "Hey, I'm going, I'm going to f-fix this some way somehow." And at the same time, this happened.
- 2:06
And I thought, "Hmm, compute on the cloud doesn't necessarily solve all of these use cases," because my dad lives in Africa, and there we don't have internet as easy as we have here, or the subscription plan, plans there are really, really bad.
- 2:21
So I thought on-device is the future. And then 2023,
- 2:27
I was investigating on GitHub some really cool projects, and I saw this project here called MLX. It's an array framework for Apple silicon. You can imagine PyTorch or TensorFlow for Apple silicon.
- 2:41
And I tried out the, the initial example, and I thought, "There's a future here." There's a future that was promised for all of us that all of the big companies like Meta and, um, Google could not really deliver because they were trying to optimize for scale for the cloud, and Apple did something different.
- 3:00
So then I started contributing to MLX. Three years later, we have over one point five million downloads, over four thousand models ported, and we work with some of the best frontier labs to deliver to you day zero support for all of your open source models.
- 3:15
You can e-imagine Gemma four, the latest Gemma four, we had day zero support for that on MLX, meaning you can run all of the best frontier open source models completely on your MacBook, on your iPhone or your iPad.
- 3:30
So let's start with vision, which for him was the-- kind of the biggest sensory deprivation. He cannot see. He cannot navigate the world. And I thought, "Well, the easiest way to do that is by giving him his vision back or giving him a system that can help him navigate the world."
- 3:49
And I think in, in 2021, I went into this hackathon, and I built these goggles with my team that could, like, tell you what's in front of you. But then MLX VLM became the second iteration of that, that allows you to do this not only via some weird, uh, glasses, but even on your iPhone.
- 4:07
You can now just pick up your phone, point it at something, and you'll be able to, you know, understand what's in front of you. And then you have omni models nowadays.
- 4:16
Uh, beyond just vision models, these are models that can also take in what? Audio. For my dad, in particular, typing is not really a reality, but he can speak, and with his speech, he can control the camera and understand what's in front of him, what he needs to do, and navigate the world.
- 4:34
So this is MLX VLM. Right now, if you use LM Studio, it's one of the main, uh, engines that powers LM Studio, powers Liquid AI models, and many other models out there.
- 4:45
But when you think on-device, you might think, "Well, that's weird. Uh, I cannot run really large models." Well, that's not true anymore. You can now run models of hundreds of billions of parameters, even on your, on your initial M1 MacBook.
- 5:01
There's a lot of, uh, improvements that the community has made that allows you to run even the largest and most, uh, abnormal models completely on-device. I have, uh, some examples that show that you can run models like Gemma 426B on an iPhone using your storage, and you can still get reasonable speeds.
- 5:23
Then after a year or [REDACTED:generic_id], I was also experimenting with, you know, how can we enable humans more and give them more accessibility, especially the ones that don't have all the sensories.
- 5:37
Um, and I thought audio is the next iteration. But it was for that and also for a very selfish reason. I wanted to be able to control my computer without being in front of my computer all the time.
- 5:48
What if I could just blurt a command and have my computer do it? This is more of like the Jarvis vision of the world, where you can just speak to your computer and have actions done for you.
- 5:58
We started off with text-to-speech, and then that on became Marvis, uh, one of our custom models that can generate audio in less than a hundred milliseconds. Um, then you, uh, we have, uh, speech-to-text, which allows you to speak to your computer and have it transcribed in real time.
- 6:14
So, for example, if you ever-- how many of you here use U- WhisperFlow? Or Super Whisper. Yeah, you can now vibe code that application. Just point Cloud Code or Codex into MLX Audio, ask, ask it to build it for you, and you'll have it in like ten minutes.
- 6:31
And then you also have speech-to-speech. So beyond the first [REDACTED:generic_id] capabilities, a big unlock is to have the computer speak back to you. So speech-to-speech is one of the core capabilities that we recently added, and we also support both Python and Swift.
- 6:46
Uh, we started off with Python because it's just much easier, it scales faster. But we also understand that native experiences matter, so with Swift, you can now build fully native applications enabled by audio intelligence as well as vision intelligence.
- 7:00
And on the, on the right side, your right side there, you have, uh, our modular pipeline that beyond just models that are speech-to-speech natively, you can actually chain a series of different capabilities to create a modular speech pipeline.
- 7:15
For, for instance, you can use... You can choose which a-automatic speech recognition model you want, you can choose which language model you want, and you can also choose which text-to-speech model you want.
- 7:26
And this way, you can create, uh, really custom and modular experiences that fit on every single hardware budget. So if you have a, a very simple M1 first generation Apple silicon, or even the latest, you can adjust that to your hardware.
- 7:42
But then you might think, "Well, speech-to-speech or text-to-speech is not that good. I've seen some videos, audios of, of speech-to-speech. That may be... I, I don't know. Is it-- Does it sound good?"
- 7:53
I can promise you that it does, and I'll show you a demo in a bit. So let's start with vision.
- 8:00
And, um, with vision, I have a couple of examples. The first one here is real-time, um, image analysis, so you can understand what's happening. It's a very simple command, and we'll make this even simpler.
- 8:14
Right now, it's in Python. It will come to Swift very soon. But if I run this command, it's gonna to run the RF detector model by, um, Roboflow. And as you can see, this is completely real time.
- 8:26
It's understanding where... And this is all running on my computer. I can actually just... To make sure that this is very clear. I don't know if I turn off the internet what's gonna happen.
- 8:36
But here it is continuously running. I can grab a glass, and it will also detect that. Of course, it's thinking it's wine, but, um, I'm sober. [laughs]
- 8:48
So this is running real time, completely on device, on my Mac. It can also run on your phone. And, uh, this is one example. I wanna show you another really cool example that you can do with this particular use case.
- 9:00
Have you ever tried, have you ever tried the, the... When you're in a meeting, you, you wanna blur the background. You, you know that Google does this and et cetera.
- 9:10
So now you can actually do this natively, and you can build this kind of experiences into your products. I'm not sure you can see that it's blurring the background, but it actually is blurring the background, and it's detecting my mask in real time and will detect other objects as well.
- 9:27
All right, so that's example number one. Number [REDACTED:generic_id] is you can run really, um, large models completely on device. Here's Gemma 4, was released, I think, a week ago, a couple...
- 9:41
Yeah, last week. It was released last week. And with MLX VLM, you just run mlx_vlm.chat_ui, and you pass in the model you want, and it should load that model and give you a simple interface for you to get started with Gradio.
- 9:55
Wait. Not sure what's happening. This is the problem with demo. Sometimes the gods don't want it to [laughs] to, to do the demo. Give me a second.
- 10:06
Okay, not sure what's happening. Mm. Okay. Well,
- 10:14
I need to quickly get something to close off.
- 10:20
Um. Mm, mm, mm. So okay, it stopped.
- 10:28
Let's start-- Let's try that again. Okay. It's trying to download some model files, but now I'll turn off the internet again just to show you that this is now running completely on device.
- 10:41
So now we have a chat window. I don't know if you can see this, or I have to zoom in more. And we can choose any image to analyze.
- 10:51
Let me see a very simple one. Okay. Okay, here. Describe this image in detail.
- 11:05
So here it is. It's saying that it's a profile of a name, man named [REDACTED:username], and it's picking up all the different details like my bio and et cetera.
- 11:17
And all of this is running on device. Um, it's using the GPU in this, uh, particular device. So this particular machine has like ninety-six gigabytes of VRAM, so I can run actually all of the models I showcased so far in real time, all of them at the same time.
- 11:35
So this is demo number [REDACTED:generic_id]. Let's now get back. I wanted to show you audio, but it seems like the Swift branch, there was something there I couldn't really figure out this morning.
- 11:49
But for the sake of examples, I will show you some of the really cool community use cases. Um, we have one here, one of the creators, uh, I didn't include his, his particular use case, but I'll show you a video in just a bit.
- 12:02
So let me try and get this in full screen
- 12:07
so that you can see better. One second. Here. Okay, so we are back. Um, the first example is grounded visual reasoning. You can use Gemma 4. You can use the model I just showcased, the RF detector, or any of our, um, um- ...
- 12:27
perception models, and you can create really cool experiences like
- 12:32
here. You can detect all the fires, you can ask it to detect particular items in the video, and this is all gonna happen on, completely on device without the use of internet.
- 12:42
And now you can have, for example, security systems that run completely on a MacBook, um, on your house, and you can analyze even your dash cam. I have a dash cam video, and I've had some really cool, or not cool, but crazy experiences.
- 12:57
And I usually use this kind of system here that I built to kind of analyze the video footage afterwards. And then you have example number [REDACTED:generic_id]. This is more of like a honorable mention, which is cartoons generated completely on device, um, using MLX Video, one of the most recent projects I, I'm running.
- 13:16
And here's one of the, I think, coolest, uh, video generated by one of our users. [laughs]
- 13:23
This is all generated on device with a simple text prompt. [whimsical music] [screaming] [laughs]
- 13:41
So... [upbeat music] [laughing] [screaming] And he did something very interesting, which is he chained... This, this is not like one video that he generated all at once.
- 13:57
What he did is, like, he chained a system - [laughs] ... that can continuously generate from the video. [laughs] So you can create a cohesive story, even though it was not, like, one-shotted.
- 14:08
And, uh, this particular, uh, system can run even on a MacBook with 16 gigabytes of, of VRAM. [screaming]
- 14:16
So yeah, pretty cool. He has a lot more, uh, on Twitter. I will put his, uh... Uh, like, I'll share, I will reshare on my Twitter, so if you, uh, check my Twitter, I'll reshare all of these videos.
- 14:27
And then I think one of the latest ones that I will show... But before that, let me show you one, one here from one of our participants in, in the...
- 14:39
on, on the floor. So if I go to Drive...
- 14:45
Let me see. In my Drive. Okay. Neywa Labs. You have...
- 14:59
Here. This is actually- It was a small voice and a small gesture ... by this gentleman here in front, Adrian. He builds this application called Locally, and using MLX Audio and Marvis TTS, he now gave that particular application the capability of speaking back to its users.
- 15:17
It was a small voice and a small gesture. Can you increase the volume? The [REDACTED:location] was a bar for professional expatriates. You could drink there for a week and never hear [REDACTED:generic_id] words in Japanese.
- 15:29
All right. So the, that is one of the examples where you can actually build really beautiful native experiences, and if you have a, a good touch of design, a killer application as well.
- 15:39
And finally, I think this is one of the most exciting parts of what I, where, where I think we are going next with on-device AI, which is
- 15:48
robotics. So last year I acquired a robot called, um, Ricci Mini, and the way that I power my Ricci Mini in particular is that I use MLX Audio, MLX Vision to give it all the capabilities or perception capabilities using its camera and audio, uh, input.
- 16:07
So here it is. It's also doing voice, real-time voice cloning of the original, uh, how do you call it? Iron Man JARVIS voice. So I hope you can hear this.
- 16:18
Hey, JARVIS.
- 16:26
Hey there. Great to see you. How's it going?
- 16:33
So you can chain and, and build a lot of really cool applications. This is just a start, and I hope that in the future you can understand, or today, from today, you can understand that you can build agents that can hear, see, and sound just like you or y- one of your loved ones today running on your
- 16:54
iPhone, iPad, Mac, or even your robot. Thank you. [clapping]
- 17:03
Questions?
- 17:07
So you already, like... Apple is like, uh, uh, promoting their neural engine a lot.
- 17:14
Yes.
- 17:14
And I've also tried it MLX.
- 17:16
Yeah.
- 17:16
But if you look at the, the usage of the neural engine, it's always like-
- 17:21
Yeah
- 17:21
... it's at zero.
- 17:22
That's a great question. So MLX uses the GPU, not the neural engine. For you to enable the neural engine, you need Core ML, and right now, uh, Core ML do- it does not really run well, like it's not an easy experience for developers.
- 17:37
I hope, uh, by WWDC Apple solves the private API issues, and when they do, we have some internal projects that can allow you to run a hybrid inference across both.
- 17:49
Yeah.
- 17:51
Okay.
- 17:51
It, it, it will be.
- 17:52
Yeah.
- 17:52
It will be. But we also think that they might be changing the n- the, the neural, uh, engine and putting some components into the GPU with, for example, the M5 series you can see that it already has some components of it, and w- we just don't know where d- what direction they are h- they are heading.
- 18:09
But it's exciting. Let's wait for WWDC. Questions? Yeah.
- 18:14
Is there a tool to see your GPU usage in real time?
- 18:18
Yes. So the easiest tool for you to get started is, um, mactop. If you run mactop It's going to pretty much show you all your usage. This is by, uh, Carson.
- 18:33
Um, he's a really cool dude. So you can see here pretty much what's happening, the GPU, CPU, um, and you can have this overlay across your, your, your device.
- 18:44
And if we do run inference, so let's say I start a new window, and if I can find that command.
- 18:52
Okay. So if I do start running inference on this... Let me put here for a bit,
- 19:00
and... Okay. You will see that the GPU now is, is gonna start moving up.
- 19:10
And if I say, "Hi, are you..." You see? The GPU is already moving up, and th- and this is one of the easiest ways that I found to, uh, track the performance.
- 19:29
Uh, any other questions? Yep.
- 19:32
Like, uh, you als- also mentioned omni models.
- 19:34
Yes.
- 19:35
Like, what's the one that you would recommend? Like, what is state-of-the-art at this point?
- 19:40
So you have a couple options. The first one is Gemma 4, the E version. They have this n- uh, Gemma 4 with the number E... Or the letter E, and then a number, E4, E2.
- 19:52
Those are omni models. They, uh, they take, um, image, audio, and text as input or any of the variation of the three. Um, and then you also have Qwen3-Omni, which is a much larger model, around thirty billion parameters, but you can also run on device.
- 20:08
Those are the top ones that I know.
- 20:10
What are your limit-- What-- I'm sorry. What are the key limitations of those models at the moment?
- 20:17
What are the key limitations? I think it depends on, on your particular use case. So there are certain things the models just cannot do. Th- you're not gonna get the performance of Claude 3 or, or 4 point, 4.6 Opus today.
- 20:29
But maybe in six months, these open source models will have that performance. So the experience should be kind of adjusted to your expe- the expectations of performance. That's the only thing I would say.
- 20:40
Outside of that, I don't see any limitations. You can run, uh, inference on hundreds of images in parallel. You can run inference on, uh, many, many documents, and you c- you now have context of up to a million thanks to a, a recent breakthrough that I made with TurboQuant.
- 20:56
So now you can actually serve, uh, one million contexts completely on device, depending on the size of the model and your hardware. But you can do that today.
- 21:06
Um, something else. I don't know the specifics about it. Like you had the, the TurboQuant paper that was released-
- 21:12
Yes
- 21:12
... recently by Google. And, and, and how far as I understood it, it was quite a general approach.
- 21:18
Mm-hmm.
- 21:18
Like, is it, like, are there efforts to, uh, like, generally implement it in this kind of- [laughs]
- 21:25
So I was one of the first people on the world to implement TurboQuant publicly.
- 21:30
Oh, interesting.
- 21:30
So, like, thirty minutes after the paper was out, I already ha- had implemented it, and I made this tweet at, like, three AM. I didn't know that it would go s- like, this viral.
- 21:40
But if you write... If you go to my profile and you write TurboQuant,
- 21:48
you'll be able to see there's this particular post here. So this was, like, twenty-fifth March, and pretty much the same day, but just was, like, midnight, and it got, like, seven hundred thousand views because of that.
- 22:00
So TurboQuant does work. Um, example is, like, the full model takes, uh, almost one gigabyte of, uh, uh, KV cache or RAM, and by using TurboQuant, you can reduce that by four X.
- 22:15
But is it, like-
- 22:16
Yeah
- 22:16
... the same quality or-
- 22:16
Uh, si- yeah, similar quality. A- as you can see, when it, when, when you see exact match, it means that it matches the performance of the, the responses of the full model.
- 22:24
And I also publi- publicize the, uh, full, um, results and performance. For example, here, uh, when you get to, like, three hundred thousand context, the performance almost doubles in terms of, like, throughput.
- 22:37
So yeah.
- 22:39
Super.
- 22:39
Yeah. Uh, there-- This is one of the many things that I try to do to enable on-device to go even further.
- 22:47
Yep. Any other questions? All right. Thank you. [audience clapping] [outro music]