# Why MLX

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/zTLJNHj0DeQ-why-mlx) · [Video](https://www.youtube.com/watch?v=zTLJNHj0DeQ)

## Why MLX: Building Agents That See, Hear, and Speak Locally

Local inference turns accessibility, voice control, and visual perception into applications that can work without a reliable connection, with MLX providing the building blocks on Apple silicon.

Basic familiarity with model inference and Python command-line tools will help with the demonstrations; no prior MLX experience is required.

### What happens when an agent cannot depend on the internet?

[0:17](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=17s)

Have you tried running AI on your phone or MacBook—and how good was the experience? The practical goal here is to deploy AI agents, including voice agents, entirely on those devices. Moving some workloads off subscriptions shifts their inference cost toward the hardware and electricity already available to the user. The route runs from the reasons for local execution through MLX, live demonstrations, and applications built by its community.

The need became personal in 2020. The speaker’s father lost his sight that year, while Apple introduced its first M1 hardware. A voracious reader could no longer access information as before, and his son promised to help him read again. His father lives in Africa, where unreliable connectivity and poor internet subscription options made a cloud-dependent solution a difficult fit. **On-device inference was an accessibility requirement:** assistance needed to remain available when a connection was not.

![Slide titled Apple Silicon beside an M1 chip image and a labeled component diagram.](https://ai.engineer/api/talk-essay-media/71843c3e32a18d2cfee4ea65378715a04949f4c666cce36a3b034636e6d372f5)

Apple Silicon: the M1 chip and its component layout.

### From an array framework to accessible perception

[2:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=147s)

In 2023, a GitHub search led the speaker to [MLX](https://github.com/ml-explore/mlx), an array framework for Apple silicon. Think of the role PyTorch or TensorFlow plays in a machine-learning application: arrays, operations, and the machinery for running models. The initial examples suggested a practical foundation for local intelligence, and he began contributing. The speaker reports more than 1.5 million downloads and more than 4,000 ported models three years later. He also describes collaboration with frontier labs and day-zero support for **[Gemma 4](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/)**, bringing supported open models to MacBooks, iPhones, and iPads.

![Neywa Labs slide titled MLX VLM + Audio with three metrics: 6 Frontier Labs partners, 1.5m+ downloads, and 4,000+ Models ported.](https://ai.engineer/api/talk-essay-media/462cf4b302f3eaa670ad599f62d20b67ea9d11c5bff33bdaac4d86e056aebc8e)

MLX VLM + Audio: 6 frontier lab partners, 1.5m+ downloads, and 4,000+ models ported.

Vision was the first application: help someone understand what is in front of them and navigate their surroundings. An earlier attempt, built with a team at a 2021 hackathon, used goggles that described nearby objects. [MLX-VLM](https://github.com/Blaizzy/mlx-vlm) became a second iteration of that idea, making the same kind of assistance possible through a phone camera. Point the phone at something and ask what it sees.

**Omni models** add audio input to that interaction. For a user who cannot readily type, speech can control the camera and request information about the scene. The speaker identifies MLX-VLM as an engine used by LM Studio and for Liquid AI models, then challenges the assumption that local inference necessarily means small models. He reports examples of models with hundreds of billions of parameters running on an early M1 MacBook, and Gemma 4 26B running on an iPhone using storage. These capacity claims come without a specified throughput or configuration; fitting a model and achieving an acceptable interactive experience remain separate questions.

### Assembling a voice agent for the available hardware

[5:23](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=323s)

Audio followed vision, both for accessibility and for a more everyday desire: controlling a computer without sitting in front of it. Speak a command and have the computer act—the JARVIS-style interaction. Work began with text-to-speech and **[Marvis](https://huggingface.co/Marvis-AI/marvis-tts-100m-v0.2)**. The speaker reports audio generation in less than 100 milliseconds, without specifying the model version, hardware, or latency measurement boundary. Speech-to-text supplies the other direction, transcribing the user’s speech in real time.

Wispr Flow and superwhisper illustrate the kind of dictation product these components can support. The speaker suggests pointing Claude Code or Codex at [MLX Audio](https://github.com/Blaizzy/mlx-audio) to build a comparable application in about ten minutes; that is an estimate, not a demonstrated build. Speech-to-speech then closes the conversational loop, allowing the computer to respond audibly. Python came first for development speed, followed by Swift support for native applications combining audio and vision.

There are two ways to assemble the conversation:

| Approach | How it works | Main choice |
| --- | --- | --- |
| Native speech-to-speech | A model handles the speech interaction | Select the integrated model |
| Modular pipeline | Speech recognition → language model → speech synthesis | Select each component separately |

In the modular version, **automatic speech recognition**, the **language model**, and **text-to-speech** are independent choices. That lets the application adapt to its hardware budget, from a first-generation M1 to a newer machine, without requiring the same model combination everywhere. The remaining question is whether the resulting speech sounds good enough for the experience being built.

### Detecting objects and masking a background

[8:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=480s)

The first live demonstration starts a camera-analysis application with a Python command. It runs Roboflow’s **[RF-DETR](https://github.com/roboflow/rf-detr)** locally, detecting objects as the scene changes. Swift support for this particular demonstration is described as forthcoming. During an attempt to show operation without an internet connection, processing continues. When the speaker lifts a glass, the detector identifies it as wine, prompting a joke about being sober. The visible demonstration runs on a Mac; phone execution is also described as possible.

The next variation applies perception to a familiar meeting feature: background blur. A real-time person mask separates the speaker from the surrounding scene so the background can be blurred while the foreground remains visible. The speaker notes that the blur may be difficult to see in the presentation, but the mechanism is the useful part: a local segmentation result becomes an input to a product’s rendering behavior.

### Opening a multimodal chat interface

[9:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=567s)

The next demonstration moves from live detection to a larger multimodal model. Gemma 4 is described as having arrived the previous week; its published release date is April 2, 2026. MLX-VLM’s `mlx_vlm.chat_ui` entry point accepts a model selection and opens a Gradio interface. With MLX-VLM’s chat dependencies installed, a small Python launcher can pass the chosen model through to that module:

```python
import subprocess
import sys

model = input("MLX-VLM model ID or local path: ").strip()
if not model:
    raise SystemExit("A model ID or local path is required.")

subprocess.run(
    [sys.executable, "-m", "mlx_vlm.chat_ui", "--model", model],
    check=True,
)
```

The interface provides a quick way to test a model before building a custom application around it.

The launch initially stalls. On retry, it attempts to download model files before the speaker switches the internet off again. That sequence exposes an operational distinction: **local inference still requires the model files to be available locally**. Once the chat window opens, he selects an image and enters “Describe this image in detail.” The response identifies a profile and describes details including the biography.

The speaker describes the demonstration machine as having 96 GB of memory and says it can run all the models shown so far simultaneously in real time. On Apple silicon, this is unified memory shared by CPU and GPU, rather than a separate dedicated VRAM pool; the demonstrated inference uses the GPU. The planned live audio demonstration does not follow: an unresolved problem in the Swift branch that morning prevents it, so community recordings provide the audio examples instead.

### Inspecting footage and continuing a generated story

[12:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=727s)

Grounded visual reasoning turns model output into a way to inspect video. Gemma 4, RF-DETR, or another perception model can look for fires or particular requested objects without sending the footage to a remote service. The displayed example asks to segment every jet and shows colored detection overlays on aircraft and their contrails. The proposed applications include home security running on a MacBook and analysis of dashcam footage.

![Slide titled Grounded Visual Reasoning using mlx-vlm showing jets and contrails with colored detection overlays and the prompt Segment every jet.](https://ai.engineer/api/talk-essay-media/ea9bb089965921d245146bf31bd27e520513b1b0a66ba981431283996edd0a5a)

Grounded visual reasoning demo highlighting jets in flight.

The speaker already uses a system like this to review dashcam footage after incidents. He then switches from analyzing video to generating it with [MLX Video](https://github.com/Blaizzy/mlx-video), showing a community-created cartoon that starts from a text prompt. The sequence is not generated as one long clip: successive generations continue from previous video, allowing a story to develop across multiple steps.

The speaker says this particular chained-generation system can run on a MacBook with 16 GB of unified memory. The model, settings, and generation time are not given. He promises to reshare more of the creator’s videos through his social profile; the demonstrated contribution is the chaining strategy that turns individual generations into a continuing sequence.

### Giving applications—and a robot—a voice

[14:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=867s)

The next recording, located in Drive under Neywa Labs, comes from Adrian, a participant in the audience who built an application called Locally. MLX Audio and Marvis TTS give it spoken output. The sample reads prose aloud, beginning with a small voice and gesture and continuing into a passage about the Chatsubo bar. It connects the speech components to a recognizable native reading experience. The speaker emphasizes that product design matters alongside the model integration.

Robotics extends the same combination of perception and speech into a physical device. The speaker acquired a Reachy Mini the previous year and describes using MLX Audio and MLX Vision to supply capabilities based on its camera and audio input. A recording demonstrates what he describes as real-time cloning of the Iron Man JARVIS voice. After a greeting, the robot responds: “Hey there. Great to see you. How’s it going?”

That is the prepared talk’s destination: agents that can hear, see, and speak in a familiar voice, running on an iPhone, iPad, Mac, or robot. The individual libraries become useful when they are composed into an interaction that serves the person using it. The questions that follow turn to the execution backend, observability, model limits, and the memory costs of longer contexts.

### Where inference runs, and how to observe it

[17:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1027s)

An audience member has noticed that Neural Engine utilization stays at zero while MLX is running. The explanation is that the demonstrated inference path uses the **GPU**, not the Neural Engine. MLX also supports CPU execution; it should not be understood as a GPU-only framework. For Neural Engine access, the speaker points to Core ML, while describing its developer experience and private-API issues as obstacles.

He describes internal work toward hybrid GPU and Neural Engine inference, contingent on Apple resolving API issues, and hopes for progress at WWDC. He also speculates about Neural Engine components moving into the GPU, citing the M5 generation, while acknowledging that Apple’s direction is unknown. These are prospective paths, not capabilities demonstrated in the session.

For observing the current path, he recommends [mactop](https://github.com/metaspartan/mactop/blob/main/README.md), whose author is Carsen Klock. It displays CPU and GPU activity, and the demonstration uses a monitoring view alongside the application. To reproduce the observation:

1. Open `mactop` and watch the utilization display.
2. Start the local inference application.
3. Submit a prompt and observe the GPU activity as the model responds.

In the demonstration, GPU usage rises after inference starts and a greeting is entered. This connects the application’s response to visible local compute activity.

### Choosing an omni model without overpromising

[19:32](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1172s)

Asked which omni models to try, the speaker names two families:

- **Gemma 4 E2B and E4B:** The edge variants accept image, audio, and text inputs in combinations. These are the published names of the variants referred to aloud as E2 and E4.
- **Qwen3-Omni:** A larger local option, described by the speaker as approximately 30 billion parameters, without identifying the precise variant or parameter accounting.

The distinction is useful for choosing a starting point: the input modalities must fit the application, and the model must fit the device.

The main limitation he identifies is capability relative to the task. Do not expect these local models to match the contemporary **Claude/Opus** models he invokes as a comparison. His suggestion that open models might reach that level within six months is a forecast. The application’s promises need to match the model’s current behavior.

He also describes inference over hundreds of images in parallel and many documents, then credits **[TurboQuant](https://arxiv.org/abs/2504.19874)** with making much longer contexts feasible. The speaker claims local operation with context lengths up to one million, depending on model size and hardware. Cache compression reduces the memory required to retain context; it does not automatically extend a model’s supported context window. For example, Google specifies 128K for Gemma 4’s edge variants and up to 256K for its larger variants, so the million-context claim needs a separate model and workload configuration.

### Reducing the cache cost of long contexts

[21:06](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1266s)

A final question asks whether Google’s general TurboQuant approach is being implemented. The speaker says he produced one of the earliest public implementations, about thirty minutes after encountering what he calls the paper’s release. Searching his profile brings up a March 25 post that he says attracted roughly 700,000 views after being posted late at night. The publication chronology needs one distinction: the paper was submitted to arXiv on April 28, 2025, while Google’s research announcement followed on March 24, 2026. The implementation account belongs to the speaker’s recollection of that publicity cycle.

The result he highlights is a reduction in **KV-cache memory**, the stored attention keys and values retained during inference. He describes a cache approaching 1 GB becoming roughly four times smaller. The inspected table gives a more precise view of the displayed comparison:

| Configuration | Exact match | Average cache, GB | Displayed reduction |
| --- | --- | --- | --- |
| Full | 6/6 | 0.703 | — |
| TurboQuant 2.5 | 6/6 | 0.143 | 4.9× smaller |
| TurboQuant 3.5 | 6/6 | 0.185 | 3.8× smaller |

The speaker interprets the exact matches as agreement with full-model responses and describes the quality as similar. The model, hardware, and six evaluation cases are not specified here, so those matches establish agreement for the displayed test, not general quality equivalence. A smaller KV cache also does not mean total application memory falls by the same factor: model weights and other allocations remain.

![Table with full, TurboQuant 2.5, and TurboQuant 3.5 rows. All show 6/6 exact match; average cache GB values are 0.703, 0.143, and 0.185, with reductions labeled 4.9x smaller and 3.8x smaller.](https://ai.engineer/api/talk-essay-media/087df97f2154e2ad1dd906c491182f2cef3513df0ee1516f2c3d847d6e6e7096)

TurboQuant results compare exact matches and average cache size against the full configuration.

The speaker reports almost doubled throughput at around 300,000 context in his published results. The workload and hardware behind that comparison are not specified in the discussion. It closes the session on a concrete engineering constraint: as context grows, cache memory and the cost of accessing it matter alongside model size. Reducing that burden is one way to extend what an on-device application can do.

## Resources

- [MLX](https://github.com/ml-explore/mlx) — Apple's machine-learning array framework with Python and Swift interfaces and shared CPU/GPU memory.
- [MLX-VLM](https://github.com/Blaizzy/mlx-vlm) — Local multimodal inference and fine-tuning, with a Gradio chat interface and KV-cache quantization.
- [Gemma 4 launch announcement](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) — Google introduces Gemma 4's model sizes, input modalities and context windows.
- [MLX Audio](https://github.com/Blaizzy/mlx-audio) — Speech synthesis, transcription and speech-to-speech tools for Apple silicon.
- [Marvis TTS v0.2](https://huggingface.co/Marvis-AI/marvis-tts-100m-v0.2) — Streaming speech model with MLX Audio examples for synthesis and reference-audio voice cloning.
- [RF-DETR](https://github.com/roboflow/rf-detr) — Roboflow's object detection and segmentation model project.
- [MLX Video](https://github.com/Blaizzy/mlx-video) — Image, video and audio generation models running on Mac through MLX.
- [mactop](https://github.com/metaspartan/mactop/blob/main/README.md) — Terminal monitoring of real-time Apple-silicon system activity.
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/abs/2504.19874) — Original research on online vector quantization, including KV-cache compression.
- [Google Research's TurboQuant explainer](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) — An explanation of TurboQuant, PolarQuant and QJL for memory-efficient vector processing.

## Slides

### AI Engineer Europe

[0:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=0s) [AI Engineer Europe logo in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/c0cd12394a262619e80b7d9e500e8aedbfe70826d25cd162d4a6bb65e981b2d8.webp)

AI Engineer
EUROPE

### Presenting sponsor

Google DeepMind is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=5s) [Presenting sponsor Google DeepMind on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/9357d8cfe1808f500415ecaa4cb4b3a0b234ca258b2d3e4cf9bc42182f8be10e.webp)

PRESENTING SPONSOR

Google DeepMind

### Platinum sponsors

Braintrust, WorkOS and OpenAI are credited as platinum sponsors.

[0:10](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=10s) [Platinum sponsors Braintrust, WorkOS and OpenAI on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/d84843a4af1873e96fba7013123a1e1f91304de10a6b029e740531d7f0dae9a1.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### Agenda

The talk makes the case for moving some subscription-based AI work on-device, then covers MLX, a demo, and community projects.

[0:46](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=46s) [Agenda listing why on-device, why MLX, a demo, and community projects; the speaker appears below the slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/76d5c9b61c5d51673acfb1e24cdaf67b6572788c14d26ec53a187eb6bf61c544.webp)

Agenda
• Why on-device?
• Why MLX?
• Demo
• Community projects

AIE
Google DeepMind
Engineer
EUROPE

### Apple Silicon

The speaker connects Apple’s M1 to his hope of helping his father read, citing limited internet access where his father lives as a reason to favor on-device computing.

[1:30](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=90s) [Apple Silicon slide showing an M1 chip beside a diagram of its CPU, GPU, memory, cache, and Neural Engine.](https://assets.ai.engineer/sites/aiecode2025/sha256/7976602517e01220a673c2fb64f016c36ddafeded7613d168fca3d706c68d869.webp)

Apple Silicon

M1

CPU
Fabric
GPU
DRAM
Neural Engine
DRAM
Cache

AIE
AI Engineer
EUROPE

### MLX

The speaker found MLX in 2023 and describes it as an array framework for Apple silicon, comparable to PyTorch or TensorFlow.

[2:22](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=142s) [White slide with the large letters MLX, with the X in light gray.](https://assets.ai.engineer/sites/aiecode2025/sha256/06eed1da832cf522ea12ad57e9d19e0859bb9cb0ff2997cd91e5b8df28a08452.webp)

MLX

Interface text:
AIE
AI Engineer EUROPE

[2:42](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=162s) [MLX wordmark on a white slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce84d5d15ca179b9812b6277899d78071cc93aa279fcc1495a816aa228026197.webp)

MLX

Interface text:
AIE
Engineering the future of AI
AI Engineer EUROPE

### MLX VLM + Audio

Neywa Labs reports 6 frontier lab partners, 1.5m+ downloads, and 4,000+ models ported. The speaker says these partnerships help deliver day-zero support for open-source models.

[3:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=180s) [Neywa Labs slide lists 6 frontier lab partners, 1.5 million-plus downloads, and 4,000-plus models ported.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b0627e9a39591e4c2ecd4a121ce4fc915dbe390ce091ecc0298cfb92c5f2489.webp)

Neywa Labs
MLX VLM + Audio

6
Frontier Labs partners

1.5m+
downloads

4,000+
Models ported

Interface text:
AIE
AI Engineer EUROPE
Engineering the future of AI

### Models are too large for local

A giraffe in a sports car illustrates the perceived size problem. The speaker says community improvements can run models with hundreds of billions of parameters on an initial M1 MacBook.

[4:43](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=283s) [Slide titled Models are too large for local, with an image of a giraffe in a sports car.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa98bef46deead772b94fe3a5f2e88b708defd3061791e32034a228b02890728.webp)

Models are too large for local

Interface text:
AIE
Engineering the future of AI
AI Engineer EUROPE

### MLX-Audio

MLX-Audio supports text-to-speech, speech-to-text and speech-to-speech in Python and Swift. Its modular pipeline chains a microphone through speech recognition, an LLM and text-to-speech to an audio player.

[5:21](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=321s) [MLX-Audio slide lists TTS, STT, STS, and Python and Swift support, with a microphone-to-audio-player pipeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/26f6cb7b56a455698cfabe0efa1cc39e7092fc1b6cac451b531cb88ecb6639fa.webp)

MLX-Audio

• TTS
• STT
• STS
• Python & Swift support

Microphone
↓
STT Model
↓
LLM
↓
TTS Model
↓
Audio Player

AIE
Engineering the future of AI

[5:45](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=345s) [MLX-Audio slide listing TTS, STT, STS, and Python and Swift support.](https://assets.ai.engineer/sites/aiecode2025/sha256/181d89c739e727b41693793cc5b8fd8c867199587793cc3b52d46169e8be8856.webp)

MLX-Audio

• TTS
• STT
• STS
• Python & Swift support

AIE
AI Enginee[illegible]
EUROPE

### When you hear AI generated voices

A meme shows a viewer abandoning an interesting YouTube video when an AI-generated voice-over starts. The speaker acknowledges doubts about voice quality and promises a later demo.

[7:42](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=462s) [A meme depicts someone abandoning an interesting YouTube video after encountering an AI-generated voice-over.](https://assets.ai.engineer/sites/aiecode2025/sha256/50d2b65e6994dd09a4af1e8a2eab6e81dc6a9b4e4202070d8c7b0d036a713476.webp)

When you hear AI generated voices

FINDING AN INTERESTING YOUTUBE VIDEO
AI GENERATED VOICE OVER

imgflip.com

Interface text:
AIE
Google DeepMind
AI Engineer Europe

### Running real-time vision

The presenter prepares a Python command to run RF-DETR for real-time image analysis; he says the demo runs on his computer.

[8:02](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=482s) [A terminal shows a command to run real-time object segmentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/84322d8ae7025efd671070333447cdc5e82ddbfadf42bfcc4fd7f749ff69d334.webp)

(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run python -m mlx_vlm.models.rfdetr.generate --task realtime --model mlx-community/rfdetr-seg-small-fp32

AIE
Engineering the future of AI
AI Engineer EUROPE

Interface text:
mlx-vlm
Problems  Output  Ports  Debug Console  GitLens  Terminal
zsh
⌘K to generate command
grounded_reasoning*  Blaizzy/mlx-vlm#985 needs reviewers  mlx-vlm  Cursor Tab  Screen Reader Optimized  Go Live

### Configuring background blur

After interrupting the camera run, the presenter adds a background-blur annotator. He says the demo blurs the background while detecting objects in real time on his Mac.

[8:58](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=538s) [A terminal shows an interrupted vision run and a new command configured for background blur.](https://assets.ai.engineer/sites/aiecode2025/sha256/736521809a37fcef01e0f957083bd9c1984f4c03867f3b515e224c8f9f7d4c85.webp)

AIE
AI Engineer EUROPE

(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run python -m mlx_vlm.models.rfdetr.generate --task realtime --model mlx-community/rfdetr-seg-small-fp32
Fetching 3 files: 100% | 3/3 [00:00<00:00, 49932.19it/s]
Download complete: : 0.00B [00:00, ?B/s]
Loading model from /Users/prince_canuma/.cache/huggingface/hub/models--mlx-community--rfdetr-seg-small-fp32/snapshots/a7a2375b4ca7b2ef0427af91b55d8176ca647b35...
Download complete: : 0.00B [00:00, ?B/s]
Source: 1280x720 @ 30fps (camera)
Press 'q' to quit
^CTraceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 818, in <module>
    main()
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 741, in main
    predictor.predict_realtime(
        source=source,
        annotator=annotator,
        task="segment" if has_seg else "detect",
    )
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 547, in predict_realtime
    result = self.predict_bgr(frame, score_threshold=threshold)
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 329, in predict_bgr
    mx.eval(*to_eval)
KeyboardInterrupt
(base) prince_canuma@MacBook-Pro-6 mlx-vlm %
(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run python -m mlx_vlm.models.rfdetr.generate --task realtime --model mlx-community/rfdetr-seg-small-fp32 --annotator blur+bg

Interface text:
mlx-vlm
Problems  Output  Ports  Debug Console  GitLens  Terminal
zsh
⌘K to generate command
grounded_reasoning*  Blaizzy/mlx-vlm#985 needs reviewers  mlx-vlm  Cursor Tab  Screen Reader Optimized  Go Live

### Launching the chat interface

After interrupting the real-time background blur demo, the presenter runs mlx_vlm.chat_ui with a Gemma 4 model to open a simple Gradio interface. The interface has not appeared yet.

[9:40](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=580s) [A terminal shows an interrupted real-time camera demo and a command to launch a model chat interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c95f3fdc8eea3b85f830325c0d7ddb8ef23326ec9eba34aacf3baab74a60792.webp)

File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 329, in predict_bgr
    mx.eval(*to_eval)
KeyboardInterrupt
(base) prince_canuma@MacBook-Pro-6 mlx-vlm %
(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run python -m mlx_vlm.models.rfdetr.generate --task realtime --model mlx-community/rfdetr-seg-small-fp32 --annotator blur+bg
Loading model from /Users/prince_canuma/.cache/huggingface/hub/models--mlx-community--rfdetr-seg-small-fp32/snapshots/a7a2375b4ca7b2ef0427af91b55d8176ca647b35...
Source: 1280x720 @ 30fps (camera)
Press 'q' to quit
^CTraceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 818, in <module>
    main()
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 741, in main
    predictor.predict_realtime(
        source=source,
        annotator=annotator,
        task="segment" if has_seg else "detect",
    )
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/rfdetr/generate.py", line 561, in predict_realtime
    out = annotator.annotate(frame, _to_annotator_result(result))
  File "/Users/prince_canuma/Projects/mlx-vlm/mlx_vlm/models/sam3/annotators.py", line 399, in annotate
    blurred = cv2.GaussianBlur(out, (k, k), 0)
KeyboardInterrupt
(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run mlx_vlm.chat_ui --model mlx-community/gemma-4-26b-a4b-it-4bit

AIE
Engineering the future of AI
AI Engineer EUROPE

Interface text:
mlx-vlm
Problems  Output  Ports  Debug Console  GitLens  Terminal
⌘K to generate command
grounded_reasoning*  Blaizzy/mlx-vlm#985 needs reviewers  mlx-vlm  Cursor Tab  Screen Reader Optimized  Go Live

### Fetching model files

After an interrupted network request, the presenter retries the MLX-VLM chat command. The terminal reports that fetching 10 model files reached 100%.

[10:31](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=631s) [Terminal shows model-file fetching after an interrupted network request.](https://assets.ai.engineer/sites/aiecode2025/sha256/07c99fe53e19b9000c0bb4434acdc4f788c74c66d943a4f0eaae69ac46783f11.webp)

File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
        pool_request.request
    )
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
File "/Users/prince_canuma/Projects/mlx-vlm/.venv/lib/python3.13/site-packages/httpcore/_backends/sync.py", line 208, in connect_tcp
    sock = socket.create_connection(
        address,
        timeout,
        source_address=source_address,
    )
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.13/socket.py", line 840, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.13/socket.py", line 977, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
KeyboardInterrupt
(base) prince_canuma@MacBook-Pro-6 mlx-vlm % uv run mlx_vlm.chat_ui --model mlx-community/gemma-4-26b-a4b-it-4bit
Fetching 10 files: 100%| | 10/10 [00:00<00:00, 12737.03it/s]
Fetching 10 files: 100%| | 10/10 [00:00<00:00, 10740.86it/s]
Download complete: : 0.00B [00:00, ?B/s] [00:00, ?B/s]
Download complete: : 0.00B [00:00, ?B/s] | 0/10 [00:00<?, ?it/s]

Engineering the future of AI
AIE
[illegible]gineer
[illegible]UROPE

Interface text:
Cursor  File  Edit  Selection  View  Go  Run  Terminal  Window  Help
Fri 10 Apr 11:50 AM
mlx-vlm
Problems  Output  Ports  Debug Console  GitLens  Terminal
⌘K to generate command
grounded_reasoning*  Blaizzy/mlx-vlm#985 needs reviewers  mlx-vlm  Cursor Tab  Screen Reader Optimized  Go Live

### Loaded MLX-VLM chat interface

The local chat window shows Gemma 4 loaded and an empty conversation. The speaker prepares to choose an image for analysis.

[10:40](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=640s) [A local chat interface shows a loaded model and an empty conversation.](https://assets.ai.engineer/sites/aiecode2025/sha256/c47193c364678d09bd058830ace9c58d9c672f47dea86d7fdbe2984b9bf11909.webp)

MLX-VLM Chat UI

Model
mlx-community/gemma-4-26b-a4b-it-4bit

Status
✓ Loaded

Generation Settings
Stop
New chat
Chatbot
Type a message...

Interface text:
MLX - Google Slides
mlx - Google Search
m1 silicon chip - Google Sear[illegible]
PyPI downloads stats for mlx
MLX-VLM Chat
127.0.0.1:7860
100%
Reset
Use via API · Built with Gradio · Settings
AIE
AI Engineer EUROPE

### Cool use cases for MLX

The speaker moves from the on-device demo to examples of MLX use cases created by the community.

[11:50](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=710s) [Section slide titled Cool use cases for MLX, shown in the slide editor.](https://assets.ai.engineer/sites/aiecode2025/sha256/73b19b81ed3fe6e51aa18131a155efdd9353a6a5e0347e1a7be94e0f71708495.webp)

Cool use cases for MLX

AI Engineer
EUROPE

AIE
Engineering the future of AI

Interface text:
MLX - Google Slides
mlx - Google Search
m1 silicon chip - Google Search
PyPI downloads stats for mlx
MLX-VLM Chat
MLX
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Trying to connect...

[12:03](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=723s) [Google Slides displays a title slide introducing MLX use cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/7968c90777446ff102a44862fda06deffdf8a7fd25058e501b88d0adafc4a022.webp)

Cool use cases for MLX

AI Engineer
EUROPE

AIE
Engineering the future of AI

Interface text:
MLX - Google Slides
mlx - Google Search
m1 silicon chip - Google Search
PyPI downloads stats for mlx
MLX-VLM Chat
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Trying to connect...

## Full transcript

[0:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=0s) [upbeat music] Good morning, everyone.

[0:17](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=17s) Uh, how many of you here have ever tried to run AI on your phone, on your MacBook?

[0:26](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=26s) How was that experience? Was good? More or less, yes. More or less? Yeah. Okay. So this talk today is for you. I'm going to show you how you can deploy and manage AI agents, or even voice agents if you will, completely on-device using MLX.

[0:45](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=45s) Uh, today's agenda, of course, we're gonna start with why on-device. A lot of you use Claude Code, uh, subscriptions, and many other subscriptions. I want to convince you today to offload some of that subscription completely on-device, and then all you need to pay is your energy bill.

[1:01](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=61s) Then I'm gonna talk about MLX, and I'll give you a small demo, and I will show you some of the amazing community projects that I've seen are built by the community using the projects that I will demonstrate today.

[1:13](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=73s) In 2020, something very magical and also weird happened. It was the best, yet also worst year of my life. It was a weird one. One-- on one side, my dad became blind, and on the other side, Apple released one of the most powerful, um, chips for on-device intelligence ever.

[1:38](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=98s) And I remember I was talking to my dad, and I said, "Hey,

[1:44](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=104s) I promise you that I'll get you back to reading." He's one of the most voracious readers I know, and losing his sight was one of the main things that kind of bummed him out.

[1:53](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=113s) He couldn't no, no longer consume information. And I made this really weird promise. I said, "Hey, I'm going, I'm going to f-fix this some way somehow." And at the same time, this happened.

[2:06](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=126s) And I thought, "Hmm, compute on the cloud doesn't necessarily solve all of these use cases," because my dad lives in Africa, and there we don't have internet as easy as we have here, or the subscription plan, plans there are really, really bad.

[2:21](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=141s) So I thought on-device is the future. And then 2023,

[2:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=147s) I was investigating on GitHub some really cool projects, and I saw this project here called MLX. It's an array framework for Apple silicon. You can imagine PyTorch or TensorFlow for Apple silicon.

[2:41](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=161s) And I tried out the, the initial example, and I thought, "There's a future here." There's a future that was promised for all of us that all of the big companies like Meta and, um, Google could not really deliver because they were trying to optimize for scale for the cloud, and Apple did something different.

[3:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=180s) So then I started contributing to MLX. Three years later, we have over one point five million downloads, over four thousand models ported, and we work with some of the best frontier labs to deliver to you day zero support for all of your open source models.

[3:15](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=195s) You can e-imagine Gemma four, the latest Gemma four, we had day zero support for that on MLX, meaning you can run all of the best frontier open source models completely on your MacBook, on your iPhone or your iPad.

[3:30](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=210s) So let's start with vision, which for him was the-- kind of the biggest sensory deprivation. He cannot see. He cannot navigate the world. And I thought, "Well, the easiest way to do that is by giving him his vision back or giving him a system that can help him navigate the world."

[3:49](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=229s) And I think in, in 2021, I went into this hackathon, and I built these goggles with my team that could, like, tell you what's in front of you. But then MLX VLM became the second iteration of that, that allows you to do this not only via some weird, uh, glasses, but even on your iPhone.

[4:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=247s) You can now just pick up your phone, point it at something, and you'll be able to, you know, understand what's in front of you. And then you have omni models nowadays.

[4:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=256s) Uh, beyond just vision models, these are models that can also take in what? Audio. For my dad, in particular, typing is not really a reality, but he can speak, and with his speech, he can control the camera and understand what's in front of him, what he needs to do, and navigate the world.

[4:34](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=274s) So this is MLX VLM. Right now, if you use LM Studio, it's one of the main, uh, engines that powers LM Studio, powers Liquid AI models, and many other models out there.

[4:45](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=285s) But when you think on-device, you might think, "Well, that's weird. Uh, I cannot run really large models." Well, that's not true anymore. You can now run models of hundreds of billions of parameters, even on your, on your initial M1 MacBook.

[5:01](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=301s) There's a lot of, uh, improvements that the community has made that allows you to run even the largest and most, uh, abnormal models completely on-device. I have, uh, some examples that show that you can run models like Gemma 4 26B on an iPhone using your storage, and you can still get reasonable speeds.

[5:23](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=323s) Then after a year or [REDACTED:generic_id], I was also experimenting with, you know, how can we enable humans more and give them more accessibility, especially the ones that don't have all the sensories.

[5:37](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=337s) Um, and I thought audio is the next iteration. But it was for that and also for a very selfish reason. I wanted to be able to control my computer without being in front of my computer all the time.

[5:48](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=348s) What if I could just blurt a command and have my computer do it? This is more of like the Jarvis vision of the world, where you can just speak to your computer and have actions done for you.

[5:58](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=358s) We started off with text-to-speech, and then that one became Marvis, uh, one of our custom models that can generate audio in less than a hundred milliseconds. Um, then you, uh, we have, uh, speech-to-text, which allows you to speak to your computer and have it transcribed in real time.

[6:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=374s) So, for example, if you ever-- how many of you here use U- Wispr Flow? Or superwhisper. Yeah, you can now vibe code that application. Just point Claude Code or Codex into MLX Audio, ask, ask it to build it for you, and you'll have it in like ten minutes.

[6:31](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=391s) And then you also have speech-to-speech. So beyond the first [REDACTED:generic_id] capabilities, a big unlock is to have the computer speak back to you. So speech-to-speech is one of the core capabilities that we recently added, and we also support both Python and Swift.

[6:46](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=406s) Uh, we started off with Python because it's just much easier, it scales faster. But we also understand that native experiences matter, so with Swift, you can now build fully native applications enabled by audio intelligence as well as vision intelligence.

[7:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=420s) And on the, on the right side, your right side there, you have, uh, our modular pipeline that beyond just models that are speech-to-speech natively, you can actually chain a series of different capabilities to create a modular speech pipeline.

[7:15](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=435s) For, for instance, you can use... You can choose which a-automatic speech recognition model you want, you can choose which language model you want, and you can also choose which text-to-speech model you want.

[7:26](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=446s) And this way, you can create, uh, really custom and modular experiences that fit on every single hardware budget. So if you have a, a very simple M1 first generation Apple silicon, or even the latest, you can adjust that to your hardware.

[7:42](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=462s) But then you might think, "Well, speech-to-speech or text-to-speech is not that good. I've seen some videos, audios of, of speech-to-speech. That may be... I, I don't know. Is it-- Does it sound good?"

[7:53](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=473s) I can promise you that it does, and I'll show you a demo in a bit. So let's start with vision.

[8:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=480s) And, um, with vision, I have a couple of examples. The first one here is real-time, um, image analysis, so you can understand what's happening. It's a very simple command, and we'll make this even simpler.

[8:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=494s) Right now, it's in Python. It will come to Swift very soon. But if I run this command, it's gonna to run the RF-DETR model by, um, Roboflow. And as you can see, this is completely real time.

[8:26](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=506s) It's understanding where... And this is all running on my computer. I can actually just... To make sure that this is very clear. I don't know if I turn off the internet what's gonna happen.

[8:36](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=516s) But here it is continuously running. I can grab a glass, and it will also detect that. Of course, it's thinking it's wine, but, um, I'm sober. [laughs]

[8:48](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=528s) So this is running real time, completely on device, on my Mac. It can also run on your phone. And, uh, this is one example. I wanna show you another really cool example that you can do with this particular use case.

[9:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=540s) Have you ever tried, have you ever tried the, the... When you're in a meeting, you, you wanna blur the background. You, you know that Google does this and et cetera.

[9:10](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=550s) So now you can actually do this natively, and you can build this kind of experiences into your products. I'm not sure you can see that it's blurring the background, but it actually is blurring the background, and it's detecting my mask in real time and will detect other objects as well.

[9:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=567s) All right, so that's example number one. Number [REDACTED:generic_id] is you can run really, um, large models completely on device. Here's Gemma 4, was released, I think, a week ago, a couple...

[9:41](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=581s) Yeah, last week. It was released last week. And with MLX VLM, you just run mlx_vlm.chat_ui, and you pass in the model you want, and it should load that model and give you a simple interface for you to get started with Gradio.

[9:55](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=595s) Wait. Not sure what's happening. This is the problem with demo. Sometimes the gods don't want it to [laughs] to, to do the demo. Give me a second.

[10:06](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=606s) Okay, not sure what's happening. Mm. Okay. Well,

[10:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=614s) I need to quickly get something to close off.

[10:20](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=620s) Um. Mm, mm, mm. So okay, it stopped.

[10:28](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=628s) Let's start-- Let's try that again. Okay. It's trying to download some model files, but now I'll turn off the internet again just to show you that this is now running completely on device.

[10:41](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=641s) So now we have a chat window. I don't know if you can see this, or I have to zoom in more. And we can choose any image to analyze.

[10:51](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=651s) Let me see a very simple one. Okay. Okay, here. Describe this image in detail.

[11:05](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=665s) So here it is. It's saying that it's a profile of a name, man named [REDACTED:username], and it's picking up all the different details like my bio and et cetera.

[11:17](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=677s) And all of this is running on device. Um, it's using the GPU in this, uh, particular device. So this particular machine has like ninety-six gigabytes of VRAM, so I can run actually all of the models I showcased so far in real time, all of them at the same time.

[11:35](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=695s) So this is demo number [REDACTED:generic_id]. Let's now get back. I wanted to show you audio, but it seems like the Swift branch, there was something there I couldn't really figure out this morning.

[11:49](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=709s) But for the sake of examples, I will show you some of the really cool community use cases. Um, we have one here, one of the creators, uh, I didn't include his, his particular use case, but I'll show you a video in just a bit.

[12:02](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=722s) So let me try and get this in full screen

[12:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=727s) so that you can see better. One second. Here. Okay, so we are back. Um, the first example is grounded visual reasoning. You can use Gemma 4. You can use the model I just showcased, the RF-DETR, or any of our, um, um- ...

[12:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=747s) perception models, and you can create really cool experiences like

[12:32](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=752s) here. You can detect all the fires, you can ask it to detect particular items in the video, and this is all gonna happen on, completely on device without the use of internet.

[12:42](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=762s) And now you can have, for example, security systems that run completely on a MacBook, um, on your house, and you can analyze even your dash cam. I have a dash cam video, and I've had some really cool, or not cool, but crazy experiences.

[12:57](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=777s) And I usually use this kind of system here that I built to kind of analyze the video footage afterwards. And then you have example number [REDACTED:generic_id]. This is more of like a honorable mention, which is cartoons generated completely on device, um, using MLX Video, one of the most recent projects I, I'm running.

[13:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=796s) And here's one of the, I think, coolest, uh, video generated by one of our users. [laughs]

[13:23](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=803s) This is all generated on device with a simple text prompt. [whimsical music] [screaming] [laughs]

[13:41](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=821s) So... [upbeat music] [laughing] [screaming] And he did something very interesting, which is he chained... This, this is not like one video that he generated all at once.

[13:57](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=837s) What he did is, like, he chained a system - [laughs] ... that can continuously generate from the video. [laughs] So you can create a cohesive story, even though it was not, like, one-shotted.

[14:08](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=848s) And, uh, this particular, uh, system can run even on a MacBook with 16 gigabytes of, of VRAM. [screaming]

[14:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=856s) So yeah, pretty cool. He has a lot more, uh, on Twitter. I will put his, uh... Uh, like, I'll share, I will reshare on my Twitter, so if you, uh, check my Twitter, I'll reshare all of these videos.

[14:27](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=867s) And then I think one of the latest ones that I will show... But before that, let me show you one, one here from one of our participants in, in the...

[14:39](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=879s) on, on the floor. So if I go to Drive...

[14:45](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=885s) Let me see. In my Drive. Okay. Neywa Labs. You have...

[14:59](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=899s) Here. This is actually- It was a small voice and a small gesture ... by this gentleman here in front, Adrian. He builds this application called Locally, and using MLX Audio and Marvis TTS, he now gave that particular application the capability of speaking back to its users.

[15:17](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=917s) It was a small voice and a small gesture. Can you increase the volume? The [REDACTED:location] was a bar for professional expatriates. You could drink there for a week and never hear [REDACTED:generic_id] words in Japanese.

[15:29](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=929s) All right. So the, that is one of the examples where you can actually build really beautiful native experiences, and if you have a, a good touch of design, a killer application as well.

[15:39](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=939s) And finally, I think this is one of the most exciting parts of what I, where, where I think we are going next with on-device AI, which is

[15:48](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=948s) robotics. So last year I acquired a robot called, um, Reachy Mini, and the way that I power my Ricci Mini in particular is that I use MLX Audio, MLX Vision to give it all the capabilities or perception capabilities using its camera and audio, uh, input.

[16:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=967s) So here it is. It's also doing voice, real-time voice cloning of the original, uh, how do you call it? Iron Man JARVIS voice. So I hope you can hear this.

[16:18](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=978s) Hey, JARVIS.

[16:26](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=986s) Hey there. Great to see you. How's it going?

[16:33](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=993s) So you can chain and, and build a lot of really cool applications. This is just a start, and I hope that in the future you can understand, or today, from today, you can understand that you can build agents that can hear, see, and sound just like you or y- one of your loved ones today running on your

[16:54](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1014s) iPhone, iPad, Mac, or even your robot. Thank you. [clapping]

[17:03](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1023s) Questions?

[17:07](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1027s) So you already, like... Apple is like, uh, uh, promoting their neural engine a lot.

[17:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1034s) Yes.

[17:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1034s) And I've also tried it MLX.

[17:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1036s) Yeah.

[17:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1036s) But if you look at the, the usage of the neural engine, it's always like-

[17:21](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1041s) Yeah

[17:21](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1041s) ... it's at zero.

[17:22](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1042s) That's a great question. So MLX uses the GPU, not the neural engine. For you to enable the neural engine, you need Core ML, and right now, uh, Core ML do- it does not really run well, like it's not an easy experience for developers.

[17:37](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1057s) I hope, uh, by WWDC Apple solves the private API issues, and when they do, we have some internal projects that can allow you to run a hybrid inference across both.

[17:49](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1069s) Yeah.

[17:51](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1071s) Okay.

[17:51](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1071s) It, it, it will be.

[17:52](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1072s) Yeah.

[17:52](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1072s) It will be. But we also think that they might be changing the n- the, the neural, uh, engine and putting some components into the GPU with, for example, the M5 series you can see that it already has some components of it, and w- we just don't know where d- what direction they are h- they are heading.

[18:09](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1089s) But it's exciting. Let's wait for WWDC. Questions? Yeah.

[18:14](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1094s) Is there a tool to see your GPU usage in real time?

[18:18](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1098s) Yes. So the easiest tool for you to get started is, um, mactop. If you run mactop It's going to pretty much show you all your usage. This is by, uh, Carson.

[18:33](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1113s) Um, he's a really cool dude. So you can see here pretty much what's happening, the GPU, CPU, um, and you can have this overlay across your, your, your device.

[18:44](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1124s) And if we do run inference, so let's say I start a new window, and if I can find that command.

[18:52](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1132s) Okay. So if I do start running inference on this... Let me put here for a bit,

[19:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1140s) and... Okay. You will see that the GPU now is, is gonna start moving up.

[19:10](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1150s) And if I say, "Hi, are you..." You see? The GPU is already moving up, and th- and this is one of the easiest ways that I found to, uh, track the performance.

[19:29](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1169s) Uh, any other questions? Yep.

[19:32](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1172s) Like, uh, you als- also mentioned omni models.

[19:34](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1174s) Yes.

[19:35](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1175s) Like, what's the one that you would recommend? Like, what is state-of-the-art at this point?

[19:40](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1180s) So you have a couple options. The first one is Gemma 4, the E version. They have this n- uh, Gemma 4 with the number E... Or the letter E, and then a number, E4, E2.

[19:52](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1192s) Those are omni models. They, uh, they take, um, image, audio, and text as input or any of the variation of the three. Um, and then you also have Qwen3-Omni, which is a much larger model, around thirty billion parameters, but you can also run on device.

[20:08](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1208s) Those are the top ones that I know.

[20:10](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1210s) What are your limit-- What-- I'm sorry. What are the key limitations of those models at the moment?

[20:17](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1217s) What are the key limitations? I think it depends on, on your particular use case. So there are certain things the models just cannot do. Th- you're not gonna get the performance of Claude 3 or, or 4 point, 4.6 Opus today.

[20:29](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1229s) But maybe in six months, these open source models will have that performance. So the experience should be kind of adjusted to your expe- the expectations of performance. That's the only thing I would say.

[20:40](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1240s) Outside of that, I don't see any limitations. You can run, uh, inference on hundreds of images in parallel. You can run inference on, uh, many, many documents, and you c- you now have context of up to a million thanks to a, a recent breakthrough that I made with TurboQuant.

[20:56](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1256s) So now you can actually serve, uh, one million contexts completely on device, depending on the size of the model and your hardware. But you can do that today.

[21:06](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1266s) Um, something else. I don't know the specifics about it. Like you had the, the TurboQuant paper that was released-

[21:12](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1272s) Yes

[21:12](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1272s) ... recently by Google. And, and, and how far as I understood it, it was quite a general approach.

[21:18](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1278s) Mm-hmm.

[21:18](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1278s) Like, is it, like, are there efforts to, uh, like, generally implement it in this kind of- [laughs]

[21:25](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1285s) So I was one of the first people on the world to implement TurboQuant publicly.

[21:30](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1290s) Oh, interesting.

[21:30](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1290s) So, like, thirty minutes after the paper was out, I already ha- had implemented it, and I made this tweet at, like, three AM. I didn't know that it would go s- like, this viral.

[21:40](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1300s) But if you write... If you go to my profile and you write TurboQuant,

[21:48](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1308s) you'll be able to see there's this particular post here. So this was, like, twenty-fifth March, and pretty much the same day, but just was, like, midnight, and it got, like, seven hundred thousand views because of that.

[22:00](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1320s) So TurboQuant does work. Um, example is, like, the full model takes, uh, almost one gigabyte of, uh, uh, KV cache or RAM, and by using TurboQuant, you can reduce that by four X.

[22:15](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1335s) But is it, like-

[22:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1336s) Yeah

[22:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1336s) ... the same quality or-

[22:16](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1336s) Uh, si- yeah, similar quality. A- as you can see, when it, when, when you see exact match, it means that it matches the performance of the, the responses of the full model.

[22:24](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1344s) And I also publi- publicize the, uh, full, um, results and performance. For example, here, uh, when you get to, like, three hundred thousand context, the performance almost doubles in terms of, like, throughput.

[22:37](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1357s) So yeah.

[22:39](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1359s) Super.

[22:39](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1359s) Yeah. Uh, there-- This is one of the many things that I try to do to enable on-device to go even further.

[22:47](https://www.youtube.com/watch?v=zTLJNHj0DeQ&t=1367s) Yep. Any other questions? All right. Thank you. [audience clapping] [outro music]
