AI Engineer Europe 2026
Running Gemma 4 On-Device: 40 Tokens/s on iPhone with MLX
Read the talk
Running Gemma 4 on an iPhone with MLX
A native chatbot shows how MLX packages, compatible model weights, and quantization turn local iPhone inference into a practical app experience.
From a talk by Adrien Grondin
Before you start: Basic familiarity with Swift app development and downloadable language-model weights will help with the integration example.
A chatbot that runs on the phone
How do you run Gemma 4 on an iPhone? Adrien Grondin’s starting point is Locally AI, the native chatbot he develops. It runs models on the device through MLX and also offers Apple Foundation alongside compatible downloadable models. The opening app views show the three pieces users encounter: a conversation, Apple Foundation, and model management.
Smaller variants of Google DeepMind’s Gemma 4 family make this phone experience possible. Grondin has shown the app running them in social posts; underneath the interface is MLX, Apple’s framework optimized for Apple silicon. The same ecosystem extends beyond the phone: Locally AI also runs on iPad and macOS.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the integration package
Gemma is one option; Qwen and Hugging Face’s SmolLM are other model families Grondin points to. For a native iOS, iPadOS, or macOS application, the integration entry point is MLX Swift LM, packaged as mlx-swift-lm. It supplies the machinery to download and run compatible models. Grondin leaves the detailed implementation to the developer and their coding agent, describing the API as straightforward. He estimates that an iOS app running a model can be assembled in less than ten minutes.
The surrounding ecosystem covers more than text chat:
- Vision and omni models: Prince’s MLX-VLM provides another route for Python and Mac work.
- Audio: MLX-Audio covers speech workloads, including text-to-speech and speech-to-speech.
- Image and video generation: MLX-Video extends the family of tools to generation workloads.
These projects explain the breadth of MLX’s ecosystem; the phone walkthrough remains focused on running a suitably sized language model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose compatible weights, then pass their ID
Installing the framework does not make every model runnable. The next dependency is a compatible set of weights. Grondin directs developers to MLX Community on Hugging Face, where contributors publish both full-size and quantized versions. At the time of the talk, he estimates roughly 4,000–5,000 uploaded models and says quantizations can appear about 30 minutes after a lab releases a model. Those are his observations about community activity, rather than catalog counts or turnaround guarantees.
His iPhone example is Gemma 4 E2B. The catalog offers multiple representations of its weights: he mentions BF16, MXFP4, and 5-bit and 6-bit variants. The model family identifies what you want to run; the particular repository identifies which representation you will download.
The integration sequence is short:
- Add
mlx-swift-lmto the application. - Select a compatible model repository from MLX Community.
- Pass that repository’s ID to the framework, whose Hugging Face integration downloads the model.
For example, the available 4-bit Gemma 4 E2B checkpoint gives a concrete identifier to put into a Swift model configuration:
swift
import MLXLMCommon
let modelConfiguration = ModelConfiguration(
id: "mlx-community/gemma-4-e2b-it-4bit"
)
This configuration selects the repository; the application’s loading and generation code then uses it. The significant handoff is the model ID, rather than a manually assembled download URL or a separate weight-transfer step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit the model to the workload
For the proposed iPhone use, full-size weights are generally too large. Quantization reduces the precision used to represent weights, making a smaller representation available at a potential cost to output quality. Grondin initially mentions 3–8 bits, then settles on 4–8 bits as his practical range.
| Weight precision | Grondin’s guidance |
|---|---|
| Below 4-bit | Output quality often suffers too much |
| 4-bit | His recommended lower bound |
| 8-bit | An option for very small models |
The choice therefore depends on model size as well as precision: the largest precision setting is most plausible when the model itself is small.
A chatbot is not the only useful destination. Grondin describes a Liquid model with 350 million parameters running through Shortcuts in his app. At that size, he uses the model for fast, efficient text processing and automation. He does not identify the exact Liquid checkpoint.
For the larger chat example, Grondin reports about 40 tokens per second for Gemma 4 E2B quantized to 4-bit on the latest iPhone. He does not name the iPhone model or provide a reproducible benchmark configuration, so this figure describes his reported setup rather than a hardware-wide expectation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What offline generation looks like
The demonstration turns that throughput into a visible interaction. Grondin describes the phone as running live and offline, streaming its response as it generates. The displayed answer is relatively long; the selected view shows a multi-day Paris itinerary in the chat interface. Streaming lets the user begin reading while generation continues.
A different interface could wait and present the completed answer. Grondin suggests a four-second wait as an example of that design choice, not a measured completion time for every response. The appropriate presentation depends on how much output the task needs.
Older iPhones can also run these models, although Grondin says they should not be expected to reach the latest-phone figure. He offers 20 tokens per second as an illustrative rate that would still be useful for many applications, rather than reporting a measurement from a named older device. The practical question is whether generation is fast enough for the particular interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The download is still part of the product
To try the experience, Grondin points attendees to an App Store QR code and describes Locally AI as free to use. Installing the app is only the first step: users also need the selected model’s weights. Grondin puts typical model downloads at roughly 1–3 GB, depending on the model.
He identifies model size as the biggest remaining barrier. Local generation removes the need to send each request to a hosted model, but it still asks users to download a substantial asset before using it. Smaller, more capable models and improving iPhone hardware make that tradeoff progressively easier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From phone inference to a local server
Grondin expects successive iPhone generations to improve usability further. He then connects the phone app to the desktop ecosystem, announcing that LM Studio has acquired Locally AI. LM Studio’s blog dates the joining announcement to April 8, 2026.
LM Studio provides a desktop interface for downloading models from Hugging Face and running them locally. It supports both llama.cpp and MLX, allowing users to compare the engines. Its model browser shows the selection and download side of that workflow.
The next step is to expose the running model through a local server and connect another application to it. Grondin describes OpenAI-style and Anthropic-style interfaces, including streaming. This changes where inference lives: a phone app can embed the runtime directly, while an application using LM Studio sends requests to a separate local serving process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tool calling and structured generation are separate capabilities
An audience question asks whether the stack supports tool calling. Grondin says yes, while drawing a separate boundary around structured generation.
| Capability | Status described in the talk |
|---|---|
| Tool calling | Supported by mlx-swift-lm |
| Structured generation | Not yet supported directly |
| Structured-generation add-ons | Packages attempting to add support |
These describe the support boundary at the time of the talk, not a promise about every later package version. Tool calling lets a model request an operation in another system; it does not by itself establish that arbitrary output will be constrained to a required structure.
Grondin mentions Hugging Face’s work on additional packages without naming one. His concrete recommendation is that mlx-swift-lm already supports the tool-calling path for connecting to other systems, and he says models have become substantially better at selecting and using tools compared with a year earlier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build an app or use the curated one
The final questions separate two entry points that can otherwise sound like one setup process:
- Developers: Install
mlx-swift-lmin their own app and obtain compatible model weights from Hugging Face. - Users: Download Locally AI from the App Store and try the experience without setting up a development environment.
The ready-made app removes the integration work, while still requiring the model download discussed earlier.
That convenience also includes a selection boundary. Locally AI does not offer every open-source model: Grondin curates the catalog and checks that included models run correctly on the iPhone. A model’s availability is not the same as its suitability for a phone. The app’s final responsibility is to turn compatible weights and a capable runtime into choices that actually work on the user’s device.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Swift package for integrating language and vision-language models into Apple-platform applications.
Google's model card explains E2B's architecture, modalities, effective parameter count and usage.
MLX Community's 4-bit checkpoint with model-ID loading examples.
Run and fine-tune vision-language and omni models on Mac using MLX.
Speech synthesis, recognition and speech-to-speech tools built on MLX.
Image, video and audio generation model tooling for Mac.
Further reading
Configure local models for Claude Code through LM Studio's Messages endpoint, including streaming and tool use.
A compact 350M-parameter model designed for edge and on-device deployment.
Read the complete timestamped transcript
- 0:00
[upbeat music] Okay.
- 0:15
Hello everybody. [clears throat] I'm going to show you today how to run Gemma 4 on iPhone with, uh, MLX. So first, let's introduce myself. I'm Adrien. You can find, uh, my Twitter if you want to learn more about, uh, all on-device things.
- 0:29
I'm the developer of, um, Locally AI, so maybe you have already seen, uh, the, the app. So Locally AI is a chatbot that allow you to run on-device models on your iPhone with, uh, MLX.
- 0:42
So I will just, uh, go through ML- what is MLX in a, in a few, few seconds. Basically, uh, lo- as I said, it's a, it's a chatbot. It's fully native.
- 0:51
You can also chat with, uh, Apple Foundation now with it. And the, and many models like, um, that are compatible with MLX and one of this model is, uh, Gemma 4.
- 1:02
So [clears throat] basically, Gemma 4, uh, by Google DeepMind have a lot of model and some of them like can run on iPhone, like the smaller ones and they are pretty great.
- 1:13
Maybe you have seen on, uh, on Twitter one of the post I've made that, uh, I should... I demo it running, uh, in the app, uh, on iPhone. It's really fast.
- 1:20
It runs really well on MLX and behind the app, so [clears throat] it's using MLX. MLX is a framework made by Apple that is optimized for Apple silicon, so mainly, mainly the, [clears throat] the chip in iPhone but also the chips on the, on the Mac.
- 1:36
The, the app Locally is available on iPad. It works also very well on this device and macOS and, uh, everything is, uh, is built to be as optimized as possible on the, on these, uh, devices.
- 1:49
So if you want, if you want to run on iPhone, um, a s- large model, so Gemma works well but you have a lot of, uh, model that you can run also.
- 1:58
The Qwen model and the, um, and the small LLM model from the Hugging Face. You-- the place you, you will want to go is GitHub and go to the repo mlx-swift-lm.
- 2:07
I won't go into detail how you implement the repo. It's... I think I will let your agent in, uh, implement that, uh, for you but it's one repo that you need to install if you're developing, um, iOS or macOS or iPadOS app.
- 2:19
And you can use, um, you can use that to simply download the model and then run it. The API is very straightforward, very simple to, to implement. In less than ten minutes, you can have an iOS app with a model that is running on your, on your device.
- 2:32
That's very simple, uh, to do as, uh, as mentioned. MLX, uh, so this is mlx-swift-lm but if you're more into Python apps or macOS app you can also run mlx-vllm from, uh, from Prince.
- 2:44
Maybe you have, uh, seen him like he's doing, um, on device for audio with MLX Audio and visual model with, uh, MLX VLM and also MLX Video to, to run, um, to run, uh, image, uh, image generation model or image, uh, or video generation model.
- 3:00
MLX it's, uh, there's... The ecosystem is getting, uh, bigger or it's getting bigger. It's g- it's really great right now. You can do pretty much everything like, uh, omni models or as I said, text-to-speech, speech to, speech, speech-to-speech.
- 3:12
There's a lot of thing that you can do with, um, with the model. And basically, let's say you integrate the, you integrate, um, this model. You can't just run any model, uh, on it.
- 3:23
Like you need to, you need to get some model and there's a good place to get the model is Hugging Face. I'm pretty sure everybody heard, uh, of Hugging Face.
- 3:31
And on Hugging Face you will want to look for MLX Community. This is where like all the, the weight of a model, the quantized weight, the full size will be uploaded.
- 3:41
So you will just be able to go to this community, look for the models. I think right now there's almost 4,000 or 5,000 model uploaded so the community is really active on it.
- 3:52
When a model, uh, is, uh, is released by your, by your lab you will directly have it almost, uh, 30 minutes after release quantized in 4-bit, 6-bit and, and everything that you can, uh, you can imagine.
- 4:02
Here you have a, an example for Gemma 4 Iturbi which is the one I, I run on, um, on iPhone. There's like a lot of variant of it for, from BF16, MX- [clears throat] MXFP4 like 5-bit, 6-bit. [clears throat]
- 4:16
There's everything. So, so you download mlx-swift-lm, install it with your agent or anything. Then you go to MLX Community and you just choose the model you want to run and then with ID you can just pass it to the framework and it will directly, uh, it, it will be integrated with Hugging Face to download the model with,
- 4:35
uh, mlx-swift-lm directly. So you just need to grab the ID and the, and pass it to, to the framework.
- 4:41
Usually when you're running the, the model on, um, on iPhone what you want to, what you want to do is quantize... is select, is selecting some quanti- quantized version of the model because the full size will be way too large.
- 4:54
What I go recommend is going, it's trying depending on the size between 3-bit and, uh, and 8-bit usually. Uh, between 4-bit and 8-bit. A bit usually under like 4-bit it's getting, it's starting to get a lot of, uh, impact on the output and, um, and the model are not that great usually.
- 5:12
4-bit is the lower I would go and 8-bit is the higher I would go if you're using really small models. And in my app for example I have like some bigger model like Gemma 4 but I also have some liquid model where that is, uh, 350 parameters and this it can run in Shortcuts.
- 5:28
So you can do a lot of kind of au- automation because the model are really fast and really efficient, uh, at that, uh, at that size to do some text processing and, uh, things like this.
- 5:39
And on the la- on the latest iPhone like if you take Gemma 4 Iturbi quantized in 4-bit it's extremely fast. Like it can run easily at, uh, 40 token, uh, 40 token for, uh, uh, per second.
- 5:51
I will just do an update in the slide because I remove, uh, I remove a slide where there's the video but maybe I can add it back because I have little bit more time.
- 6:01
Just go like this. So just to show you, uh, in demo what, what the 40 token per second means.
- 6:10
And that's running live offline. And as you can see, it's like really fast, like 40 token per second is more than acceptable for a lot of use case. This is, of course, streaming.
- 6:22
You can also like not do streaming and do a UI that just will wait for, for four second. And here the output is quite long, so it's generating a lot of tokens.
- 6:31
So on device with MLX, it's working. It's here. Like, um, it's really not easy and really not hard to, to integrate. As I said, if you go to the, to the repo MLX Swift LM, very...
- 6:44
It's a breeze to, to install. On top of that, as I said, like again, latest iPhone is really great, but it works also with older iPhone. Like you will not get 40 token per second, which is quite fast.
- 6:55
But even if you get 20 tokens per second, that's already great and useful for a lot of, um, for a lot of application, a lot of use case that you would want to do with, um, with your, um, with your, uh, with your app.
- 7:11
You can go to and scan this QR code if you want to try it by yourself. If you have a, if you have an iPhone, the app is on the App Store.
- 7:17
It's free to use. Only thing is that you will have to download the model that's u-usually around one gig-gigabytes or, or three gigabytes. Really depend on, uh, on which model.
- 7:27
But, uh, that's the biggest barrier right now. It's the size of a model, but this also is getting better. Model are getting smaller or getting smarter and, um, they also, uh, the iPhone is getting better.
- 7:38
So next iPhone and second and the next, next iPhone, everything will just, um, is reaching really great, uh, usability from, um, what I can see. And also on top of that, like, uh, maybe you have heard the news yesterday, uh, Locally AI have been acquired by, uh, LM Studio.
- 7:55
If you don't know, uh, LN- LM Studio, [clears throat] it's basically, um, uh, kind of a s- AI studio for all your local models. So you can download the, the model with L- any model with LM Studio directly from Hugging Face.
- 8:10
You can run them and you can open the server. You can run them with Llama CPP, but also MLX. So you can really compare the different engine, how they, uh, how they work.
- 8:19
You can, um, as I said, you can open a, a server locally and connect your application to this, uh, to this hosted server, uh, with va-various, um, various response, response type.
- 8:31
For example, open I- OpenAPI, uh, response type or, um, [clears throat] or, um, or Anthropic response type, uh, for streaming anything and you can just get any model runnings, uh, really easily with that.
- 8:45
So, and I want to thank you. That was a very short introduction how you can do the same and run any model like in Gemma 4 if you want on your, on your iPhone.
- 8:54
If you have any question also. [audience applauding]
- 9:02
Uh, does, does this support tool calling?
- 9:04
Sorry?
- 9:04
Does this support tool calling?
- 9:05
Yes. Like, so yeah, I forgot about that. So it support tool calling. Um, not yet, um, custom, um, not yet structured, structured generation. There are some package on top of MLX Swift LM that, uh, are trying to, to make this, uh, make this working.
- 9:20
Um, I will let you do... Uh, Hugging Face is doing it, but you, you can easily find them, uh, online. But, uh, MLX Swift LM, yes, it support tool calling, so that's really useful if you want to do tool calling and call other s- other system.
- 9:33
And the model mo- the model are getting also better at tool calling. They were not so great like a year ago. Now it's getting much better.
- 9:43
I think you mentioned two things. First was the GitHub repo.
- 9:46
Yes.
- 9:46
And then the second thing was the app.
- 9:49
Um, so you have the GitHub repo for, um, for MLX Swift LM. That's the, the package that will install in your app. And then you need to go to Hugging Face and, uh, and getting, uh, for, to, to get the weight of a model.
- 10:02
But for a, for a normal user, they can just download the app from the App Store?
- 10:06
Oh, and yes, if you want to try my app, uh, you can. If you want to try it like right now without having anything to install, you can do that.
- 10:13
And you can choose any open source model inside the app?
- 10:16
Uh, there's a selection, but, uh, that they are not... It's not any. Like I'm ensuring that all the model runs correctly, um, on the, on the iPhone because not all of them work, uh, work well.
- 10:29
Thank you very much. [audience applauding] [outro jingle]