Running Gemma 4 On-Device: 40 Tokens/s on iPhone with MLX

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:15

   Hello everybody. [clears throat] I'm going to show you today how to run Gemma 4 on iPhone with, uh, MLX. So first, let's introduce myself. I'm Adrien. You can find, uh, my Twitter if you want to learn more about, uh, all on-device things.

3. 0:29

   I'm the developer of, um, Locally AI, so maybe you have already seen, uh, the, the app. So Locally AI is a chatbot that allow you to run on-device models on your iPhone with, uh, MLX.

4. 0:42

   So I will just, uh, go through ML- what is MLX in a, in a few, few seconds. Basically, uh, lo- as I said, it's a, it's a chatbot. It's fully native.

5. 0:51

   You can also chat with, uh, Apple Foundation now with it. And the, and many models like, um, that are compatible with MLX and one of this model is, uh, Gemma 4.

6. 1:02

   So [clears throat] basically, Gemma 4, uh, by Google DeepMind have a lot of model and some of them like can run on iPhone, like the smaller ones and they are pretty great.

7. 1:13

   Maybe you have seen on, uh, on Twitter one of the post I've made that, uh, I should... I demo it running, uh, in the app, uh, on iPhone. It's really fast.

8. 1:20

   It runs really well on MLX and behind the app, so [clears throat] it's using MLX. MLX is a framework made by Apple that is optimized for Apple silicon, so mainly, mainly the, [clears throat] the chip in iPhone but also the chips on the, on the Mac.

9. 1:36

   The, the app Locally is available on iPad. It works also very well on this device and macOS and, uh, everything is, uh, is built to be as optimized as possible on the, on these, uh, devices.

10. 1:49

    So if you want, if you want to run on iPhone, um, a s- large model, so Gemma works well but you have a lot of, uh, model that you can run also.

11. 1:58

    The Qwen model and the, um, and the small LLM model from the Hugging Face. You-- the place you, you will want to go is GitHub and go to the repo mlx-swift-lm.

12. 2:07

    I won't go into detail how you implement the repo. It's... I think I will let your agent in, uh, implement that, uh, for you but it's one repo that you need to install if you're developing, um, iOS or macOS or iPadOS app.

13. 2:19

    And you can use, um, you can use that to simply download the model and then run it. The API is very straightforward, very simple to, to implement. In less than ten minutes, you can have an iOS app with a model that is running on your, on your device.

14. 2:32

    That's very simple, uh, to do as, uh, as mentioned. MLX, uh, so this is mlx-swift-lm but if you're more into Python apps or macOS app you can also run mlx-vllm from, uh, from Prince.

15. 2:44

    Maybe you have, uh, seen him like he's doing, um, on device for audio with MLX Audio and visual model with, uh, MLX VLM and also MLX Video to, to run, um, to run, uh, image, uh, image generation model or image, uh, or video generation model.

16. 3:00

    MLX it's, uh, there's... The ecosystem is getting, uh, bigger or it's getting bigger. It's g- it's really great right now. You can do pretty much everything like, uh, omni models or as I said, text-to-speech, speech to, speech, speech-to-speech.

17. 3:12

    There's a lot of thing that you can do with, um, with the model. And basically, let's say you integrate the, you integrate, um, this model. You can't just run any model, uh, on it.

18. 3:23

    Like you need to, you need to get some model and there's a good place to get the model is Hugging Face. I'm pretty sure everybody heard, uh, of Hugging Face.

19. 3:31

    And on Hugging Face you will want to look for MLX Community. This is where like all the, the weight of a model, the quantized weight, the full size will be uploaded.

20. 3:41

    So you will just be able to go to this community, look for the models. I think right now there's almost 4,000 or 5,000 model uploaded so the community is really active on it.

21. 3:52

    When a model, uh, is, uh, is released by your, by your lab you will directly have it almost, uh, 30 minutes after release quantized in 4-bit, 6-bit and, and everything that you can, uh, you can imagine.

22. 4:02

    Here you have a, an example for Gemma 4 Iturbi which is the one I, I run on, um, on iPhone. There's like a lot of variant of it for, from BF16, MX- [clears throat] MXFP4 like 5-bit, 6-bit. [clears throat]

23. 4:16

    There's everything. So, so you download mlx-swift-lm, install it with your agent or anything. Then you go to MLX Community and you just choose the model you want to run and then with ID you can just pass it to the framework and it will directly, uh, it, it will be integrated with Hugging Face to download the model with,

24. 4:35

    uh, mlx-swift-lm directly. So you just need to grab the ID and the, and pass it to, to the framework.

25. 4:41

    Usually when you're running the, the model on, um, on iPhone what you want to, what you want to do is quantize... is select, is selecting some quanti- quantized version of the model because the full size will be way too large.

26. 4:54

    What I go recommend is going, it's trying depending on the size between 3-bit and, uh, and 8-bit usually. Uh, between 4-bit and 8-bit. A bit usually under like 4-bit it's getting, it's starting to get a lot of, uh, impact on the output and, um, and the model are not that great usually.

27. 5:12

    4-bit is the lower I would go and 8-bit is the higher I would go if you're using really small models. And in my app for example I have like some bigger model like Gemma 4 but I also have some liquid model where that is, uh, 350 parameters and this it can run in Shortcuts.

28. 5:28

    So you can do a lot of kind of au- automation because the model are really fast and really efficient, uh, at that, uh, at that size to do some text processing and, uh, things like this.

29. 5:39

    And on the la- on the latest iPhone like if you take Gemma 4 Iturbi quantized in 4-bit it's extremely fast. Like it can run easily at, uh, 40 token, uh, 40 token for, uh, uh, per second.

30. 5:51

    I will just do an update in the slide because I remove, uh, I remove a slide where there's the video but maybe I can add it back because I have little bit more time.

31. 6:01

    Just go like this. So just to show you, uh, in demo what, what the 40 token per second means.

32. 6:10

    And that's running live offline. And as you can see, it's like really fast, like 40 token per second is more than acceptable for a lot of use case. This is, of course, streaming.

33. 6:22

    You can also like not do streaming and do a UI that just will wait for, for four second. And here the output is quite long, so it's generating a lot of tokens.

34. 6:31

    So on device with MLX, it's working. It's here. Like, um, it's really not easy and really not hard to, to integrate. As I said, if you go to the, to the repo MLX Swift LM, very...

35. 6:44

    It's a breeze to, to install. On top of that, as I said, like again, latest iPhone is really great, but it works also with older iPhone. Like you will not get 40 token per second, which is quite fast.

36. 6:55

    But even if you get 20 tokens per second, that's already great and useful for a lot of, um, for a lot of application, a lot of use case that you would want to do with, um, with your, um, with your, uh, with your app.

37. 7:11

    You can go to and scan this QR code if you want to try it by yourself. If you have a, if you have an iPhone, the app is on the App Store.

38. 7:17

    It's free to use. Only thing is that you will have to download the model that's u-usually around one gig-gigabytes or, or three gigabytes. Really depend on, uh, on which model.

39. 7:27

    But, uh, that's the biggest barrier right now. It's the size of a model, but this also is getting better. Model are getting smaller or getting smarter and, um, they also, uh, the iPhone is getting better.

40. 7:38

    So next iPhone and second and the next, next iPhone, everything will just, um, is reaching really great, uh, usability from, um, what I can see. And also on top of that, like, uh, maybe you have heard the news yesterday, uh, Locally AI have been acquired by, uh, LM Studio.

41. 7:55

    If you don't know, uh, LN- LM Studio, [clears throat] it's basically, um, uh, kind of a s- AI studio for all your local models. So you can download the, the model with L- any model with LM Studio directly from Hugging Face.

42. 8:10

    You can run them and you can open the server. You can run them with Llama CPP, but also MLX. So you can really compare the different engine, how they, uh, how they work.

43. 8:19

    You can, um, as I said, you can open a, a server locally and connect your application to this, uh, to this hosted server, uh, with va-various, um, various response, response type.

44. 8:31

    For example, open I- OpenAPI, uh, response type or, um, [clears throat] or, um, or Anthropic response type, uh, for streaming anything and you can just get any model runnings, uh, really easily with that.

45. 8:45

    So, and I want to thank you. That was a very short introduction how you can do the same and run any model like in Gemma 4 if you want on your, on your iPhone.

46. 8:54

    If you have any question also. [audience applauding]

47. 9:02

    Uh, does, does this support tool calling?

48. 9:04

    Sorry?

49. 9:04

    Does this support tool calling?

50. 9:05

    Yes. Like, so yeah, I forgot about that. So it support tool calling. Um, not yet, um, custom, um, not yet structured, structured generation. There are some package on top of MLX Swift LM that, uh, are trying to, to make this, uh, make this working.

51. 9:20

    Um, I will let you do... Uh, Hugging Face is doing it, but you, you can easily find them, uh, online. But, uh, MLX Swift LM, yes, it support tool calling, so that's really useful if you want to do tool calling and call other s- other system.

52. 9:33

    And the model mo- the model are getting also better at tool calling. They were not so great like a year ago. Now it's getting much better.

53. 9:43

    I think you mentioned two things. First was the GitHub repo.

54. 9:46

    Yes.

55. 9:46

    And then the second thing was the app.

56. 9:49

    Um, so you have the GitHub repo for, um, for MLX Swift LM. That's the, the package that will install in your app. And then you need to go to Hugging Face and, uh, and getting, uh, for, to, to get the weight of a model.

57. 10:02

    But for a, for a normal user, they can just download the app from the App Store?

58. 10:06

    Oh, and yes, if you want to try my app, uh, you can. If you want to try it like right now without having anything to install, you can do that.

59. 10:13

    And you can choose any open source model inside the app?

60. 10:16

    Uh, there's a selection, but, uh, that they are not... It's not any. Like I'm ensuring that all the model runs correctly, um, on the, on the iPhone because not all of them work, uh, work well.

61. 10:29

    Thank you very much. [audience applauding] [outro jingle]