Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

Talk transcript

1. 0:00

   [upbeat music] Cool.

2. 0:15

   Awesome.

3. 0:16

   Um, hi, everyone. Um, I'm Florina Muntenescu. I'm a developer relations engineer working on lots of things AI, both intelligent experience, is what we're gonna talk about today, and also developer productivity.

4. 0:28

   So if you wanna talk about that, find me at the booth later on.

5. 0:33

   Uh, I'm Oli, uh, PM for Android AI. Uh, our work spans a bit of everything. I've been working on this since before we called AI and back when we were calling it just ML.

6. 0:42

   Um, so we do a bit of everything from helping build features, uh, things that run directly in the OS with applied AI, so things like background tasks that optimize the screen brightness and system memory and everything, through to developer tools and infrastructure to make it easier, people who are building, uh, features on Android, uh, and right down

7. 0:59

   to the lowest levels of hardware acceleration and OS optimization with many of our silicon partners. So we do a bit of everything, so happy to talk about any of that as well.

8. 1:06

   Uh, we want to make this a conversation where you're asking the questions, and then we're trying to answer as many as possible of them. Um, but I'm not sure, like, how familiar you all are with, like, what's available on, uh, on Android about how to build intelligent experiences.

9. 1:25

   How... Would you raise your hand if you already know how to build intelligent experiences on Android with, like, on-device, hybrid, or cloud? Okay. We're gonna do a short, like, TLDR, and then hopefully from there you might also get some ideas of, like, what kind of questions you, you want to ask.

10. 1:45

    Uh, okay. So if you wanna build intelligent experiences on Android, you can use, uh, on-device, uh, models. You can use hybrid, so use on-device when on-device is available. But otherwise, do the inference on the cloud or just do cloud inference fully.

11. 2:01

    Um, so when it comes to building on-device, um, uh, experiences, it means that the prompts are processed, uh, directly on the device with no data being sent to the server.

12. 2:13

    Uh, this is great because of multiple reasons. It means that you're able to, uh, take advantage of local processing, uh, so sensitive data, like, I don't know, banking information doesn't have to leave the device.

13. 2:24

    You have the ability to do offline work, and of course it means that there's no additional cost for all of the inference.

14. 2:33

    Uh, so use cases like, you know, sensitive data, like, or personalization or anything that requires, like, maybe a shorter context window, like translations and so on, are things that you can do on-device.

15. 2:46

    So there are two main ways you can do this in Android. You can use the ML Kit's, uh, GenAI APIs where you get access to Gemini Nano, which is our on-device model.

16. 2:57

    Or if you need something that's much more customizable, your own models, your own custom models, then you can use LiteRT LLM. There's another talk after this talking about LiteRT LLM, so we're just gonna talk about the ML Kit's GenAI APIs.

17. 3:11

    So we said that these are the ones that give you access to Gemini Nano. So this is our most efficient models, model for on-device tasks, and it's using the same architecture as Gemma 4 that you probably have heard about that was launched, I think, last week.

18. 3:26

    But Gemini Nano is optimized for Android devices. So the way, um, uh, Gemini Nano comes on your device is through the AICore system service. So with this it means that you only have one model on the device, and then all of the apps are using that same model through this AICore system.

19. 3:45

    Because it's, uh, Gemini, uh, Nano, it means that we're optimizing for, uh, the hardware, so you get like, what is it? Uh, lower latency and, um, uh, faster execution for AI tasks.

20. 3:59

    Yeah. So you can think about this as, you know, imagine you're using a cloud service, right? And everything is kind of provided for you. You don't have to worry about setting up the LLMs, running them on devices, getting your TP inference, et cetera.

21. 4:11

    Um, you just focus on your feature, your prompt, and then the service provides everything. We're doing the same thing for on-device, right? So we get the models to the device.

22. 4:19

    We make sure they're optimized, take advantage of the specific hardware that's available on each device at runtime, um, and sort of basically package that all up for you so you don't have to worry about any of that stuff.

23. 4:29

    Um, and it's also, it has privacy and, uh, safety considerations. So this means that, you know, your requests are not gonna be, I don't know, messed up with all of the other apps' requests.

24. 4:39

    They run in isolation, and then the input and the output data is not, uh, stored on device at all, so it's all as private and secure as possible.

25. 4:49

    Okay. So how do you access these, uh, the Gemini Nano? So you do this using the ML Kit GenAI APIs. So actually, the, the GenAI ML Kit APIs are part of the bigger ML Kit APIs where you also get access to, uh, APIs end models for, like, vision and natural language.

26. 5:07

    Uh, for GenAI specifically, we actually have a bunch of APIs specialized on specific tasks like summarization, proofreading, rewrite, and so on. But the most powerful of them is actually the prompt API.

27. 5:22

    Um, this means that you're able to send natural language, uh, request to Gemini Nano. For now, it supports text and image as input and text only as output.

28. 5:33

    So with the prompt API, it means that you can easily use it for, like, stuff like image understanding, uh, content assistance, content analysis, entity extraction. So I would pretty much say whatever use case you wanna do, prompt API is gonna be able to help you.

29. 5:50

    Okay. What, um, the Gemini Nano models are only available on, like, Pixel 9, Pixel 10, kinda like that generation of devices, not just on Pixel devices, but also on other OEMs.

30. 6:02

    So what do you do if you wanna get access or be able to use AI on other devices? Well, you can do this by using, um, cloud when the local model is not available for that device, so that increases the reach of, uh, your feature.

31. 6:17

    So to do this, you would use the Firebase AI logic. Uh, we've launched the hybrid inference a couple of weeks ago. Um, so this means that you're able to decide, like if, uh, we have, uh, Gemini Nano available on device, then you can run that inference on device.

32. 6:34

    Otherwise, it can run on the cloud. And especially now, I think, with, uh, um, Gemma 4, when the next generation of Gemini Nano will be available, and I think we can already use it w- using the AI Core, uh, preview, uh, it means that you're able to have the-- a similar experience using on device with Gemma-- Gemini

33. 6:53

    Nano 4, and then also in the cloud with, like, the Gemini Flash, uh, models. But if you want access to even more powerful models like the, uh, the Pro Flash flashlight, and you want to run all of this in the cloud, you can also do this using the Firebase AI logic.

34. 7:12

    So this gives you access to both Gemini API and Vertex AI and Gem- uh, Gemini Developer API, um, as providers.

35. 7:19

    Wow.

36. 7:20

    Okay.

37. 7:20

    That was a whirlwind.

38. 7:21

    Yeah, sorry. [laughs]

39. 7:23

    Um, I think this is amazing. Um, we just wanted to kinda set the scene because with Android, we're really trying to make sure there is a comprehensive offering, right?

40. 7:30

    So everything that you need, whether it's running things locally on device for low latency responses, for private inference, whether it's going to the cloud for our most powerful models, whether it's something in between, we're building out a solution for you that-

41. 7:44

    Mm-hmm

42. 7:44

    ... covers each of those points, and we're trying to make those APIs as consistent as possible so that it's easy for you to blend what is needed. Um, so we wanted to kinda give you a, a high-level view of everything.

43. 7:54

    Um, but we're also happy to go super deep on any of these things. We're happy to talk about use cases. We're happy to talk about what we're seeing in the industry, what we think is interesting, where we're going.

44. 8:03

    Um, but we really just wanted to set the scene a little bit-

45. 8:05

    Mm-hmm

46. 8:05

    ... with some of the different things we've got available today. Florina, anything?

47. 8:10

    Yeah.

48. 8:10

    Oh, okay.

49. 8:11

    That's all I said. So I think actually now we're handing it over to all of you. Okay, go for it.

50. 8:16

    Um, my main question is like, have you checked amount of RAM usage or, like, the battery concerns that we might have-

51. 8:24

    Mm-hmm

52. 8:25

    ... if we start running Gemini Nano models or, uh, LiteRT, uh, models?

53. 8:28

    Yeah. A-absolutely. So as Florina mentioned, these models today, we've shrunk them down as much as we could whilst trying to-

54. 8:35

    Yeah

55. 8:35

    ... maintain all that capability. But that does mean they need flagship capabilities, right, right now. Um, for battery concerns, that's one of the reasons why we've produced AICore, which is to basically optimize everything as much as we can so that you can know that you can rely on that and say, "If it's available, I'm gonna be able

56. 8:51

    to use it, and I'm gonna get the best performance out of it." Um, yes, there is battery impact. If you are running this nonstop, you are gonna run down the battery fairly quickly.

57. 9:00

    Um, but what we're finding is that the kinds of use cases where people are using it today are things like, um, you know, a user's coming and asking a question, and they're responding or they're manipulating some data, and it's happening at that point in the flow.

58. 9:13

    Users maybe use it 10, 20 times a day. That sort of usage is really not, you know, concerning for battery life.

59. 9:19

    Mm-hmm.

60. 9:19

    Where we're seeing more batch use cases, so I have a bunch of stuff that I want to process, oftentimes that's not necessarily latency sensitive, so people are using that in the background, perhaps when it's running over charge overnight, and they can basically run that continuously until they've completed all their tasks.

61. 9:35

    Mm-hmm.

62. 9:35

    Um, but this is one of the reasons why we're trying to build into the platform so that we can sort of do all those optimizations for you, so you don't need to worry about it.

63. 9:41

    If, however, you have very specific needs and you want to go with custom models, we have a bunch of tools for profiling, for trying to determine what the impact is gonna be for you.

64. 9:50

    But of course, it does need you to do some of that work if you're gonna go custom.

65. 9:54

    Mm-hmm. Cool.

66. 9:55

    Sorry, can I just give you this hand mic? Is there anywhere that-

67. 10:01

    Do you want it? Question. And we can just pass that around if you've got a question. Thank you very much.

68. 10:05

    Um, yeah, quick question regarding-- and two, actually two questions. First one, quick one. Uh, so in the, in the case of using ML Kit-

69. 10:13

    Mm-hmm

70. 10:13

    ... which will be, I would say, the most optimized one, uh, offering, uh, it means, uh, does that mean that we are using, um, an, a model that is shared cross apps-

71. 10:23

    Mm-hmm

72. 10:23

    ... on device?

73. 10:24

    Yeah.

74. 10:24

    Yeah.

75. 10:25

    Okay. If yes, then, um, is, um, how are we sure is it managing the scale? Imagine, like, a user have 100 app that actually all, in two years' time, they will all using that, uh, same model.

76. 10:39

    Uh, how are you handling that to get, I would say, uh, latency as good as possible?

77. 10:43

    Yeah. So, so that's exactly one of the reasons why we're doing this at a system level, right? If you imagine these models, we're talking about the smallest ones are, like, one gigabyte to be useful.

78. 10:53

    The ones we're shipping are actually closer to three, four gigabytes in total. Um, it's not really very easy to ship that as an app developer yourself. It really requires you to have an absolutely killer feature to justify that to an end user.

79. 11:06

    Mm-hmm.

80. 11:06

    If we put this in the system, we do that once, and everyone can share and benefit from it, well, that shares that cost, right? So that's one of the reasons why we're doing this.

81. 11:13

    So we have no concerns about hundreds of apps using this because we've centralized that cost. In terms of lots and lots of apps using this to generate features, what we tend to find is that, yes, it does have a battery impact.

82. 11:25

    But by centralizing that, by queuing things, by making sure that there is some overall system management, we can inform the user of like, "By the way, this app is using quite a lot of quota to do this.

83. 11:36

    Do you want to do that?" Right? We don't ask that, but as in we're attributing it to the battery impact. And we've seen this time and time again with things like GPS or Wi-Fi.

84. 11:44

    Mm-hmm.

85. 11:45

    If the user feels they're getting value out of the app, they're very happy to use that. They're happy to spend their battery on the features they love. If, however, the app is doing something that doesn't provide a lot of value to the user, they might not want to use that, right?

86. 11:56

    So we think it's really important to just make these capabilities available, make it easy for you to use them sensibly.

87. 12:02

    Mm-hmm.

88. 12:02

    And then developers will build amazing features, and users will choose what they want to spend their battery on.

89. 12:07

    But from a developer perspective, you don't need to worry about that. You just do your inference, and then that's it. And then the AI Core does all of the-

90. 12:16

    Correct

91. 12:16

    ... the handling.

92. 12:16

    We handle the scheduling. We handle making sure that-

93. 12:18

    Mm-hmm

94. 12:18

    ... you know, things are getting queued up appropriately and stuff, so you don't need to worry about that.

95. 12:22

    The only the probably thing to think about is if there are other apps using that model, they're probably e-ep-- the our prompt will get in the queue, basically

96. 12:30

    So if you're trying to obviously access it from the background, then yes, you'll be queued. If you're in the foreground, of course, you're gonna have top priority. Um, so whichever app is currently being used actively by the user is obviously going to be prioritized by the system.

97. 12:41

    Okay. Had a question in the back.

98. 12:43

    Yeah, I have... Maybe you can hear from me. I have a couple of questions. I have Android 12 on my phone, which is not a Pixel, right?

99. 12:52

    Mm-hmm.

100. 12:52

     It's Asus, but doesn't matter. And I have used, um, in this-

101. 12:56

     A bit closer

102. 12:57

     ... Assistant. Can you hear me better now?

103. 12:58

     Yeah. Great.

104. 12:59

     So I have a couple of questions. Uh, the first one is, um,

105. 13:04

     when I'm asking Google for to do... solve something for me, is it doing locally on my mobile or is it doing remotely?

106. 13:11

     When you are-- When you say when I'm asking Google, uh, do you mean like the Gemini app or-

107. 13:16

     Yeah. Whatever, whatever I have on my Android by default. I did not install anything. It is there. I just say, "Hey, Google-"

108. 13:21

     Mm-hmm.

109. 13:21

     "... what's the temperature outside?" And it will give me an answer. Is it running this locally or is it running remotely?

110. 13:27

     So that would be the Gemini app or like the search app that has integration with the Gemini APIs. To be honest, I'm not sure. My expectation would be that it's, uh, running this, uh, remote.

111. 13:40

     Right. So then my next question makes a lot more sense. How can I then ins-install the things that you have presented here?

112. 13:46

     Mm-hmm.

113. 13:46

     And, and second part to this, is it possible to write skills to-

114. 13:51

     Mm-hmm

115. 13:51

     ... for it locally to improve the kind of answers that it gives me if I notice that it is giving me consistently the wrong answers for whatever reason?

116. 13:59

     Mm-hmm.

117. 13:59

     Possible to do that?

118. 14:00

     So what you're talking about is like two different, uh, s- uh, user journeys 'cause if you're building your own app, you wouldn't interact with the Gemini app. That's a completely different app.

119. 14:10

     But rather you would interact with the model itself.

120. 14:13

     Yes.

121. 14:13

     Therefore, if you're using an on-device model, you don't need to care about the, the cloud. All of that inference will be on device.

122. 14:20

     Yes. So I'm, I'm asking precisely about that. If I installed what you showed here-

123. 14:24

     Yeah

124. 14:24

     ... locally, assuming it is possible-

125. 14:26

     Yes. Yeah. Yeah

126. 14:26

     ... can I then write, uh, write skills to improve the kind of responses it gives me?

127. 14:30

     I don't think you would write the skills. Um-

128. 14:34

     No. So I mean, it, it depends on the kind of experience you're trying to build, right? But if you think about a skill, it's just something that you're sticking into the prompt-

129. 14:41

     Yes

130. 14:41

     ... alongside the rest of your query.

131. 14:43

     Yes.

132. 14:43

     So the tools we're providing here are more sort of low level and designed for you to be able to build back on top of that, right? So if, for example, you went and installed something like Pocket Claw or Open Claw on the phone itself-

133. 14:56

     I see

134. 14:56

     ... that would then basically take those skills, compose them into a prompt, which it would run through this API.

135. 15:01

     Great.

136. 15:01

     Um, so what we're focusing on here is building the lower layers to enable you or others to build those things on top.

137. 15:08

     Thank you.

138. 15:10

     I guess the battery is dying.

139. 15:11

     Time.

140. 15:13

     Sorry.

141. 15:14

     Five minutes. We started late.

142. 15:16

     We start. Okay.

143. 15:17

     Uh, there was one question there. Um-

144. 15:20

     Um, so, uh, what's the difference between, uh, the model that we have, like the Gemini Nano model versus any model that we have on the, like AI Edge Gallery app?

145. 15:29

     Mm-hmm.

146. 15:29

     What level of access can we have? Um, because I don't think we have system file level access, uh, using, uh, any of these APIs or do we?

147. 15:36

     Okay.

148. 15:37

     If we want to implement anything via that, let's say via a picture file-

149. 15:40

     Mm-hmm

150. 15:40

     ... stored in a system, uh, folder, can we have that using these APIs, or is it isolated in its own memory?

151. 15:47

     Yeah. So the Gemini Nano model itself is completely contained by AICore, right? So you don't need to worry about how to set that up and configure it. You use the APIs to access it, and it's all handled by the system.

152. 15:58

     In terms of building a Rags-like solution, yes, absolutely, you can do that with the prompt API. Um, we are looking at extending some of the APIs we have, for example, adding an embedding API soon, and that will make it easier for building these kind of Rag-like solutions.

153. 16:10

     Um, but yes, it's totally possible to do that. In terms of the relationship between AI Edge Gallery and AICore, AI Edge Gallery is a showcase to show you what's possible, right?

154. 16:19

     You can use AICore backing in that, or you can use custom models. We really want you to see the full breadth of things you can do. Um, AI Edge Gallery, I think, is a really good way of showing the frontier of what you can do, right?

155. 16:32

     But it requires more uplift. There's more work you have to do to take advantage of those things. AICore is really focused on making it easy for you to just focus on the prompt, focus on the specific feature rather than any of the setup.

156. 16:43

     And build like production level apps.

157. 16:45

     Yeah.

158. 16:45

     Okay.

159. 16:46

     Cool.

160. 16:46

     And system access?

161. 16:48

     And?

162. 16:48

     Do you have system access through the, uh, code?

163. 16:51

     When you say system, do you mean like system prompt or-

164. 16:53

     Files

165. 16:54

     ... files? So-

166. 16:55

     Like if I-- Like I have taken a lot of pictures today.

167. 16:56

     Yeah.

168. 16:57

     Mm-hmm.

169. 16:57

     I want to then, like, summarize everything together-

170. 17:00

     Mm-hmm

171. 17:01

     ... let's say.

172. 17:01

     Yeah.

173. 17:01

     Make that note.

174. 17:02

     Yeah. So the ML Kit, uh, GenAI API, the prompt API allows both text and the image input.

175. 17:08

     Yeah.

176. 17:08

     So you can give it access, yeah.

177. 17:10

     So all the things your app normally has access to in terms of files and so on, you can then obviously pass them through the API, uh, and run them just like, just like you would do on a server-based inference, right?

178. 17:18

     In terms of s- creating the prompt, feeding the information in.

179. 17:23

     Is there any vectorizing em-embedding model also available for this API? Like I want to vectorize some of my text notes so that they're like, you know, I can get similarity between some other notes maybe.

180. 17:32

     Yeah.

181. 17:33

     So does this have that or?

182. 17:35

     So we don't yet have an embedding API.

183. 17:37

     Okay.

184. 17:37

     We will soon.

185. 17:38

     Cool.

186. 17:39

     Mm-hmm.

187. 17:39

     Um, so if you're looking at things like the Gemera embedding model-

188. 17:43

     Mm-hmm

189. 17:43

     ... um, that's what we're going to make available so that you can use that directly from the API.

190. 17:48

     Cool. Oh, you want a last question?

191. 17:49

     Last question.

192. 17:50

     We're, we're being kicked out.

193. 17:51

     Thanks. Um, yeah, there's a diversity of devices, you know. Um, and there's also a diversity of models. You showed the prompt API and, you know, embeddings are coming soon, but how do I think about this sort of that matrix of-

194. 18:03

     Mm-hmm

195. 18:04

     ... you know, device, like what's capable on a given device versus-

196. 18:07

     Mm-hmm

197. 18:07

     ... like what, what models could be used.

198. 18:09

     Mm-hmm.

199. 18:09

     You know, are there more, uh, use cases than just LLM and embedding model, or are there other models that also run on device?

200. 18:17

     Mm-hmm.

201. 18:17

     Um, and then will they be widespread across all devices, or do, do I need the latest flagships to, to use those?

202. 18:25

     So a-as Florina showed, the ML Kit API s- covers a range of things, so text, OCR, vision, all sorts of stuff. Those models, the classical models, are much smaller.

203. 18:34

     They can run on a super large range of device, like billion-plus devices, no problem. Um, the new ones, the GenAI APIs, those ones require currently fairly flagship devices from the last couple of years.

204. 18:44

     What we're doing with AICore is we package that up and say, "Hey, you can call this API, and if it's available, you know it's gonna run well." If, however, that reach is not big enough for you and you want to basically do the work to test on a wider range of devices, LiteRT-LLM, which Cormac's gonna talk about

205. 18:59

     in a second, can help you do that, but you will then need to do the work for testing. Cormac hopefully will talk about some of the tools we have available that make it easier to do that testing, but you will need to do that testing yourself.

206. 19:09

     For AICore, we cover that all, so you know that if it's covered by AICore, it's gonna run well.

207. 19:14

     Thanks.

208. 19:14

     We're gonna be around, um, today and tomorrow, so please come and ask us questions.

209. 19:19

     Thanks a lot.

210. 19:19

     Thank you.

211. 19:20

     Thank you very much. [clapping] [outro music]