AI Engineer Europe 2026
Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind
Read the talk
Gemini Nano on Android: local inference, shared models, and device limits
Android offers local, hybrid, and cloud inference, but building a useful feature also means understanding who manages the model, where app permissions apply, and which devices can run it.
From a talk by Florina Muntenescu and Oli Gaymond
Before you start: Familiarity with Android applications, model prompts, and the distinction between local and server inference will help; the Kotlin example uses only string composition.
Where should an Android AI feature run?
Where should an intelligent Android feature run—and how much infrastructure should its developer manage? Florina Muntenescu introduces the developer perspective: intelligent experiences and developer productivity. Oli Gaymond, working on Android AI product management, describes a platform spanning screen-brightness and system-memory optimization, developer infrastructure, and hardware acceleration with silicon partners. The inference APIs sit within that larger system.
The first choice is where to process a request.
| Approach | Where inference runs |
|---|---|
| On-device | On the phone |
| Hybrid | Locally when available; otherwise in the cloud |
| Cloud | On a remote service |
These are deployment choices for an application feature, not three different kinds of user interface.
Consider a feature handling banking information. With local inference, the prompt is processed on the phone without sending that information to a server. The same feature can work offline, and local calls avoid additional server inference charges. That does not eliminate the phone’s battery or storage costs. Personalization and shorter-context tasks, including translation, are other examples Muntenescu gives for local processing.
Android offers two paths into this work. ML Kit’s GenAI APIs expose Gemini Nano through a managed interface. For your own models or more customization, LiteRT-LM, called LiteRT LLM in the session, gives you more control over deployment. The distinction is how much of the model infrastructure you want to own.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
AICore makes the model a system service
Muntenescu connects Nano’s architecture to Gemma 4. That connection applies to the next generation, Gemini Nano 4, described in Google’s Gemma 4 preview announcement, rather than every existing Nano version. Android delivers the model through AICore, a system service, so applications can share a model instead of each distributing their own copy.
Gaymond compares AICore to a managed cloud service: you provide the feature and its prompt; the service handles getting the model onto the device and running it. AICore selects optimizations for the hardware available at runtime. The intended benefit is lower latency and faster execution without requiring each application developer to configure the inference stack.
Sharing the model does not mean sharing request context. Requests run in isolation. The non-retention guarantee concerns AICore’s processing: it does not retain inputs and outputs after processing a request. It does not prevent the calling application from storing data itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From task APIs to hybrid inference
ML Kit already includes vision and natural-language capabilities. Its GenAI interfaces add specialized operations such as summarization, proofreading, and rewriting. The more general Prompt API accepts a natural-language request, with text and image inputs and text output. That supports image understanding, content assistance, content analysis, and entity extraction without requiring a separate task API for every feature.
Device availability creates the next constraint. Muntenescu cites Pixel 9- and Pixel 10-generation hardware, along with devices from other manufacturers, as the class supporting Nano. A feature intended for a wider audience needs another inference path when the local model is unavailable.
Firebase AI Logic’s Android hybrid inference supplies that routing: use Nano when it is available locally, otherwise run in the cloud. Muntenescu says the capability had launched a couple of weeks before the session. She points tentatively to next-generation Nano access through the AICore Developer Preview, anticipating a similar experience between local Nano 4 and cloud Gemini Flash; she does not present a quality comparison. Firebase AI Logic also provides cloud-only access to Gemini Pro and Gemini Flash-Lite, with the Gemini Developer API and Vertex AI as providers.
The platform goal is to make these paths easier to combine: local inference for privacy and responsive interactions, cloud inference for more powerful models, and a hybrid path between them. Gaymond describes API consistency as ongoing work, so developers can blend the capabilities their feature needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Battery cost depends on the workload
The first audience question asks about RAM and battery consumption for Nano and LiteRT models. Gaymond explains that shrinking models while preserving capability still leaves them dependent on flagship hardware. AICore exists partly to optimize that execution, but continuous inference can drain the battery quickly.
Gaymond says that intermittent question-answering or data manipulation used 10–20 times per day is not concerning for battery life. This is a workload observation, with no specified device, token workload, measured energy use, or battery percentage. It should not be read as a battery budget for a different application.
For latency-insensitive batches, he describes processing in the background while the phone charges overnight. AICore’s purpose is to absorb platform optimization work; choosing a custom model shifts more of that work back to the developer, who must profile its resource impact using the available tools. The discussion supplies no quantified runtime-RAM answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model API is below the assistant
Another attendee asks about the default assistant on an Asus phone running Android 12: when they ask for the temperature outside, does the answer come from the phone or a server? Muntenescu cannot establish which app or inference path is involved and says only that she expects it to run remotely. A familiar voice interface does not, by itself, identify where inference happens.
The follow-up asks how to install the local capability and add skills to improve incorrect answers. Muntenescu separates two user journeys: using the Gemini application and building an application that calls a model. These APIs serve the latter. An application calling an on-device model performs that inference locally; it is not extending the separate Gemini app.
Gaymond places skills one layer higher. A skill can be instructions composed into a prompt alongside the user’s query. He mentions Pocket Claw or OpenClaw as hypothetical software that could assemble those instructions and call the API, rather than as integrations demonstrated in the session.
For example, an application could compose a note-formatting instruction and a user’s text before submitting the resulting prompt. In Kotlin, that composition can remain independent of the inference client:
kotlin
fun composePrompt(instructions: String, query: String): String = """
Instructions:
$instructions
User request:
$query
""".trimIndent()
fun main() {
val prompt = composePrompt(
instructions = "Turn the supplied notes into a concise checklist. Preserve names and dates.",
query = "Call Maya on Friday. Send the draft to Leon on Monday."
)
println(prompt)
}
This prepares a request; it does not execute inference or install a skill into Nano. The higher-level application owns the instructions and workflow, while the API supplies the lower-level model capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model access does not grant file access
The next question compares Nano with models in Google AI Edge Gallery, then asks whether these APIs can access pictures stored in system folders. Gaymond starts with the model boundary: Nano remains contained and configured within AICore, and applications access it through APIs. The Prompt API can support a retrieval-augmented generation workflow; he also points to a forthcoming embedding API as a way to make retrieval easier.
Gallery and AICore serve different purposes. Gaymond describes Gallery as a showcase of what local models can do, potentially using AICore backing or custom models. Exploring those capabilities can require more integration work. AICore emphasizes managed deployment and execution so a production application can focus on its prompt and feature.
The attendee makes the file question concrete: take the pictures captured during the day and turn them into a combined note. The Prompt API’s text-and-image input can support that kind of request, but the application must supply the information. It can pass files it already has permission to access; the inference API does not grant unrestricted access to the phone’s filesystem. Constructing the prompt and feeding it accessible content follows the same basic pattern as server-based inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompting and embeddings are separate capabilities
An attendee asks specifically about vectorizing text notes to find similarities between them. Gaymond’s answer is explicit: an embedding API is not yet available through this interface. He says it will arrive soon and identifies a Gemma embedding model as the intended capability. That is a forecast, not an available API in the session. Being able to place retrieved text into a prompt does not mean the same interface can already generate vectors for retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Broader device reach means owning more testing
The final question brings the discussion back to the device-and-model matrix. Are language models and embeddings the whole on-device offering, and does every capability require a recent flagship? Gaymond distinguishes classical ML Kit models—text processing, OCR, and vision—from the larger generative models. Gaymond describes classical ML Kit models as reaching a billion-plus devices, while the GenAI APIs require fairly flagship devices from the preceding couple of years. The device-count claim is a broad reach estimate, not a compatibility guarantee for every API.
The remaining choice is who owns validation across that hardware range.
| Path | Device testing responsibility |
|---|---|
| AICore | Platform manages supported model/device combinations |
| LiteRT LLM | Developer tests chosen models on target devices |
Gaymond presents AICore availability as assurance that the managed model will run well on that device. If that supported range is too narrow, LiteRT LLM lets developers pursue a wider range, but they must test it themselves, even with supporting tools. He hands that topic to Cormac’s following session. A managed model service reduces deployment and testing work; broader custom-model reach requires taking that work back.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Overview of on-device APIs, supported devices, model sharing and application quotas.
Instructions for enrolling a supported device and testing preview models and prompts.
Setup and capability limits for experimental inference using local models and cloud fallback.
Google's open-source framework for deploying language models on edge devices.
An application and source repository for exploring locally running generative AI models.
Further reading
The April 2026 announcement connecting Gemma 4 previews with the forthcoming Gemini Nano 4 generation.
Explains how EmbeddingGemma produces text representations for search and retrieval.
Read the complete timestamped transcript
- 0:00
[upbeat music] Cool.
- 0:15
Awesome.
- 0:16
Um, hi, everyone. Um, I'm Florina Muntenescu. I'm a developer relations engineer working on lots of things AI, both intelligent experience, is what we're gonna talk about today, and also developer productivity.
- 0:28
So if you wanna talk about that, find me at the booth later on.
- 0:33
Uh, I'm Oli, uh, PM for Android AI. Uh, our work spans a bit of everything. I've been working on this since before we called AI and back when we were calling it just ML.
- 0:42
Um, so we do a bit of everything from helping build features, uh, things that run directly in the OS with applied AI, so things like background tasks that optimize the screen brightness and system memory and everything, through to developer tools and infrastructure to make it easier, people who are building, uh, features on Android, uh, and right down
- 0:59
to the lowest levels of hardware acceleration and OS optimization with many of our silicon partners. So we do a bit of everything, so happy to talk about any of that as well.
- 1:06
Uh, we want to make this a conversation where you're asking the questions, and then we're trying to answer as many as possible of them. Um, but I'm not sure, like, how familiar you all are with, like, what's available on, uh, on Android about how to build intelligent experiences.
- 1:25
How... Would you raise your hand if you already know how to build intelligent experiences on Android with, like, on-device, hybrid, or cloud? Okay. We're gonna do a short, like, TLDR, and then hopefully from there you might also get some ideas of, like, what kind of questions you, you want to ask.
- 1:45
Uh, okay. So if you wanna build intelligent experiences on Android, you can use, uh, on-device, uh, models. You can use hybrid, so use on-device when on-device is available. But otherwise, do the inference on the cloud or just do cloud inference fully.
- 2:01
Um, so when it comes to building on-device, um, uh, experiences, it means that the prompts are processed, uh, directly on the device with no data being sent to the server.
- 2:13
Uh, this is great because of multiple reasons. It means that you're able to, uh, take advantage of local processing, uh, so sensitive data, like, I don't know, banking information doesn't have to leave the device.
- 2:24
You have the ability to do offline work, and of course it means that there's no additional cost for all of the inference.
- 2:33
Uh, so use cases like, you know, sensitive data, like, or personalization or anything that requires, like, maybe a shorter context window, like translations and so on, are things that you can do on-device.
- 2:46
So there are two main ways you can do this in Android. You can use the ML Kit's, uh, GenAI APIs where you get access to Gemini Nano, which is our on-device model.
- 2:57
Or if you need something that's much more customizable, your own models, your own custom models, then you can use LiteRT LLM. There's another talk after this talking about LiteRT LLM, so we're just gonna talk about the ML Kit's GenAI APIs.
- 3:11
So we said that these are the ones that give you access to Gemini Nano. So this is our most efficient models, model for on-device tasks, and it's using the same architecture as Gemma 4 that you probably have heard about that was launched, I think, last week.
- 3:26
But Gemini Nano is optimized for Android devices. So the way, um, uh, Gemini Nano comes on your device is through the AICore system service. So with this it means that you only have one model on the device, and then all of the apps are using that same model through this AICore system.
- 3:45
Because it's, uh, Gemini, uh, Nano, it means that we're optimizing for, uh, the hardware, so you get like, what is it? Uh, lower latency and, um, uh, faster execution for AI tasks.
- 3:59
Yeah. So you can think about this as, you know, imagine you're using a cloud service, right? And everything is kind of provided for you. You don't have to worry about setting up the LLMs, running them on devices, getting your TP inference, et cetera.
- 4:11
Um, you just focus on your feature, your prompt, and then the service provides everything. We're doing the same thing for on-device, right? So we get the models to the device.
- 4:19
We make sure they're optimized, take advantage of the specific hardware that's available on each device at runtime, um, and sort of basically package that all up for you so you don't have to worry about any of that stuff.
- 4:29
Um, and it's also, it has privacy and, uh, safety considerations. So this means that, you know, your requests are not gonna be, I don't know, messed up with all of the other apps' requests.
- 4:39
They run in isolation, and then the input and the output data is not, uh, stored on device at all, so it's all as private and secure as possible.
- 4:49
Okay. So how do you access these, uh, the Gemini Nano? So you do this using the ML Kit GenAI APIs. So actually, the, the GenAI ML Kit APIs are part of the bigger ML Kit APIs where you also get access to, uh, APIs end models for, like, vision and natural language.
- 5:07
Uh, for GenAI specifically, we actually have a bunch of APIs specialized on specific tasks like summarization, proofreading, rewrite, and so on. But the most powerful of them is actually the prompt API.
- 5:22
Um, this means that you're able to send natural language, uh, request to Gemini Nano. For now, it supports text and image as input and text only as output.
- 5:33
So with the prompt API, it means that you can easily use it for, like, stuff like image understanding, uh, content assistance, content analysis, entity extraction. So I would pretty much say whatever use case you wanna do, prompt API is gonna be able to help you.
- 5:50
Okay. What, um, the Gemini Nano models are only available on, like, Pixel 9, Pixel 10, kinda like that generation of devices, not just on Pixel devices, but also on other OEMs.
- 6:02
So what do you do if you wanna get access or be able to use AI on other devices? Well, you can do this by using, um, cloud when the local model is not available for that device, so that increases the reach of, uh, your feature.
- 6:17
So to do this, you would use the Firebase AI logic. Uh, we've launched the hybrid inference a couple of weeks ago. Um, so this means that you're able to decide, like if, uh, we have, uh, Gemini Nano available on device, then you can run that inference on device.
- 6:34
Otherwise, it can run on the cloud. And especially now, I think, with, uh, um, Gemma 4, when the next generation of Gemini Nano will be available, and I think we can already use it w- using the AI Core, uh, preview, uh, it means that you're able to have the-- a similar experience using on device with Gemma-- Gemini
- 6:53
Nano 4, and then also in the cloud with, like, the Gemini Flash, uh, models. But if you want access to even more powerful models like the, uh, the Pro Flash flashlight, and you want to run all of this in the cloud, you can also do this using the Firebase AI logic.
- 7:12
So this gives you access to both Gemini API and Vertex AI and Gem- uh, Gemini Developer API, um, as providers.
- 7:19
Wow.
- 7:20
Okay.
- 7:20
That was a whirlwind.
- 7:21
Yeah, sorry. [laughs]
- 7:23
Um, I think this is amazing. Um, we just wanted to kinda set the scene because with Android, we're really trying to make sure there is a comprehensive offering, right?
- 7:30
So everything that you need, whether it's running things locally on device for low latency responses, for private inference, whether it's going to the cloud for our most powerful models, whether it's something in between, we're building out a solution for you that-
- 7:44
Mm-hmm
- 7:44
... covers each of those points, and we're trying to make those APIs as consistent as possible so that it's easy for you to blend what is needed. Um, so we wanted to kinda give you a, a high-level view of everything.
- 7:54
Um, but we're also happy to go super deep on any of these things. We're happy to talk about use cases. We're happy to talk about what we're seeing in the industry, what we think is interesting, where we're going.
- 8:03
Um, but we really just wanted to set the scene a little bit-
- 8:05
Mm-hmm
- 8:05
... with some of the different things we've got available today. Florina, anything?
- 8:10
Yeah.
- 8:10
Oh, okay.
- 8:11
That's all I said. So I think actually now we're handing it over to all of you. Okay, go for it.
- 8:16
Um, my main question is like, have you checked amount of RAM usage or, like, the battery concerns that we might have-
- 8:24
Mm-hmm
- 8:25
... if we start running Gemini Nano models or, uh, LiteRT, uh, models?
- 8:28
Yeah. A-absolutely. So as Florina mentioned, these models today, we've shrunk them down as much as we could whilst trying to-
- 8:35
Yeah
- 8:35
... maintain all that capability. But that does mean they need flagship capabilities, right, right now. Um, for battery concerns, that's one of the reasons why we've produced AICore, which is to basically optimize everything as much as we can so that you can know that you can rely on that and say, "If it's available, I'm gonna be able
- 8:51
to use it, and I'm gonna get the best performance out of it." Um, yes, there is battery impact. If you are running this nonstop, you are gonna run down the battery fairly quickly.
- 9:00
Um, but what we're finding is that the kinds of use cases where people are using it today are things like, um, you know, a user's coming and asking a question, and they're responding or they're manipulating some data, and it's happening at that point in the flow.
- 9:13
Users maybe use it 10, 20 times a day. That sort of usage is really not, you know, concerning for battery life.
- 9:19
Mm-hmm.
- 9:19
Where we're seeing more batch use cases, so I have a bunch of stuff that I want to process, oftentimes that's not necessarily latency sensitive, so people are using that in the background, perhaps when it's running over charge overnight, and they can basically run that continuously until they've completed all their tasks.
- 9:35
Mm-hmm.
- 9:35
Um, but this is one of the reasons why we're trying to build into the platform so that we can sort of do all those optimizations for you, so you don't need to worry about it.
- 9:41
If, however, you have very specific needs and you want to go with custom models, we have a bunch of tools for profiling, for trying to determine what the impact is gonna be for you.
- 9:50
But of course, it does need you to do some of that work if you're gonna go custom.
- 9:54
Mm-hmm. Cool.
- 9:55
Sorry, can I just give you this hand mic? Is there anywhere that-
- 10:01
Do you want it? Question. And we can just pass that around if you've got a question. Thank you very much.
- 10:05
Um, yeah, quick question regarding-- and two, actually two questions. First one, quick one. Uh, so in the, in the case of using ML Kit-
- 10:13
Mm-hmm
- 10:13
... which will be, I would say, the most optimized one, uh, offering, uh, it means, uh, does that mean that we are using, um, an, a model that is shared cross apps-
- 10:23
Mm-hmm
- 10:23
... on device?
- 10:24
Yeah.
- 10:24
Yeah.
- 10:25
Okay. If yes, then, um, is, um, how are we sure is it managing the scale? Imagine, like, a user have 100 app that actually all, in two years' time, they will all using that, uh, same model.
- 10:39
Uh, how are you handling that to get, I would say, uh, latency as good as possible?
- 10:43
Yeah. So, so that's exactly one of the reasons why we're doing this at a system level, right? If you imagine these models, we're talking about the smallest ones are, like, one gigabyte to be useful.
- 10:53
The ones we're shipping are actually closer to three, four gigabytes in total. Um, it's not really very easy to ship that as an app developer yourself. It really requires you to have an absolutely killer feature to justify that to an end user.
- 11:06
Mm-hmm.
- 11:06
If we put this in the system, we do that once, and everyone can share and benefit from it, well, that shares that cost, right? So that's one of the reasons why we're doing this.
- 11:13
So we have no concerns about hundreds of apps using this because we've centralized that cost. In terms of lots and lots of apps using this to generate features, what we tend to find is that, yes, it does have a battery impact.
- 11:25
But by centralizing that, by queuing things, by making sure that there is some overall system management, we can inform the user of like, "By the way, this app is using quite a lot of quota to do this.
- 11:36
Do you want to do that?" Right? We don't ask that, but as in we're attributing it to the battery impact. And we've seen this time and time again with things like GPS or Wi-Fi.
- 11:44
Mm-hmm.
- 11:45
If the user feels they're getting value out of the app, they're very happy to use that. They're happy to spend their battery on the features they love. If, however, the app is doing something that doesn't provide a lot of value to the user, they might not want to use that, right?
- 11:56
So we think it's really important to just make these capabilities available, make it easy for you to use them sensibly.
- 12:02
Mm-hmm.
- 12:02
And then developers will build amazing features, and users will choose what they want to spend their battery on.
- 12:07
But from a developer perspective, you don't need to worry about that. You just do your inference, and then that's it. And then the AI Core does all of the-
- 12:16
Correct
- 12:16
... the handling.
- 12:16
We handle the scheduling. We handle making sure that-
- 12:18
Mm-hmm
- 12:18
... you know, things are getting queued up appropriately and stuff, so you don't need to worry about that.
- 12:22
The only the probably thing to think about is if there are other apps using that model, they're probably e-ep-- the our prompt will get in the queue, basically
- 12:30
So if you're trying to obviously access it from the background, then yes, you'll be queued. If you're in the foreground, of course, you're gonna have top priority. Um, so whichever app is currently being used actively by the user is obviously going to be prioritized by the system.
- 12:41
Okay. Had a question in the back.
- 12:43
Yeah, I have... Maybe you can hear from me. I have a couple of questions. I have Android 12 on my phone, which is not a Pixel, right?
- 12:52
Mm-hmm.
- 12:52
It's Asus, but doesn't matter. And I have used, um, in this-
- 12:56
A bit closer
- 12:57
... Assistant. Can you hear me better now?
- 12:58
Yeah. Great.
- 12:59
So I have a couple of questions. Uh, the first one is, um,
- 13:04
when I'm asking Google for to do... solve something for me, is it doing locally on my mobile or is it doing remotely?
- 13:11
When you are-- When you say when I'm asking Google, uh, do you mean like the Gemini app or-
- 13:16
Yeah. Whatever, whatever I have on my Android by default. I did not install anything. It is there. I just say, "Hey, Google-"
- 13:21
Mm-hmm.
- 13:21
"... what's the temperature outside?" And it will give me an answer. Is it running this locally or is it running remotely?
- 13:27
So that would be the Gemini app or like the search app that has integration with the Gemini APIs. To be honest, I'm not sure. My expectation would be that it's, uh, running this, uh, remote.
- 13:40
Right. So then my next question makes a lot more sense. How can I then ins-install the things that you have presented here?
- 13:46
Mm-hmm.
- 13:46
And, and second part to this, is it possible to write skills to-
- 13:51
Mm-hmm
- 13:51
... for it locally to improve the kind of answers that it gives me if I notice that it is giving me consistently the wrong answers for whatever reason?
- 13:59
Mm-hmm.
- 13:59
Possible to do that?
- 14:00
So what you're talking about is like two different, uh, s- uh, user journeys 'cause if you're building your own app, you wouldn't interact with the Gemini app. That's a completely different app.
- 14:10
But rather you would interact with the model itself.
- 14:13
Yes.
- 14:13
Therefore, if you're using an on-device model, you don't need to care about the, the cloud. All of that inference will be on device.
- 14:20
Yes. So I'm, I'm asking precisely about that. If I installed what you showed here-
- 14:24
Yeah
- 14:24
... locally, assuming it is possible-
- 14:26
Yes. Yeah. Yeah
- 14:26
... can I then write, uh, write skills to improve the kind of responses it gives me?
- 14:30
I don't think you would write the skills. Um-
- 14:34
No. So I mean, it, it depends on the kind of experience you're trying to build, right? But if you think about a skill, it's just something that you're sticking into the prompt-
- 14:41
Yes
- 14:41
... alongside the rest of your query.
- 14:43
Yes.
- 14:43
So the tools we're providing here are more sort of low level and designed for you to be able to build back on top of that, right? So if, for example, you went and installed something like Pocket Claw or Open Claw on the phone itself-
- 14:56
I see
- 14:56
... that would then basically take those skills, compose them into a prompt, which it would run through this API.
- 15:01
Great.
- 15:01
Um, so what we're focusing on here is building the lower layers to enable you or others to build those things on top.
- 15:08
Thank you.
- 15:10
I guess the battery is dying.
- 15:11
Time.
- 15:13
Sorry.
- 15:14
Five minutes. We started late.
- 15:16
We start. Okay.
- 15:17
Uh, there was one question there. Um-
- 15:20
Um, so, uh, what's the difference between, uh, the model that we have, like the Gemini Nano model versus any model that we have on the, like AI Edge Gallery app?
- 15:29
Mm-hmm.
- 15:29
What level of access can we have? Um, because I don't think we have system file level access, uh, using, uh, any of these APIs or do we?
- 15:36
Okay.
- 15:37
If we want to implement anything via that, let's say via a picture file-
- 15:40
Mm-hmm
- 15:40
... stored in a system, uh, folder, can we have that using these APIs, or is it isolated in its own memory?
- 15:47
Yeah. So the Gemini Nano model itself is completely contained by AICore, right? So you don't need to worry about how to set that up and configure it. You use the APIs to access it, and it's all handled by the system.
- 15:58
In terms of building a Rags-like solution, yes, absolutely, you can do that with the prompt API. Um, we are looking at extending some of the APIs we have, for example, adding an embedding API soon, and that will make it easier for building these kind of Rag-like solutions.
- 16:10
Um, but yes, it's totally possible to do that. In terms of the relationship between AI Edge Gallery and AICore, AI Edge Gallery is a showcase to show you what's possible, right?
- 16:19
You can use AICore backing in that, or you can use custom models. We really want you to see the full breadth of things you can do. Um, AI Edge Gallery, I think, is a really good way of showing the frontier of what you can do, right?
- 16:32
But it requires more uplift. There's more work you have to do to take advantage of those things. AICore is really focused on making it easy for you to just focus on the prompt, focus on the specific feature rather than any of the setup.
- 16:43
And build like production level apps.
- 16:45
Yeah.
- 16:45
Okay.
- 16:46
Cool.
- 16:46
And system access?
- 16:48
And?
- 16:48
Do you have system access through the, uh, code?
- 16:51
When you say system, do you mean like system prompt or-
- 16:53
Files
- 16:54
... files? So-
- 16:55
Like if I-- Like I have taken a lot of pictures today.
- 16:56
Yeah.
- 16:57
Mm-hmm.
- 16:57
I want to then, like, summarize everything together-
- 17:00
Mm-hmm
- 17:01
... let's say.
- 17:01
Yeah.
- 17:01
Make that note.
- 17:02
Yeah. So the ML Kit, uh, GenAI API, the prompt API allows both text and the image input.
- 17:08
Yeah.
- 17:08
So you can give it access, yeah.
- 17:10
So all the things your app normally has access to in terms of files and so on, you can then obviously pass them through the API, uh, and run them just like, just like you would do on a server-based inference, right?
- 17:18
In terms of s- creating the prompt, feeding the information in.
- 17:23
Is there any vectorizing em-embedding model also available for this API? Like I want to vectorize some of my text notes so that they're like, you know, I can get similarity between some other notes maybe.
- 17:32
Yeah.
- 17:33
So does this have that or?
- 17:35
So we don't yet have an embedding API.
- 17:37
Okay.
- 17:37
We will soon.
- 17:38
Cool.
- 17:39
Mm-hmm.
- 17:39
Um, so if you're looking at things like the Gemera embedding model-
- 17:43
Mm-hmm
- 17:43
... um, that's what we're going to make available so that you can use that directly from the API.
- 17:48
Cool. Oh, you want a last question?
- 17:49
Last question.
- 17:50
We're, we're being kicked out.
- 17:51
Thanks. Um, yeah, there's a diversity of devices, you know. Um, and there's also a diversity of models. You showed the prompt API and, you know, embeddings are coming soon, but how do I think about this sort of that matrix of-
- 18:03
Mm-hmm
- 18:04
... you know, device, like what's capable on a given device versus-
- 18:07
Mm-hmm
- 18:07
... like what, what models could be used.
- 18:09
Mm-hmm.
- 18:09
You know, are there more, uh, use cases than just LLM and embedding model, or are there other models that also run on device?
- 18:17
Mm-hmm.
- 18:17
Um, and then will they be widespread across all devices, or do, do I need the latest flagships to, to use those?
- 18:25
So a-as Florina showed, the ML Kit API s- covers a range of things, so text, OCR, vision, all sorts of stuff. Those models, the classical models, are much smaller.
- 18:34
They can run on a super large range of device, like billion-plus devices, no problem. Um, the new ones, the GenAI APIs, those ones require currently fairly flagship devices from the last couple of years.
- 18:44
What we're doing with AICore is we package that up and say, "Hey, you can call this API, and if it's available, you know it's gonna run well." If, however, that reach is not big enough for you and you want to basically do the work to test on a wider range of devices, LiteRT-LLM, which Cormac's gonna talk about
- 18:59
in a second, can help you do that, but you will then need to do the work for testing. Cormac hopefully will talk about some of the tools we have available that make it easier to do that testing, but you will need to do that testing yourself.
- 19:09
For AICore, we cover that all, so you know that if it's covered by AICore, it's gonna run well.
- 19:14
Thanks.
- 19:14
We're gonna be around, um, today and tomorrow, so please come and ask us questions.
- 19:19
Thanks a lot.
- 19:19
Thank you.
- 19:20
Thank you very much. [clapping] [outro music]