AI Engineer Europe 2026
From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google
Read the talk
Building on-device agents with skills and tiny models
An on-device agent can load skills into a system model or run a specialized model inside the app. Cormac Brick walks through both paths, from JavaScript interfaces to synthetic-data fine-tuning.
From a talk by Cormac Brick
Before you start: Familiarity with model inference, prompts, and function calling will help; no prior experience with Google AI Edge is required.
What if the device’s built-in AI does not do your task?
How do you add an agent skill to a phone—and what do you do when the intelligence already on the device does not cover your use case? One option is a simple skill harness on top of Android’s AI Core. Another is to ship a model specialized for your app. Cormac Brick uses tiny LLM to mean a model with fewer than a billion parameters: small enough to embed in an application when you need more customization than the system provides.
Both approaches pursue the same local-inference benefits: lower latency, privacy, offline use, reliability, and potential savings. The architectural question is where the model comes from and how much of its behavior the application needs to own.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where local inference runs
Brick introduces himself as a software engineer and tech lead on Google AI Edge. Its stack separates application integration from model execution: LiteRT-LM supplies the language-model harness for an app that downloads or bundles its own model, while MediaPipe supports other model-driven tasks. Both sit on LiteRT, formerly TensorFlow Lite, the cross-framework execution runtime.
Depending on the platform and workload, developers can choose CPU, GPU, or NPU execution. Brick reports that LiteRT supports more than 2.7 billion devices, with a version included in Android itself. Its reach also extends beyond Android: the platform diagram includes iOS, macOS, Linux, Windows, Web, and IoT. Gemma 4 is part of that broader deployment story; detailed platform-performance results are deferred to a colleague’s session.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use system intelligence or ship your own
System generative AI starts with intelligence installed on the device. On Android, Brick’s example is Gemini Nano through AI Core, illustrated with a summarization API. Apple’s intelligence on iOS provides a conceptual parallel. Brick identifies Gemma 4 E2B and E4B as the base models for the Nano offering he describes. The attraction is practical: the model is optimized and preloaded, so using it does not increase the app’s size. If it meets the task’s needs, this is the starting point he recommends.
In-app generative AI gives the developer more control. LiteRT-LM can accompany an app or a webpage, and a smaller model can reach devices that do not offer the desired system model.
| Choice | Model delivery | Main advantage | Main constraint |
|---|---|---|---|
| System AI | Preloaded on the device | Optimized inference without adding model size to the app | Available system capabilities |
| In-app AI | Bundled or downloaded by the application | Customization and broader device reach | More integration and model work |
That extra work becomes worthwhile for a narrow, highly customized task. The rest of the progression explores both sides: newly available Gemma 4 skills, then the capabilities of tiny models after the preceding six to twelve months of development.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A restaurant picker in Google AI Edge Gallery
Google AI Edge Gallery is the team’s playground for running models locally on Android and iOS. It demonstrates both tiny models and Gemma 4, using AI Core to provide the Gemma model when that service is available. Alongside skills, the app includes chat, image questions, audio transcription, and support for third-party models such as Qwen and Phi. Built with LiteRT-LM, its Android source also serves as an integration example.
After skipping a longer app video, Brick plays a short restaurant roulette demonstration. The skill loads restaurants, selects a winner, and presents the result through a roulette interface. Gemma 4 runs inside a small agent harness; Brick describes the implementation as something achievable with a few lines of code or a suitable coding-agent prompt. The visible result is an application interaction, rather than just a paragraph generated by the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Load skill details only when needed
The harness begins with the app’s system prompt and a catalog of skill descriptions. Those descriptions tell the model what it can do without putting every skill’s functions and implementation details into the initial prompt. A load-skill tool retrieves the full instructions on demand.
For a request to show the Google office’s location, the sequence is:
- The model recognizes map navigation as the relevant skill.
- It calls the load-skill tool to retrieve that skill’s details.
- The tool returns the instructions the model needs.
- The model uses the show-JS tool to display the location inside the app.
The following JavaScript illustrates the catalog/detail boundary for that same map request. The initial prompt receives only the description; the full instructions become a tool result after selection.
javascript
const skills = new Map([
["map-navigation", {
description: "Show a requested location on a map.",
instructions: [
"Identify the location requested by the user.",
"Use the show-JS tool to display its map in the app."
].join("\n")
}]
]);
const skillCatalog = [...skills].map(([name, skill]) => ({
name,
description: skill.description
}));
function loadSkill(name) {
const skill = skills.get(name);
if (!skill) throw new Error(`Unknown skill: ${name}`);
return skill.instructions;
}
const request = "Show me the location of the Google office.";
const initialContext = { request, skills: skillCatalog };
const toolResult = loadSkill("map-navigation");
Here, toolResult supplies instructions; loading it does not itself display a map. The subsequent rendering operation is a separate part of the harness.
Embedding the agent in an app makes that rendering step useful. A skill can contain JavaScript that the application calls to create an interface. Brick explains the map flow without playing a corresponding map demo, then connects it back to the restaurant roulette: custom JavaScript renders the wheel. On-device inference does not make every skill operation offline; a skill that retrieves external information can still need a network connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Package and deploy a tiny model
The second path begins with a model the application owns. LiteRT-LM runs a single model-format file containing the information needed for execution. It is open source and supports multiple platforms. At talk time, C++ and Java interfaces were available, while Swift and JavaScript APIs were forthcoming; Brick also tied release of Gallery’s iOS source to completion of the Swift work. The runtime repository now lists Swift and web JavaScript as Early Preview, introduced in v0.12. That later status does not establish when Gallery’s iOS source became available.
The deployment flow is straightforward:
- Start with a model in Transformers.
- Export it using LiteRT-Torch.
- Evaluate it with the desktop LiteRT-LM reference runtime or load it into Gallery.
- Integrate LiteRT-LM into the application for deployment.
The runtime supports Gemma 4 and other models, so the choice of model remains separate from the application’s inference integration.
For a tiny model, model selection and task design are closely coupled. A prebuilt fixed-function model may already handle vision-language work or transcription. Otherwise, Brick’s guidance is to narrow the task and fine-tune: at around 100–200 million parameters, the model needs a sharply focused job. The team has used synthetic training data to specialize models at this scale for internal and app deployments.
The export demonstration shows a Qwen model running on a desktop GPU through LiteRT-LM run. A separate example uses Apple’s FastVLM-0.5B, optimized for a Qualcomm NPU through the same stack. The 0.5B designation describes the language-model component; FastVLM also has a vision encoder. The examples illustrate two independent deployment choices: which model fits the task, and which available hardware backend executes it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach a small model the app’s functions
FunctionGemma makes the specialization strategy concrete. Published with the DeepMind team the preceding December, it is a 270-million-parameter model based on Gemma 3 technology. Brick qualifies its robust function calling with an essential condition: it needs fine-tuning for the task.
Brick reports nearly 2,000 prefill tokens per second and 140 decode tokens per second for FunctionGemma on a Pixel 7. Prefill processes the input context; decode generates the response. The talk does not specify the backend, quantization, or input and output lengths for those measurements. His proposed applications include text-to-function and voice-to-function calling, where a small model converts a request into an action the app can handle.
Alongside a FunctionGemma video explainer, Brick points to the FunctionGemma Tuning Lab, a Hugging Face Space for defining functions, uploading custom data, and fine-tuning the model. This moves the task from asking a general model to infer an application’s conventions toward training a small model specifically for those conventions.
The app-intents example includes actions such as adding a calendar item or an email. Brick recalls approximately 46% success with FunctionGemma out of the box. He initially describes supplying seven functions to the fine-tuning workflow. With a larger model or AI Core, those function definitions might instead go into the system prompt. For this tiny model, the team used Flash to generate a synthetic dataset and then trained with that data using a tuning tool or internal tooling.
Brick reports that fine-tuning produced success rates above 90% for eight of ten functions, with the remaining two in the 80s. The initial seven-function description and later ten-function result are not reconciled, and the talk supplies no held-out split or scoring protocol; these are reported results for the team’s app-intent task, not a single aggregate model-accuracy figure. The improvement comes from specializing the model for the application’s actions, at the cost of dataset generation and training work beyond prompting. Brick presents that extra work as a route to robust deployment at app scale, then briefly points back to the available prebuilt tiny models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Chain specialized models for offline transcription
Eloquent provides a second application of the same approach. It is a transcription app with personalization for vocabulary that generic transcription often misses: technical jargon, people’s names, and a user’s preferred keywords. At talk time, Brick says it was available only on iOS and not yet in Europe. Its architectural interest is how the team assembled the transcription pipeline from tiny models.
The pipeline has two stages:
- Automatic speech recognition: a Gemma 3-based ASR engine converts speech into text.
- Text polishing: a second Gemma 3-based model cleans the transcription, including removing fillers such as ums and ahs.
Brick describes each model as having a few hundred million parameters. Chained together, they provide offline transcription that can use a personal dictionary. Rather than requiring one small model to perform every part of the experience, the application combines models with distinct jobs.
Brick presents Eloquent as a production example of what becomes possible after investing in specialization, even though its availability was still limited. It brings the prepared portion of the talk back to the original choice: use system generative AI where its capabilities fit, or build app-level generative AI when the product needs control over the model and its task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Selecting a skill is easier than coordinating several
The first audience question asks how many skills tiny models can support before performance deteriorates. Brick’s answer concerns the four-billion-parameter model, rather than the sub-billion models discussed in the fine-tuning section. The team had been experimenting for only two to three weeks, with roughly one week of public availability. Brick says the four-billion-parameter model selects reasonably well among about eight skills enabled by default.
The more useful distinction is between successive turns and a single compound request. A user can ask for a fact through a Wikipedia skill, then ask to see the relevant place on Google Maps. Each turn gives the model a new, explicit task. Brick describes that sequence as robust, while calling multiple skills within a single answer works only sometimes.
| Interaction | Reported behavior |
|---|---|
| Select one of the default skills | Reasonably good |
| Use different skills across conversation turns | Robust |
| Coordinate multiple skills in one answer | Works sometimes |
The harness is still simple, and the team is still investigating its limits. The demonstrated ability to select and load a skill should therefore be distinguished from reliably planning a multi-skill answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What changes from MediaPipe, and where to find benchmarks
The next question starts from an existing deployment workflow: take a Python model, convert it to TFLite, quantize it, and bundle it into a MediaPipe .task file. Where does LiteRT-LM fit? Brick explains that, for stock LLMs, the LiteRT-LM format effectively replaces the .task bundle—a transition he dates to the previous year.
That replacement has a specific scope. MediaPipe .task files remain useful for broader tasks such as face mesh, which can include additional task-specific code. LiteRT-LM instead provides dedicated LLM packaging intended to be simpler and accessible through open developer tooling. It also bundles the tokenizer. The answer establishes the format boundary, although it does not provide a step-by-step migration guide.
The final question concerns CPU performance, clarified after an initial misunderstanding about TPU. Brick gives no CPU benchmark in this answer. He directs developers to the Gemma LiteRT-LM model card, which the team updates with performance numbers for new platforms, and to a colleague’s session covering Gemma and other models on different hardware. An audience member supplies the follow-up time—2:30 the next day—and Brick confirms it. For an actual deployment choice, those platform-specific measurements are the next evidence to consult.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore local models, function-calling demos and agent skills, with app installation links and Android source.
Instructions for creating custom skills for Google AI Edge Gallery.
Runtime source, deployment guides and release information for running language models on edge devices.
Google's model card and weights for the small function-calling model used in the talk.
Setup and usage instructions for defining tools, importing training examples, fine-tuning and evaluating FunctionGemma.
Further reading
- Agent Skills with Gemma 4Article
Google AI Edge's launch article explains Gallery skills for retrieving information and producing interactive content.
Original research on FastVLM's efficient vision encoder and reduced time to first token.
Read the complete timestamped transcript
- 0:00
[upbeat music] Yeah.
- 0:15
So while we wait for it to come up, because I know we're short of time, uh, I'm gonna talk about, um, uh, agents on device. So I know whoever asked the question about skills and AI Core, we have an answer to that.
- 0:27
Uh, we've built a simple skill harness on top of AI Core that you can build skills on. Be able to show that. Also gonna talk about TinyLLMs, uh, which are...
- 0:36
We would call LLMs that are, like, smaller than a billion parameters, that are small enough to build into your app if you want to have more customization or you want to do something that isn't already available for you in AI Core.
- 0:46
So that's the gist. So, uh, quick overview of AI Edge. Well, how we think about, like, small language models, uh, TinyLLMs, and system gen AI. Then we're gonna take a quick look at agent skills, which is something we can build on top of, of kind of system gen AI or the new models that are coming down the
- 1:03
pipe. Um, and then we're gonna take a quick look at tiny models.
- 1:10
So that's that. Okay. So I'll... Yeah. Cool. Yep. I'll... Okay. Yeah. Feel free. [chuckles]
- 1:21
Um, okay. So AI Edge, uh, SLMs and TLMs. Um, okay. So I think, uh, Ollie already covered this. We know it's great to do things on device. Latency, privacy, offline use, reliability, or savings, depending on thing.
- 1:34
These are all motivations to do things locally.
- 1:37
Um, me, by way of intro, didn't really do this. Um, I kind of... I'm a software engineer and kind of tech lead working on the Google AI Edge stack.
- 1:46
So that's, um... We have MediaPipe, which is an asset some people may be familiar with. We have LiteRT-LM, which is a LLM harness that you can integrate with your app, um, where you download the model and ship the model with your app.
- 1:59
And then we also have kind of LiteRT as a runtime that supports both LiteRT-LM and MediaPipe. Uh, it's kind of formerly known as TensorFlow Lite, which is a kind of cross-framework runtime for running, um, models.
- 2:10
And all of that can run on CPU, GPU, or NPU, uh, depending on the platform and depending what's best. And you as a developer get to choose. Uh, yeah.
- 2:19
It's already trusted at scale. Um, like the LiteRT runtime, there's a version of that built into Android OS. Uh, lots of Android apps already use it. So, so it does, uh, supports over two point seven billion devices.
- 2:31
Like, lots and lots of daily invocations and lots and lots of Android apps, uh, leverage this.
- 2:37
Uh, but also works far beyond Android as well. So we support all of these platforms. Um, and, uh, for example, Gemma is available on many of these platforms. Our team, like, uh, is giving another talk tomorrow, so you can hear more about Gemma performance on all of these types of platforms and how we're able to do, uh,
- 2:54
really useful things with the latest Gemma Four models.
- 2:58
Um, but then building on Ollie's and Florina's talk, this is kind of key idea, is, uh, we have, um, system-level gen AI, which is something that will be pre-installed into the system.
- 3:09
So there's Gemini Nano via AI Core. This is an example of the summarization API. Um, Apple also has something going on with their intelligence on iOS that I probably know a lot less about.
- 3:19
Um, but as a concept, right, um, as an app developer, when you go to build a mobile app, this is kind of one choice, is there will often be an el- uh, some form of intelligence built into the system that you can leverage, um, which is, uh, you know, highly optimized, as kind of Ollie and Florina covered,
- 3:36
um, that's available for use with your app. Um, then, so this is kind of typically like small language models. Like, for, for Nano, it is the Gemma Four E2B and E4B are the base models for, for what we ship there.
- 3:52
Um, that's really capable, highly optimized, preloaded with device. If you can use it, it's great. Your app doesn't get any bigger. Um, and if it meets your use case needs, it's a great place to start.
- 4:01
If you want, like, more, uh, if you have a more specific task, um, that you want to do that's kind of highly customized or something really boutique, um, you can use in-app gen AP-- gen AI.
- 4:12
Uh, so that's with the LiteRT-LM runtime. Uh, that can be loaded with your app or even your webpage, right? Um, and this offers kind of a higher degree of customization and reach.
- 4:22
Like, it's definitely more work, um, but, uh, yeah, you've kind of access to, uh, smaller models that can run on lots of devices, um, and full customization. So it's clearly, uh, a lot more work, but it's the other option that's available.
- 4:36
Okay. So, uh, rest of the talk, uh, 15 minutes, gonna cover two key ideas. One is, um, hey, how do you do skills on device? 'Cause this is something new that we can do with Gemma Four.
- 4:48
It came out last week. We have a few examples of that. This is one key idea. The other idea I wanna cover is, hey, for tiny models, what can you actually do with those types of models today?
- 4:56
Because we've actually made a lot of progress in this in the last six to 12 months. So I kind of just wanna share what's the state-of-the-art with TinyLLMs, and if you want to use one in your app, how do you go about that?
- 5:06
Okay. So this is... Wow, there's a lot on this screen. This is, um, uh, an app that our team have developed that works on both iOS and Android for running LLMs locally.
- 5:19
Um, and here we show both really tiny LLMs, so you can see what they can do. But also, 'cause Gemma Four just came out, we're also using this to showcase what, uh, how Gemma Four can work, um, on Android and iOS as well.
- 5:33
And this actually builds on AI Core. When AI Core is available on the device, it will use AI Core to kind of, uh, provide the Gemma model for the app.
- 5:41
So skills is the thing I want to kind of go into deeply today. But there's a bunch of other things in the app, like you can do AI chat, you can ask image, you can do audio scribe, and there's lots of example models, and the app also supports freebie models, like kind of, uh, Qwen or Phi or
- 5:57
these types of models. If you just wanna load a model, get a feel for how it performs on device. Um, and this app is also open source in Android, and it's built using LiteRT-LM.
- 6:07
So it's both a neat way for you to try things out, but also if you're keen, you can kind of dive into the code and see, hey, how does it all kind of hang together, um, as an example for LiteRT-LM Okay.
- 6:18
But, uh, we're gonna dive into skills 'cause this is kind of a topic du jour. Um, okay, I'm not gonna play this video 'cause I don't have enough time.
- 6:27
But yeah, this, uh, app is available Android, iOS, uh, code available on GitHub as well.
- 6:34
Um, there we go. Let's take a picture.
- 6:38
Okay. Um, okay. And the app is called, uh, Google AI Edge Gallery.
- 6:44
So this is the video we will watch 'cause it's shorter and meets my time budget, and we don't... Could we get sound? Or an agented task. Sorry. I'll go- Hey, Gemma.
- 6:55
So this uses a restaurant roulette skill, and we'll see how that's built in a moment. It loads the restaurants, selects one. Winner, right? So that's an example of something neat that you can build like, um, with a simple agent harness on top of Gemma 4 that's like, um, really just a few line...
- 7:18
Like, pretty easy to do with a few lines of code or a few lines of the right, uh, vibe coding prompt, as we'll see in a minute.
- 7:25
Okay. Ugh, okay. Here. I don't know how that... Ah, yeah. Okay.
- 7:33
Uh, dun, dun, dun. Okay, I kinda got lost a little there. Okay. All right. Sorry. Back, back, back to where we were supposed to be. So what's actually happening under the hood?
- 7:43
So like I was saying, this is built on like, um... This is built just using a prompt, right? And here you can provide... We have our own system prompt in our app.
- 7:53
Uh, then we also put the skill descriptions into the prompt, so the, um, so the model is aware of the types of skills it can use, but it doesn't have to see all of the functions and details of the skill.
- 8:04
That's only kind of loaded on demand. Um, so we actually have a, a loads, uh, a load skill tool call built into the model that then, like selectively... So if you say, "Hey, can you show or select the location of the Google office?"
- 8:17
It'll then know, "Wow, I should use the map skill." It then loads the skill for map navigation. Um, the tool responds, and then it uses the show JS tool, um, to show you the location, um, in the app as well.
- 8:32
So one of the things that's neat about being in an app is you can put simple JavaScript into the skill that we then call as part of the skill.
- 8:40
Uh, so this is how... Like, I don't have the corresponding demo for this, but this would kind of pop up a nice, um, uh, kinda like JavaScript UI of kinda Google Maps to kinda just show you, uh, in the app right there.
- 8:51
Uh, similar to the restaurant roulette, that was a custom JavaScript, uh, to do the rendering, to do the, uh, roulette wheel piece.
- 8:59
Okay. So you can create your own skill as well. Um, the app supports this. Uh, sorry. I'll... Yeah. Instructions on GitHub. I don't know if I can pass this page too fast.
- 9:09
Uh, but also, um, and to create your own skill, there's full instructions there if you wanna kind of handwrite it out. Uh, this works really well though. Uh, so we can use skills to write skills.
- 9:18
So we have, um, Gemini CLI or Cloud Code, like our team have done like about 80 skills. They've had a lot of fun with it. Um, so this is an example of a prompt which works really reliably.
- 9:30
Um, in Gemini CLI, we actually have an ADB skill as well, um, that, that we, our team uses a lot. So you can even debug and test by saying, "Hey, you have access to a device, um, via ADB," and you can, um, also ask to test, uh, that.
- 9:45
So this type of thing actually works really, really well, and it's fun, and you can then create a skill. And then in the app, there's a dot dot dot button, and you can go to load your own skill from a URL if you kind of publish it to your custom...
- 9:56
to your own GitHub. Uh, it's kind of really easy to do from within the app. Uh, you can then also, uh, let us know in our discussion on GitHub, um, that you've created a skill, and then other people can check out your skill and kinda use that as well.
- 10:09
So these are some things. This has only been out, like, since last Thursday, but these are some example skills that the community have built. So feel free to do it and tag it up here.
- 10:18
Okay. So that's skills. So 10... the last 10 minutes, we are gonna spend on TLMs. Um, or probably more ideally maybe five or six of the minutes so there's time for questions.
- 10:28
Okay. So LiteRT-LM, this is the runtime that we have that, um, we use for running models. It runs models in Lite or TLM format, which is a single file that packages everything we need to know about the model in order to be able to run it.
- 10:43
Uh, it's open source, it's fast, and it works on multiple platforms. Uh, and there is a Swift API and a JavaScript API coming soon. At the moment, if you go to the GitHub, you can see the C++ and Java version.
- 10:56
And when we publish the Swift version, we will also publish, um, we'll also open source the iOS app at that point in time. So if you go to gallery for...
- 11:04
At the moment, you can only see the code for Android, but, um, hopefully in the next few weeks, um, we're, uh, we can get the Swift work finished, have a really good API, and then we'll be able to, um, open source that as well.
- 11:18
So yeah. And it supports Gemma 4 as well on all of these devices. Uh, also supports loads of other models, but understandably Gemma 4 is our favorite.
- 11:28
Um, so then to deploy a tiny model, what do you do? So typically starting Transformers, we then have a package called LiteRT-Torch that, um, can help you export the model and then LiteRT-LM.
- 11:39
There's actually a reference version of that that you can use on your desktop as well if you wanna try out a model. Um, you can either try it out for desktop, or you can load it into the gallery and see it perform there.
- 11:49
And then you can, uh, deploy with LiteRT-LM. It's worth noting for smaller models, you would either pick a fixed function model like a visual language model or a transcription model or something like this.
- 12:01
So there are some pre-built models available on our, on our, um, on our, uh, Transformers page that you can use. But, uh, something we also see that's really common is people fine-tuning models, um, because certainly once you get down to like 200 or 100 million parameters- For that model to work, it needs to have a very narrow
- 12:19
and focused task, and, um, we've had a lot of success deploying those models internally and in an app, a different app that you're gonna see in a minute, um, by doing kind of fine-tuning using synthetic data.
- 12:32
So this is what the export and in-inference flow looks like. So this is, um, this is showing... Okay, on the left-hand side, it's showing us exporting a Qwen 6 model and then running that, um, on desktop with LiteRT-LM run, so you can just see how that behaves using a GPU, for example.
- 12:48
Uh, the right-hand side is showing a different example, which is Apple's FastVLM. Um, this is a visual language model that's only 500 million parameters, and this is optimized and running on, like, a, um...
- 13:02
This is running on the Qualcomm NPU, um, that's also available through our stack, uh, NPU optimization. So this is an example of that happening end-to-end. And this is running really quick 'cause it's using hardware acceleration, and this is...
- 13:13
model is just, uh, that particular model is 500 million parameters, by way of example. Another example that we've spent a bunch of time with, uh, the DeepMind team on was publishing FunctionGemma, something we published last, uh, December.
- 13:28
This was, uh, based on Gemma 3 technology. Uh, this is only 270 million parameters, um, but it's robust function calling when fine-tuned. Typo there. Um, and this is then...
- 13:40
It's small, and it's really fast, even on legacy devices. So if you go all the ways back to a Pixel 7, this still, uh, can process almost 2,000 tokens per second prefill and 140 decode.
- 13:49
So it's really useful for lots of, um, uh... It's really useful for lots of simple use cases, like you can do text-to-function calling or voice-to-function calling using this size model.
- 14:00
And there is a whole YouTube video on this called FunctionGemma, uh, if you want to find out lots more details about how to do it. We also, uh, from, um, uh, have a FunctionGemma fine-tuning lab.
- 14:14
So if you search FunctionGemma fine-tuning lab, I don't have, uh, here. Um, this is available as a Hugging Face space, so you can kind of import. You can define functions, upload your own data, and, um, see how to kind of fine-tune FunctionGemma.
- 14:29
And this is kind of recommended for really high, um, for really robust function calling. So we have an example in the app called, um, like, app intents, where it'll do, like, this, the thing you saw previously of, like, add calendar or add email.
- 14:44
So when we took FunctionGemma out of the box, our success rate in that was, I think, 46% or something like that. Then we put it through this fine-tuning flow, where we were like, "Hey, we have these seven functions."
- 14:56
Um, and instead of providing that via a system prompt, which is what you would do if you were using a larger model or if you're on a device with AI Core, for example.
- 15:04
Um, but, um, you instead need to kind of synthetically create a dataset, right? Is, is typically the workflow. We use Flash to synthetically create a dataset, um, upload it to this type of tool, or we obviously have our own internal tools.
- 15:18
But that then got that 46%, um, to over... Like, it was over 90% for eight of the 10 functions we were trying, and two of the functions were a bit lower in the kind of 80s.
- 15:28
Um, so you can get really robust and reliable function calling using this fine-tuning workflow. Yes, it's, it's a bit more work than just prompting a larger model, um, but it does allow you to kind of ship something robust in your app at scale.
- 15:43
Oh, sorry. Going the wrong direction. Oh, yeah. So then prebuilt, uh, tiny models are here. Um,
- 15:52
yeah. Okay. I'm going to pull... Stop for questions. Okay. So we have... And I, I don't wanna go into this in too much detail. We've another app. Uh, I'll just speed run this for one minute.
- 16:01
We also have another app called Eloquent, which is a transcription service. Um, but what's more interesting than the app was just an example of, like, how we built that.
- 16:11
So it also supports things like personalization. So it does transcription with, with your own favorite keywords. So if you use a lot of, like, tech jargon or a lot of people's names, transcription service d-don't always get that correct.
- 16:22
Um, sadly, this is only available on iOS and not available in Europe yet, right? So this will be increasingly available, uh, soon. But the more interesting thing for the purpose of this conversation is under the hood, this is something we've built using tiny LLMs ourselves.
- 16:36
So this uses a ASOR engine that we have built based on Gemma 3 technology, and then it also has, like, something we call, like, a text polishing engine that we've also built, uh, with Gemma 3 technology.
- 16:47
And both-- Each of these models are only a few hundred million parameters, uh, but chained together they can create a really compelling offline, like, offline transcription service that is able to leverage your personal dictionary, right?
- 17:00
And also, like, the polishing also removes ums and ahs and that sort of stuff, right? Which is also a common gripe with kind of, um, kind of offline transcription apps.
- 17:09
But yeah. So not really available widely yet. It'll be available incre- But for the purpose of this conversation, it's more just sort of like a proof of life example.
- 17:18
So, like, this does work in production, uh, once you put in the effort to kind of fine-tune a model, and you can create pretty compelling things.
- 17:26
Okay. So then it's not available on iOS in Europe, so that's not true. So yeah. Takeaway is system gen AI, app gen AI. That's the kind of the, uh, overall, um, yeah, overall kind of message and wrap-up.
- 17:36
Happy to take questions. I have three whole minutes. Uh, I think person there was first.
- 17:41
Yeah, yeah. So, uh, talking about skills. So, uh, do you personally, maybe your team, how many skills can you start with these tiny models before performance starts to deteriorate?
- 17:52
Um, yeah. We are still putting models on the clock there. [chuckles] Uh, so we've literally been playing with the model for about two to three weeks now. It's been in public for about one week.
- 18:02
We, we see, like, within a single conver- So we can provide, like, certainly for the four billion parameter model, like, if you d- By default, we enable about eight skills, and it's able to choose between the eight skills reasonably well, right?
- 18:15
Um, within a conversation- You're able to say, "Hey," like, you know, like, um, like, "Find me out a fact on using a Wikipedia scale." Then, "Oh, wow. Show me where that is on Google Maps."
- 18:27
So if you have a conversation that uses scale, scale, scale, that works really robustly. The thing we're still working on that's harder is through a single, like, um, interaction with the app for the app to know to call multiple skills as part of a single answer.
- 18:40
Ooh. That's, um... That works sometimes, right? And we're still-- Like, that's something we're still figuring out how to make that more robust, right? Um, but yeah. Like, it's all in a...
- 18:49
Just our, like, our agent harness thing is really simple, so I'm sure we'll figure that out. But we're still kind of discovering the limits of how far we can push the model.
- 18:58
Uh, yeah.
- 18:59
Uh, is there a particular migration format of documentation available when moving from MediaPipe, uh, inference API to, uh, LiteRT-LM? The reason I'm asking is currently the process which I used to follow-
- 19:10
Yeah
- 19:10
... is if I have a model that's in Python and I convert it to TFLite-
- 19:14
Yeah.
- 19:14
... quantize it, bundle it. But the bundling happens in MediaPipe, MediaPipe task file format, I think, or task files.
- 19:20
Yeah.
- 19:20
And that can be imported to the AI Core ML app. So LiteRT-LM, where does it fit in the stack? Uh, is it-
- 19:26
Yeah. LiteRT-LM file format for, just for stock, um, uh, LLMs is effectively a replacement for .task file. That's a transition we made last year. .task files are still useful for things like, um, the c- like, a task file creates more things than just an LLM model, right?
- 19:44
So there is like a face mesh task, and obviously that has a lot of other code as well. But for LLMs, we want something dedicated, simpler, that people can use with open developer tooling.
- 19:53
So it, it does bundle the, uh, tokenizer as well?
- 19:56
Yeah. So yeah. So it bundles things like the tokenizer, but it's just the LLM model. Okay. Uh, yeah.
- 20:01
What about CPU? How is the performance on TPU or...
- 20:04
On TPU?
- 20:06
CPU.
- 20:07
CPU. Um-
- 20:07
'Cause I saw, I saw it, it, it also-
- 20:09
Yeah. So there is another talk tomorrow, uh, from some of my colleagues, uh, including Wei, who's here in the second row. Um, and that has lots of performance data, uh, on Gemma and various models on, um...
- 20:22
Yeah. You can also check out our model card in the meantime if you search Gemma LiteRT-LM model card. We, um, we keep a kind of running... Like, we update that whenever we have new performance numbers on new platforms, so there's a lot of comprehensive data there as well.
- 20:36
Awesome.
- 20:36
Cool. All right. I'm at zero seconds and it's flashing at me.
- 20:40
Yeah. At two thirty tomorrow.
- 20:42
Yeah. Yeah. Oh yeah, two thirty. Sorry. Chin Chin's here. Sorry, Chin Chin. Didn't see you there. [audience applauding] All right. Thanks, all. [upbeat music]