From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google

How do you add an agent skill to a phone—and what do you do when the intelligence already on the device does not cover your use case? One option is a simple skill harness on top of Android’s AI Core. Another is to ship a model specialized for your app. Cormac Brick uses tiny LLM to mean a model with fewer than a billion parameters: small enough to embed in an application when you need more customization than the system provides.

Both approaches pursue the same local-inference benefits: lower latency, privacy, offline use, reliability, and potential savings. The architectural question is where the model comes from and how much of its behavior the application needs to own.

Brick introduces himself as a software engineer and tech lead on Google AI Edge. Its stack separates application integration from model execution: LiteRT-LM supplies the language-model harness for an app that downloads or bundles its own model, while MediaPipe supports other model-driven tasks. Both sit on LiteRT, formerly TensorFlow Lite, the cross-framework execution runtime.

Depending on the platform and workload, developers can choose CPU, GPU, or NPU execution. Brick reports that LiteRT supports more than 2.7 billion devices, with a version included in Android itself. Its reach also extends beyond Android: the platform diagram includes iOS, macOS, Linux, Windows, Web, and IoT. Gemma 4 is part of that broader deployment story; detailed platform-performance results are deferred to a colleague’s session.

System generative AI starts with intelligence installed on the device. On Android, Brick’s example is Gemini Nano through AI Core, illustrated with a summarization API. Apple’s intelligence on iOS provides a conceptual parallel. Brick identifies Gemma 4 E2B and E4B as the base models for the Nano offering he describes. The attraction is practical: the model is optimized and preloaded, so using it does not increase the app’s size. If it meets the task’s needs, this is the starting point he recommends.

In-app generative AI gives the developer more control. LiteRT-LM can accompany an app or a webpage, and a smaller model can reach devices that do not offer the desired system model.

That extra work becomes worthwhile for a narrow, highly customized task. The rest of the progression explores both sides: newly available Gemma 4 skills, then the capabilities of tiny models after the preceding six to twelve months of development.

Google AI Edge Gallery is the team’s playground for running models locally on Android and iOS. It demonstrates both tiny models and Gemma 4, using AI Core to provide the Gemma model when that service is available. Alongside skills, the app includes chat, image questions, audio transcription, and support for third-party models such as Qwen and Phi. Built with LiteRT-LM, its Android source also serves as an integration example.

After skipping a longer app video, Brick plays a short restaurant roulette demonstration. The skill loads restaurants, selects a winner, and presents the result through a roulette interface. Gemma 4 runs inside a small agent harness; Brick describes the implementation as something achievable with a few lines of code or a suitable coding-agent prompt. The visible result is an application interaction, rather than just a paragraph generated by the model.

The harness begins with the app’s system prompt and a catalog of skill descriptions. Those descriptions tell the model what it can do without putting every skill’s functions and implementation details into the initial prompt. A load-skill tool retrieves the full instructions on demand.

For a request to show the Google office’s location, the sequence is:

1. The model recognizes map navigation as the relevant skill.
2. It calls the load-skill tool to retrieve that skill’s details.
3. The tool returns the instructions the model needs.
4. The model uses the show-JS tool to display the location inside the app.

The following JavaScript illustrates the catalog/detail boundary for that same map request. The initial prompt receives only the description; the full instructions become a tool result after selection.

const skills = new Map([
  ["map-navigation", {
    description: "Show a requested location on a map.",
    instructions: [
      "Identify the location requested by the user.",
      "Use the show-JS tool to display its map in the app."
    ].join("\n")
  }]
]);

const skillCatalog = [...skills].map(([name, skill]) => ({
  name,
  description: skill.description
}));

function loadSkill(name) {
  const skill = skills.get(name);
  if (!skill) throw new Error(`Unknown skill: ${name}`);
  return skill.instructions;
}

const request = "Show me the location of the Google office.";
const initialContext = { request, skills: skillCatalog };
const toolResult = loadSkill("map-navigation");

Here, toolResult supplies instructions; loading it does not itself display a map. The subsequent rendering operation is a separate part of the harness.

Embedding the agent in an app makes that rendering step useful. A skill can contain JavaScript that the application calls to create an interface. Brick explains the map flow without playing a corresponding map demo, then connects it back to the restaurant roulette: custom JavaScript renders the wheel. On-device inference does not make every skill operation offline; a skill that retrieves external information can still need a network connection.

The Gallery skill authoring guide supports writing a skill by hand, but the team also uses Gemini CLI and Claude Code to create them. Brick reports that the team had made roughly 80 skills. The recursive aspect is practical: a coding agent can use a skill that explains how to author another skill.

The development loop extends onto the device:

1. Create the skill. Follow the documented structure directly or give those instructions to a coding agent.
2. Test through ADB. Brick describes a Gemini CLI ADB skill that lets the coding agent debug and test against an attached device.
3. Publish the files. Put the skill in a personal GitHub repository.
4. Load it into Gallery. Use the app’s ellipsis menu to load a custom skill from its URL.
5. Share it. Post in the repository’s GitHub discussions so others can discover and try it.

Brick shows community examples that had appeared shortly after the feature’s launch the previous Thursday. The distribution mechanism lets users try a new skill without rebuilding Gallery.

The second path begins with a model the application owns. LiteRT-LM runs a single model-format file containing the information needed for execution. It is open source and supports multiple platforms. At talk time, C++ and Java interfaces were available, while Swift and JavaScript APIs were forthcoming; Brick also tied release of Gallery’s iOS source to completion of the Swift work.

The deployment flow is straightforward:

1. Start with a model in Transformers.
2. Export it using LiteRT-Torch.
3. Evaluate it with the desktop LiteRT-LM reference runtime or load it into Gallery.
4. Integrate LiteRT-LM into the application for deployment.

The runtime supports Gemma 4 and other models, so the choice of model remains separate from the application’s inference integration.

For a tiny model, model selection and task design are closely coupled. A prebuilt fixed-function model may already handle vision-language work or transcription. Otherwise, Brick’s guidance is to narrow the task and fine-tune: at around 100–200 million parameters, the model needs a sharply focused job. The team has used synthetic training data to specialize models at this scale for internal and app deployments.

The export demonstration shows a Qwen model running on a desktop GPU through LiteRT-LM run. A separate example uses Apple’s FastVLM-0.5B, optimized for a Qualcomm NPU through the same stack. The 0.5B designation describes the language-model component; FastVLM also has a vision encoder. The examples illustrate two independent deployment choices: which model fits the task, and which available hardware backend executes it.

FunctionGemma makes the specialization strategy concrete. Published with the DeepMind team the preceding December, it is a 270-million-parameter model based on Gemma 3 technology. Brick qualifies its robust function calling with an essential condition: it needs fine-tuning for the task.

Brick reports nearly 2,000 prefill tokens per second and 140 decode tokens per second for FunctionGemma on a Pixel 7. Prefill processes the input context; decode generates the response. The talk does not specify the backend, quantization, or input and output lengths for those measurements. His proposed applications include text-to-function and voice-to-function calling, where a small model converts a request into an action the app can handle.

Alongside a FunctionGemma video explainer, Brick points to the FunctionGemma Tuning Lab, a Hugging Face Space for defining functions, uploading custom data, and fine-tuning the model. This moves the task from asking a general model to infer an application’s conventions toward training a small model specifically for those conventions.

The app-intents example includes actions such as adding a calendar item or an email. Brick recalls approximately 46% success with FunctionGemma out of the box. He initially describes supplying seven functions to the fine-tuning workflow. With a larger model or AI Core, those function definitions might instead go into the system prompt. For this tiny model, the team used Flash to generate a synthetic dataset and then trained with that data using a tuning tool or internal tooling.

Brick reports that fine-tuning produced success rates above 90% for eight of ten functions, with the remaining two in the 80s. The initial seven-function description and later ten-function result are not reconciled, and the talk supplies no held-out split or scoring protocol; these are reported results for the team’s app-intent task, not a single aggregate model-accuracy figure. The improvement comes from specializing the model for the application’s actions, at the cost of dataset generation and training work beyond prompting. Brick presents that extra work as a route to robust deployment at app scale, then briefly points back to the available prebuilt tiny models.

Eloquent provides a second application of the same approach. It is a transcription app with personalization for vocabulary that generic transcription often misses: technical jargon, people’s names, and a user’s preferred keywords. At talk time, Brick says it was available only on iOS and not yet in Europe. Its architectural interest is how the team assembled the transcription pipeline from tiny models.

The pipeline has two stages:

• Automatic speech recognition: a Gemma 3-based ASR engine converts speech into text.
• Text polishing: a second Gemma 3-based model cleans the transcription, including removing fillers such as ums and ahs.

Brick describes each model as having a few hundred million parameters. Chained together, they provide offline transcription that can use a personal dictionary. Rather than requiring one small model to perform every part of the experience, the application combines models with distinct jobs.

Brick presents Eloquent as a production example of what becomes possible after investing in specialization, even though its availability was still limited. It brings the prepared portion of the talk back to the original choice: use system generative AI where its capabilities fit, or build app-level generative AI when the product needs control over the model and its task.

The first audience question asks how many skills tiny models can support before performance deteriorates. Brick’s answer concerns the four-billion-parameter model, rather than the sub-billion models discussed in the fine-tuning section. The team had been experimenting for only two to three weeks, with roughly one week of public availability. Brick says the four-billion-parameter model selects reasonably well among about eight skills enabled by default.

The more useful distinction is between successive turns and a single compound request. A user can ask for a fact through a Wikipedia skill, then ask to see the relevant place on Google Maps. Each turn gives the model a new, explicit task. Brick describes that sequence as robust, while calling multiple skills within a single answer works only sometimes.

The harness is still simple, and the team is still investigating its limits. The demonstrated ability to select and load a skill should therefore be distinguished from reliably planning a multi-skill answer.

The next question starts from an existing deployment workflow: take a Python model, convert it to TFLite, quantize it, and bundle it into a MediaPipe .task file. Where does LiteRT-LM fit? Brick explains that, for stock LLMs, the LiteRT-LM format effectively replaces the .task bundle—a transition he dates to the previous year.

That replacement has a specific scope. MediaPipe .task files remain useful for broader tasks such as face mesh, which can include additional task-specific code. LiteRT-LM instead provides dedicated LLM packaging intended to be simpler and accessible through open developer tooling. It also bundles the tokenizer. The answer establishes the format boundary, although it does not provide a step-by-step migration guide.

The final question concerns CPU performance, clarified after an initial misunderstanding about TPU. Brick gives no CPU benchmark in this answer. He directs developers to the Gemma LiteRT-LM model card, which the team updates with performance numbers for new platforms, and to a colleague’s session covering Gemma and other models on different hardware. An audience member supplies the follow-up time—2:30 the next day—and Brick confirms it. For an actual deployment choice, those platform-specific measurements are the next evidence to consult.