TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM

How much intelligence can an application put on a phone—and what should it ask that intelligence to do? Cormac Brick approaches that question through Google AI Edge, which builds infrastructure for both Google products and open-source deployment, working closely with the Gemma team. His own path began with running GoogLeNet on a USB accelerator attached to a Raspberry Pi at NeurIPS 2016, continued through Intel laptop NPU architecture, and brought him to Google’s edge-AI team. The workshop follows two paths: general-purpose local models extended with skills, then tiny models specialized and shipped inside applications.

Live voice translation makes the deployment question concrete. Brick describes a Pixel feature his team shipped the previous year: translating speech while a conversation is happening. The latency budget makes a cloud round trip difficult to accommodate; local inference becomes part of meeting the user-experience requirement. Messaging assistance creates a different reason to stay local: users want their messages to remain encrypted and on their phones. Offline operation removes connectivity as a dependency, while laptop users are also experimenting with small models for portions of desktop agent workflows to reduce service costs.

The underlying stack separates application components from inference. MediaPipe supports features such as Photos Unblur and YouTube Shorts effects. LiteRT, formerly TensorFlow Lite, supplies general inference; LiteRT-LM adds the machinery needed to run language models. The broader stack serves Android system services and third-party apps, but also reaches iOS, macOS, Linux, Windows, web, and embedded devices. CPU and GPU deployment can share a model artifact; an NPU requires a specially compiled artifact. That distinction matters for the privacy-focused assistants, voice agents, and local tool-calling workflows built above it.

A restaurant app should not necessarily download its own multibillion-parameter model just to help someone choose dinner. One deployment pattern puts a shared foundation model in the operating system. Brick describes these system models as typically having 2–5 billion parameters, with Android’s AICore exposing capabilities such as summarization and prompting, and Apple Intelligence following a parallel approach. At the workshop, this route primarily serves premium devices.

For in-app deployment, the task becomes the unit of design. Brick reports useful results from models in the 100–500 million parameter range on bounded work such as summarization, transcription, and voice-to-action. A small enough task can make a tiny model useful to many more users.

FunctionGemma illustrates that specialization: a 270-million-parameter model released in December for function calling. The team adapted it to ten Android-relevant voice-to-function actions. Brick initially reports approximately 85–90% reliability on an internal evaluation. The important engineering requirement is the adaptation step: in his team’s experience, models below roughly 500 million parameters generally need task-specific fine-tuning for production reliability, while some models around that size and above can handle broader tasks.

Gemma 4, launched the week before the workshop, supplies the larger models used for the skills demonstrations. Its E2B and E4B names describe effective parameter sizes, not total parameter counts. Brick explains that approximately two or four billion parameters must stay resident in RAM, while additional parameters serve embedding lookups.

The runtime memory-maps the embedding and per-layer embedding tables. During the autoregressive loop, it accesses the rows needed for the current token instead of loading the entire table. Brick describes those accesses as hundreds or thousands of bytes for the next-token step; the operating system can then evict pages associated with older accesses. This separates the model’s total stored weights from the working set that must remain resident throughout inference.

At the workshop, E2B and E4B are available for experimentation and on the future AICore roadmap, rather than already integrated into AICore. Brick points to the AICore and DeepMind sessions for roadmap and architecture details. Larger sibling models also matter at the edge, but target consumer laptops with more memory—he gives a machine with roughly 32 GB of RAM as an example.

The capability change that unlocks skills is the combination of thinking and function calling. A developer can describe a skill, give the description to the model, and let it choose how to use the capability. E2B and E4B accept audio, images, and text; the larger launch models accept images and text. This generation also moves to Apache 2.0 licensing, a practical change for downstream deployment.

A LiteRT-LM file packages the LiteRT model with its tokenizer and other components required to run it. The portable package serves mobile, desktop, and embedded deployments; image input makes simple visual inspection and other IoT applications especially interesting. Performance remains an active optimization effort involving Google’s team and partners including Intel, Raspberry Pi, and Qualcomm.

Brick cites thousands of tokens per second for E2B on a high-end Android GPU and a MacBook, and about 133 tokens per second on Raspberry Pi. These figures need the prefill versus decode distinction: prefill processes an input prompt, while decode generates successive output tokens. The later LiteRT-LM performance documentation identifies the matching Raspberry Pi 5 16GB CPU result as 133 prefill tokens/s and 8 decode tokens/s, and identifies the Android and Mac figures in the thousands as prefill. The later table clarifies the metric, but does not establish the workshop’s complete benchmark configuration.

The Raspberry Pi example is aimed at simple image analysis with reasonable latency. A Qualcomm IoT and robotics development platform takes another route, using its NPU for acceleration. E4B runs across multiple device classes too, but its larger parameter workload produces proportionally lower prefill and decode performance. Model portability therefore does not eliminate the need to measure the workload on the hardware that will actually ship.

Google AI Edge Gallery is the workshop’s mobile test environment. Available on Android and iOS, it offers chat, image questions, audio transcription and translation, audio-to-function calling, and agent skills. Skills extend what the model can do beyond producing a better answer to an existing multimodal task.

The first recorded workflow starts with a mood journal. The user logs an entry that includes eight hours of sleep and plans to spend time with Amy, then asks for a trend analysis over the previous seven days. The model records observations and later reads the diary to summarize them. A subsequent request asks for the day’s calendar in a bullet list. The implementation uses skill instructions and callable JavaScript; it does not require a custom fine-tune for the journal interaction.

The conversation expands into map lookup and Wikipedia queries for current Oscars information, including best picture and nominations. The journal skill can add, remove, and query entries; the Wikipedia skill supplies knowledge newer than the model’s training. Next, a breakfast image becomes input to a mood-music skill that calls a web service to compose and play lo-fi music. Local model inference does not make every skill offline: the music service is an explicit network dependency.

Skills can also change the output format. Instead of returning three summary bullets as plain text, JavaScript can display them as cards or flashcards. The model can select that presentation when it appears useful. This gives skills three distinct jobs: retrieving information, performing actions, and presenting results through something richer than a chat message.

Loading every tool’s full documentation into a small model creates a large context before useful work begins. Gallery instead uses progressive disclosure. Initially, the model sees one-line skill descriptions. If a description matches the request, it calls a tool to load that skill’s detailed instructions and available functions. Brick contrasts this with eagerly supplying every function definition: a condensed context saves tokens and gives a lightweight model less irrelevant material to reason over.

The same structure supports several kinds of extension:

• Input tools: Wikipedia or a weather service supplies additional information.
• Output tools: Maps and cards provide new presentation formats.
• Domain knowledge: A customer CRM or local retrieval system supplies application-specific data.

A skill contains SKILL.md and optionally scripts or assets. Its metadata supports discovery; the full instructions arrive only when needed. A PDF skill, for example, can advertise text-and-table extraction without putting its extraction procedure into every conversation. Custom JavaScript then provides a way to extend both Android and iOS applications.

The execution loop is small enough to follow directly:

1. Add enabled skill descriptions to the application’s system prompt.
2. Let the model select a relevant skill from its metadata.
3. Call the load-skill tool and return the contents of SKILL.md into context.
4. Let the model call the run-JavaScript tool using the newly loaded instructions.
5. Execute the skill’s JavaScript and return its result to the conversation.

The load operation changes what the model knows how to invoke; the execution operation performs the work.

Constrained decoding strengthens the tool-call boundary. The runtime applies constraints while generating a tool call, rather than constraining all natural-language output. Because it knows the finite set of permitted tools, it can enforce something more specific than syntactically valid JSON: the call must match an allowed tool. Brick says this work began while porting FunctionGemma and is particularly helpful for approximately two-billion-parameter models. The incremental benefit becomes less important with much larger, more capable models.

Gallery lets users toggle skills, controlling even which short descriptions remain in context. One example loads a virtual piano whose keys can be tapped to play sounds. Custom skills can be imported from a URL, and skills that call external services can ask users for API secrets. Community submissions live in GitHub discussions, with selected skills promoted into the app’s featured collection. The animal-introduction example finishes with a lion card containing a photograph, description, and Play Sound control.

Underneath, an orchestrator maintains a skill registry and exposes three basic tools: load a skill, run JavaScript, and run an intent. Skill content may include persona and scenario instructions for role-play, entirely local JavaScript for an offline experience, or a web API call such as the music composer. Android native intents provide another execution route. Brick offers Wi-Fi control as an example, but native capabilities depend on what the host application implements and exposes; a skill file alone does not grant arbitrary operating-system access.

Restaurant roulette combines language understanding, retrieval, and an interactive view. The request is spoken in French: find a French restaurant in San Francisco, but respond in English. Brick says the team produced about 80 skills using tools including Antigravity and Gemini CLI. This skill requests an API key, searches for ten restaurants, returns location and cuisine information, and renders a roulette wheel in a web view.

The workshop describes a requireSecret flag and an index.js file. For new implementations, the Gallery Agent Skills guide uses metadata.require-secret and a scripts/index.html entry point, which can import JavaScript; secrets pass directly to scripts rather than through model context. Its native-action guidance also requires app-side implementation for additional actions. The example source and skill specification provide the concrete contract for authoring, whether the developer writes the skill directly or uses a coding assistant such as Claude Code.

The team’s preferred authoring workflow uses skills to write skills. Give Gemini CLI, Claude Code, or Antigravity the Gallery documentation, several examples, and a description of the desired feature. The workshop’s example is an offline archiver: photograph a London landmark such as a Churchill statue, retrieve relevant Wikipedia content, store and index it locally, then read it on the flight home. Retrieval requires connectivity, but the saved archive supports later offline use.

Testing can enter the same loop. With a phone connected over ADB, Gemini CLI’s Android ADB skill can interact with the app, check that the new skill does what it claims, and iterate after basic validation. Brick estimates that more than half of the team’s roughly 80 internal skills were initially created this way. The useful workflow joins documentation, implementation, and device testing rather than stopping when code has been generated.

An audience question establishes an important model-size qualification. Brick says the shown examples default to E4B; E2B can run skills too, but may need simpler skills or fewer enabled at once. The restaurant clip itself announces effective 2B, so it should not be treated as a controlled comparison between the two sizes. Asked about a local-model browser integration, Brick says the team has not tried that integration.

Gallery also connects prototyping to application development. It is an open-source project built on LiteRT and LiteRT-LM; developers can use the underlying APIs and compatible model artifacts to build on the same acceleration infrastructure. Community examples range from cat entertainment to blackjack. On mobile, Brick expects the larger models to reach production primarily through shared system GenAI, while desktop and embedded applications can load them directly. For broad in-app deployment, the next part of the workshop turns to tiny models—defined here as fewer than one billion parameters.

LiteRT-LM supplies the complete autoregressive execution loop behind Gallery. At the workshop, Brick describes C++ and Java APIs, a Python API added the preceding week, and Swift APIs as forthcoming. Hardware acceleration sits behind consistent application-facing APIs, although supported artifacts vary: earlier models had MediaTek and Intel support, while corresponding Gemma 4 artifacts had not yet been published. These are the workshop’s platform and API availability statements.

The ordinary conversion path has four stages:

1. Start with a model in Transformers.
2. Use LiteRT Torch’s generative optimizations and quantization workflow.
3. Produce a LiteRT-LM artifact.
4. Prototype with Gallery or integrate LiteRT-LM directly into the target application.

This keeps model preparation separate from the choice of application platform.

For deeper customization, the Torch Generative API provides native PyTorch building blocks for common language-model architectures. It supports constructing and training a custom tiny model, not only fine-tuning an existing checkpoint, while arranging the computation for efficient on-device execution.

Underneath these paths, optimized libraries do much of the work, including XNNPACK for CPU execution and dedicated GPU optimization code. For an NPU, the runtime dispatches work through the relevant API to the device driver. The build changes more than the application API: developers can keep a similar integration while changing the compiled artifact and execution target.

The export path is not limited to Gemma. Brick shows a Qwen example and explains that Gallery can import a LiteRT-LM file and run it on a selected backend such as GPU. He cannot definitively identify that demonstration’s handset; Pixel and Samsung S25 devices are both common in the team’s testing. An optional icon beneath a chat exposes prefill and decode statistics, so developers can load a compatible model artifact and measure their own phone. Desktop prototyping is also available through the litert-lm run command.

Apple’s FastVLM supplies a different tiny-model example: a roughly 500-million-parameter vision-language model repeatedly describing incoming video scenes. The demonstration uses Qualcomm acceleration, with Brick tentatively identifying the handset as an S25. He estimates that four-bit quantization could bring the added app payload to roughly 250–260 MB, but says that quantization was not performed for this demonstration. The model is broadly useful within a bounded domain—image-to-description—whereas a smaller Gemma 3 270M checkpoint generally needs specialization for the application’s particular task. Small transcription and pairwise translation models offer other ready-made narrow capabilities.

FunctionGemma’s training examples cover dataset formatting and further function-calling fine-tuning. Mobile Actions specializes it for ten mobile functions; Tiny Garden uses voice-to-function calling in a game inside Gallery. These are concrete examples of taking a model already oriented toward function calling and training it for a particular application’s action space.

Checkpoint suffixes describe what training has already happened:

• PT: A pretrained checkpoint, before instruction fine-tuning. It gives expert users control over subsequent behavior without first undoing an existing instruction-tuned personality.
• IT: An instruction-fine-tuned checkpoint. FunctionGemma deliberately starts here because function calling is already part of its learned behavior; further specialization is still expected.

Gemma 3 offered both kinds of checkpoint. Brick gives full tuning of a larger model on sovereign or enterprise data as another possible expert workflow. EmbeddingGemma, meanwhile, provides a roughly 300-million-parameter text embedding model for retrieval: a useful tiny transformer, but not an autoregressive LLM.

A complete application often needs models around the LLM as well. Voice activity detection and denoising can run as non-autoregressive models through LiteRT, while LiteRT-LM adds autoregressive execution for language generation. The architecture code also supports custom variants of Llama, Moonshine, Phi, and Qwen. The practical distinction remains task fit: some approximately 500M-class models already provide a standard feature, while smaller models often need application-specific training.

AI Edge Eloquent turns the tiny-model workflow into a production application. Presented as an iOS app, it records dictation and automatically polishes the transcript, removing interjections and cleaning up spoken phrasing. Personal vocabulary handles another common failure: a transcription system may hear the technical term LoRA and write the name Laura. Eloquent accepts a biasing dictionary so the output can favor the user’s intended spelling. The core dictation and cleanup experience runs offline without a per-use cloud service charge.

An audience question asks where this correction happens: inside speech recognition, through a later replacement operation, or in a second model? The vocabulary can be entered manually or populated through an optional Google-account connection that looks for unusual words in sources such as Gmail. Personal names are especially useful entries; the demonstration includes team members Jianing and Cyril.

The answer is a modular pipeline. Microphone input goes to a speech-recognition engine, which produces an unfiltered transcript. A separate personalization path supplies relevant terms. Both enter a dedicated tiny text-polishing LLM. Keeping recognition and polishing separate allows the application to reuse recognition weights for other features and exposes the intermediate transcript for debugging. A single combined model is possible in principle, but separate models make reuse and inspection easier.

Brick describes the recognition and polishing models as unpublished derivatives from the Gemma 3 270M lineage. The general training workflow uses a stronger cloud LLM to generate synthetic examples for the desired behavior, then fine-tunes a tiny base model. He describes datasets ranging from low single-digit millions to tens of millions of examples depending on the task’s ambition, without specifying Eloquent’s exact dataset size. The shipped app demonstrates the same specialization workflow the team recommends to other developers, and which it also applies to other internal products.

A keyboard-integration question leads back to what the polishing model has learned. Its instruction fine-tuning teaches it to use supplied special words, correct similar-sounding terms, remove interjections, and interpret spoken revisions such as “Scratch that.” The behavior is trained into the tiny model; it is not a Gallery skill. Brick suggests that a skill-based version on a four-billion-parameter model could be an interesting experiment, but that is a different implementation.

The following JavaScript illustrates the input boundary for that polishing stage. For the teaching transcript Um, have you trained a Laura for this?, the application supplies LoRA as relevant vocabulary and requests a cleaned result. The intended output is Have you trained a LoRA for this?; constructing the request leaves that output pending.

function buildPolishingRequest(transcript, relevantTerms) {
  return {
    instruction: [
      "Polish the supplied transcript while preserving its meaning.",
      "Correct similar-sounding words using the relevant terms.",
      "Remove interjections and resolve spoken revisions.",
      `Relevant terms: ${relevantTerms.join(", ")}`
    ].join("\n"),
    transcript
  };
}

const request = buildPolishingRequest(
  "Um, have you trained a Laura for this?",
  ["LoRA"]
);

The application can vary vocabulary at inference time because the model has already learned how to use that vocabulary.

For these tiny models, the deployment recipe combines synthetic task data, an off-the-shelf base such as Gemma 3 270M, fine-tuning, and quantization. The goal is a compelling narrow feature that can reach many devices. Brick points to full-fine-tuning Colabs shipped with Gemma 3 270M and FunctionGemma; the official FunctionGemma fine-tuning tutorial provides an entry point for that model’s specialization workflow.

Asked for before-and-after numbers on Mobile Actions, Brick recalls an improvement from forty-something percent to approximately 86% across ten functions. He recalls roughly eight simpler functions exceeding 90%, around 93%, with two harder functions lowering the overall result. These are his approximate internal-evaluation recollections. The public FunctionGemma release reports a different pair, 58% to 85% accuracy; the available evidence does not establish that the two accounts use the same evaluation version.

Across the team’s tiny-model evaluations, Brick reports gains of roughly 20–40 evaluation points from fine-tuning. For a model in the roughly 200M–270M class, that can make specialization essential. The exception is a model already trained for exactly the narrow task required, such as transcription. A developer inventing a new narrow task should expect to supply the corresponding training, rather than assume a small general checkpoint will perform it reliably through prompting alone.

The web-deployment question exposes a separate limit. At the workshop, the Colab workflow reaches a LiteRT artifact, while LiteRT-LM browser support remains work in progress. The later browser API uses supported web-specific model artifacts; it should not be read as evidence that every .litertlm file could run in a browser at the time. Likewise, the published tiny Gemma checkpoints discussed here belong to Gemma 3. Brick says some Gemma 4 fine-tuning recipes are available for the initial small and medium models, but does not announce a future tiny Gemma 4 release.

Availability also has a product dimension. One attendee reports being able to find Eloquent on the web but not download it from a European App Store; Brick says he will pass that feedback along. His prepared conclusion returns to the deployment split: shared system models can serve mobile applications and larger models can serve embedded platforms, but phone RAM and memory cost make tiny models the broad-reach in-app option. Stronger tiny models and easier fine-tuning workflows are the direction he wants to make more accessible.

Safety on the edge has several layers. Brick defers model-level safety details to the Gemma team. An operating-system vendor can then add product-specific input and output checks around a system model, so the raw published checkpoint and the deployed OS service are different safety surfaces. Tiny application models typically expose a narrower task and API. Eloquent transforms supplied content rather than inviting open-ended generation; that narrows the safety problem, but does not remove the need to address it.

For an attendee wanting to run larger Gemma models on an RTX 5090, Brick says the tool flow supports NVIDIA GPUs but cannot provide a specific configuration guide. He suggests investigating the LiteRT route and NVIDIA’s Gemma support through TensorRT-LLM. This is a direction to investigate, not a verified RTX 5090 deployment recipe.

Multi-skill execution introduces another boundary: switching skills across turns is easier to establish than reliably chaining several within one request. Gallery supports multiple enabled skills, and Brick gives an explicit chain: look up a topic in Wikipedia, summarize it in three bullets, and display flashcards. The journal demonstration already switches among mood, meeting, and Wikipedia requests across a conversation. Single-request stacking had worked with explicit prompting, but the team had only had the model for a few weeks and was still exploring the limits; most published examples used a single skill.

None of this depends on hidden Gallery-specific training. The base model was trained for agentic workflows, thinking, and function calling; the Gallery skill architecture came afterward, through experimentation. There was no additional instruction tuning specifically for its skill-selection format. That leaves room for other orchestrators and skill structures to use the same general capabilities.

An attendee describes agent loops that readily grow toward 100K tokens and asks about E2B’s context window. Here the recording’s recalled limits need correction: Google’s original Gemma 4 launch announcement specifies 128K for E2B/E4B and 256K for the larger launch models, rather than the 32K/128K figures exchanged in the room. Model capacity is separate from Gallery’s configured limit, which Brick recalls only approximately as 8K or 12K for performance.

Long context also consumes memory through the KV cache. Brick says the small models were optimized for per-input-token cache requirements because they target edge devices, but provides no bytes-per-token figure or measured long-context memory total. The earlier progressive-disclosure design remains useful even when a model supports a large context: the application still has latency, memory, and reasoning-quality reasons to load only relevant instructions.

App availability and source availability are also separate. Gallery runs on both iOS and Android, but an audience question about iOS source receives a coming-soon answer that names macOS, leaving the exact pending platform release unclear. Brick recalls iOS publication in January and Android availability since the previous summer. The stated development goal is clearer: users should be able to experiment in the app, obtain its source, and reproduce the behavior they have just tried.

The final question returns to the cost of customization. If every task gets a separate fine-tuned multigigabyte model, storage and memory accumulate. For mobile E2B-class deployments, Brick recommends starting with prompting and skills; for tiny models, he recommends task fine-tuning. LoRA offers another route for larger local models. He points to Apple’s foundation-model framework as supporting adapters, while leaving AICore’s plans to its roadmap discussion.

For an embedded robotics platform running a two-billion-parameter model, Brick recommends task-specific LoRA adapters. He describes runtime support for hot-swapping adapters while leaving the base model loaded, and estimates adapter sizes at roughly 8–100 MB depending on rank. The deployment mechanism is to retain one expensive shared base and change the small task-specific weights around it. That provides a third customization choice alongside prompt-based skills and fully specialized tiny models, without duplicating the entire base for every task.