# Why Large? Tiny LMs & Agents on Edge/Robotics

Cormac Brick

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/hacEQHHhu2Q-why-large-tiny-lms-agents-on-edge) · [Video](https://www.youtube.com/watch?v=hacEQHHhu2Q)

## Why Large? Tiny LMs & Agents on Edge/Robotics

Local intelligence depends on more than fitting weights in memory: the task, runtime, hardware and training data determine whether a small device can respond usefully.

Basic familiarity with language models, inference and fine-tuning will help; no robotics background is required.

### Intelligence beyond expensive robots

[0:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=17s)

How do you put intelligence into ordinary, inexpensive devices—not just expensive robots? A model that works on a powerful development machine may still be too large to bundle with a mobile app, distribute through a browser or run alongside the rest of a device’s software. The starting question is therefore **how little model a useful feature needs**.

Cormac Brick introduces himself as a tech lead on Google’s AI Edge team, which builds deployment tools including [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM), LiteRT and MediaPipe. The team also supplies technology to Google products and works with the **Gemma** team on compatibility and performance across devices. The software stack connects applications to execution on CPUs, GPUs and NPUs.

![Background slide with team responsibilities beside a stack showing Your App, MediaPipe, LiteRT-LM and LiteRT above CPU, GPU and NPU.](https://ai.engineer/api/talk-essay-media/433a73a5a502547f351d5b0e59437ca59aabd078c9536ff8d8d4e67814941df4)

Google AI Edge software stack for deploying models across devices.

Mobile and browser distribution make model footprint a product constraint. The team’s usual path is to develop a capability for an internal application, then release the tools through open source where feasible. That experience provides two distinct deployment paths: use a small general-purpose model directly, or specialize a much smaller model for a bounded task.

### Local inference trades cloud costs for a memory budget

[2:28](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=148s)

Running inference locally offers several benefits that matter independently:

- **Latency:** responses can be fast and consistent without a network round trip.
- **Privacy:** input data stays on the device.
- **Offline availability:** a feature continues working without reception.
- **Operating cost:** repeated interactions do not each require a paid cloud inference call.

The last benefit becomes substantial at mobile-app or browser scale. Even inexpensive tokens add up when multiplied by a large population and frequent interactions.

The corresponding constraint is hardware cost, especially DRAM. Brick reports phone manufacturers reducing installed memory and cites a roughly 2.5× Raspberry Pi price increase since launch, though the precise board comparison is unclear. The engineering consequence is straightforward: **quantization and task-appropriate model sizing affect the device’s bill of materials**, not just an inference benchmark.

Deployment also has to accommodate a heterogeneous collection of devices. Meanwhile, much of the research effort goes toward larger language models and mixture-of-experts architectures, leaving the lowest end of the model-size range less studied. Finding the smallest model that satisfies a task is therefore a development problem in its own right.

### Small models are often ready to prompt

[4:42](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=282s)

Here, a small model means roughly 1–4 billion parameters, with 1–2 billion a common starting range. Models in this tier can arrive through the operating system—Android AICore on high-end phones or Apple Intelligence—or be bundled by an application vendor.

For IoT and robotics, Brick gives a device-memory rule of thumb of 4–8 GB DRAM for this model tier. That pushes deployment toward laptops, phones and higher-end electronics, while excluding many lower-cost devices and browser environments.

The attraction is a comparatively simple development workflow. Quantize to reduce footprint, then try **zero-shot prompting** for the feature. LoRA adapters offer another route to adaptation, and models in this range can already be reasonably capable at function calling and agent skills. If the hardware budget fits, a custom training pipeline may be unnecessary.

### Compressed weights are only part of the footprint

[6:06](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=366s)

The concrete example is [**Gemma 4 E2B**](https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/). Brick describes its reasoning as comparable to a much larger **Gemma 3** model from roughly a year earlier, with useful answers obtainable through zero-shot prompting. That capability comes with substantial work to reduce the memory occupied by the model.

The E2B example mixes 2-bit, 4-bit and 8-bit quantization, which Brick reports as about 2.9 bits per resident weight. Per-layer embeddings provide another optimization. The slide distinguishes the model’s file size from its multimodal and text-only in-memory footprints; Brick gives 841 MB for the text-only weights. Those are different accounting boundaries, so a download size and a resident-weight size should not be treated as interchangeable.

![Stacked bar chart comparing model file size, multimodal in-memory size and text-only in-memory size, labeled 2488, 1144 and 841 MB.](https://ai.engineer/api/talk-essay-media/9299fa287ddf08a2a72e28471b87518bfa380a696099d4b67e5bd0cab7014eea)

Gemma 4 E2B model file size and memory footprint.

Weights are only the first part of the working memory budget. Brick estimates roughly 2 GB of active RAM after adding the runtime and KV cache, then at least 4 GB of device RAM once the operating system and other workloads are included. A model can fit its weights comfortably and still leave too little memory for a healthy application.

### The same model produces different interaction budgets

[7:27](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=447s)

The next question is how quickly that model runs. Brick reports the following throughput for the roughly 2-billion-parameter example using the team’s runtime and toolchain:

| Hardware | Prefill | Decode |
|---|---:|---:|
| Raspberry Pi, without MTP | — | About 7.6 tokens/s |
| Jetson Orin Nano | — | About 24 tokens/s |
| Qualcomm IoT board, NPU | Almost 4,000 tokens/s | About 31 tokens/s |

On Raspberry Pi, Brick says enabling multi-token prediction, or **MTP**, may roughly double speed depending on the task. He also notes that NVIDIA’s own toolchain may run faster than the team’s Jetson implementation. The Qualcomm board is not identified in the spoken explanation.

**Prefill** processes the input; **decode** produces the response. That distinction matters for visual interaction. Brick gives Gemma 4 image budgets of about 500 tokens for a medium-resolution image and 1,120 for a high-resolution image. From the Qualcomm prefill capacity, he estimates roughly three high-resolution frames per second while retaining useful decoding speed. This is a capacity estimate from image-token processing, rather than a measured end-to-end camera application rate.

These speeds make small models attractive for near-real-time applications when the product can afford the processor and DRAM. The runtime also supports similarly sized community models, giving developers alternatives with different strengths within the same hardware tier.

### Test the interaction, not just whether the model runs

[9:28](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=568s)

[AI Edge Gallery](https://github.com/google-ai-edge/gallery) provides a direct way to try small models on iOS and Android and see their speed on a phone. The application is fully open source, so the same experiment also exposes how an app integrates the open-source runtime. It is both a test surface and an implementation to inspect.

The robotics example is Open Duck Mini v2, an open-source hobby project Brick attributes to DeepMind engineer Xavier. He recommends its video without playing it during the talk. He describes two robots, one using a Jetson Nano and the other a Raspberry Pi, that accept voice and image input, read signs, react and nod their heads.

Brick describes good real-time interaction on the Jetson version, while the Raspberry Pi version works but responds much more slowly. That difference exposes a product-level requirement: successfully completing inference does not mean the robot responds quickly enough for the intended interaction.

If a ready-to-prompt small model meets the product’s hardware and interaction constraints, there is no need to go smaller. But older laptops and lower-cost consumer devices may remain out of reach. Sometimes the AI feature is only a small part of an application and must run while everything else continues working. In that setting, lower memory use and faster responses protect overall system health.

### Tiny models narrow the task to widen device reach

[12:39](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=759s)

Tiny models occupy roughly the 50–500-million-parameter range in this discussion. Brick reports deploying models as small as 50 million parameters. These models are easier to bundle natively with an application and can target devices with less than 2 GB RAM, potentially considerably less. They can also run very quickly, but development becomes more specialized: find an off-the-shelf model that already performs the task, or fine-tune one for the required outcome.

Several fixed-task capabilities are already useful starting points:

- **Speech recognition:** an ASR model converts audio into text.
- **Vision:** a compact visual model supplies basic awareness of a scene.
- **Embeddings:** a text embedding model supports processing and matching text.

For vision, Brick demonstrates [**Apple FastVLM**](https://github.com/apple/ml-fastVLM), a 0.5-billion-parameter model, running on Android with hardware acceleration. The example shows a way to add visual awareness without starting from a larger general-purpose model. ASR and embedding models offer similarly focused building blocks.

### From a spoken request to a device function

[14:24](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=864s)

For custom tasks, the starting points include Gemma 3 at 270 million parameters and [**FunctionGemma**](https://ai.google.dev/gemma/docs/functiongemma/model_card), which receives additional pretraining for function-calling patterns. Brick reports about 45 decode tokens/s on Raspberry Pi for this tiny-model example, attributing the improvement to reading less memory at each decoding step.

The specialized Mobile Actions model takes text in and produces function calls. Brick reports above 86% reliability for a model covering about ten output functions, including calendar scheduling and Wi-Fi toggling. The published FunctionGemma model card separately reports 85% after specialization, versus 58% before, for identifying and formatting Mobile Actions system calls; it does not establish that its evaluation is identical to the talk’s result. Neither figure should be read as end-to-end speech-command reliability.

The useful interface boundary is a structured request that application code can interpret. For a Wi-Fi request, that boundary can be illustrated with this JSON contract:

```json
{
  "input": "Turn off Wi-Fi",
  "proposed_call": {
    "name": "set_wifi_enabled",
    "arguments": {
      "enabled": false
    }
  }
}
```

The function name here defines an application interface; producing the call does not itself change the device setting. In the demo, a separate ASR model sits in front of the function caller, making the sequence audio → text → function call.

Voice-to-function calling is particularly useful on small devices whose settings menus are awkward to navigate. A person can ask for an outcome instead of locating the right control. The selected calendar frame shows a populated **Tennis practice** event form, including date and time fields and a Save button. It is an intermediate state: the form has been populated, but the event is not yet saved.

![Custom fine-tuning slide beside a phone displaying a Tennis practice event form with date and time fields, a Save button and an onscreen keyboard.](https://ai.engineer/api/talk-essay-media/49941344db204bdceed37c1b99b527ff9ba4361ca8d4b9aa08829d7182bd7aa0)

Mobile action demo showing a populated “Tennis practice” calendar form.

### Spend the training effort on one task

[16:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=977s)

The fine-tuning workflow begins with the device constraints, before the training investment:

1. **Choose a base model.** Check that its performance and memory footprint fit the target device.
2. **Build synthetic task data.** Generate examples of the inputs and outputs the feature needs.
3. **Fine-tune for the outcome.** Teach the compact model the bounded behavior rather than expecting broad zero-shot capability.

The open-source [Mobile Actions dataset](https://huggingface.co/datasets/google/mobile-actions) provides a starting point for recreating the function-calling example by fine-tuning FunctionGemma.

Across the team’s work, Brick reports using roughly 10,000–10 million synthetic examples to obtain high task reliability through fine-tuning. This broad range reflects different models and tasks, rather than a prescribed dataset size for every feature.

Summarization and proofreading illustrate the trade. A 2–4-billion-parameter model can perform these tasks reasonably reliably with less specialization work. Brick reports that investing in synthetic data and single-task fine-tuning can produce the same or better quality with a much smaller model. The payoff is a smaller footprint, more responsive interaction and access to a wider set of devices—the properties that make deployment at large scale practical.

### Offline dictation combines two specialized models

[17:47](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1067s)

A production dictation application turns that specialization workflow into a complete feature. Brick describes subscription-free voice dictation that runs entirely on the device. It also removes fillers such as ums and uhs and supports personalization toward words and names relevant to the user.

The architecture separates an **ASR engine** from a **text-polish engine**. Both are fine-tuned Tiny Gemma models, described as being in the low hundreds of millions of parameters rather than given exact sizes. The recognition stage produces text; the polish stage cleans it up. Brick describes very good quality from this fully offline application, available to try on iOS, for a feature that would previously have required a server-backed subscription.

The same deployment logic extends to browser features. Brick points to Chrome summarization and [proofreading APIs](https://developer.chrome.com/docs/ai/proofreader-api), describing them at the time as developer-preview features. His point is that tiny models can make such capabilities available to a wider hardware population; the preview framing is not a claim of current availability across all Chrome installations.

### Make specialization easier, and perception faster

[19:33](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1173s)

For consumer devices and entry-level robotics, the deployment choice turns on what the product can afford. Small LLMs are easy to use and can run especially quickly on NPUs. Tiny models reach more devices, but robust voice-to-function calling requires investment in an appropriate synthetic dataset and fine-tuning. The model’s size and the development effort have to be considered together.

Asked about broader ambitions for tiny models, Brick identifies easier, more general voice-to-function calling as a key goal. One possibility is to have an agent generate the synthetic training data, reducing the work required to specialize a model and making the process accessible to more developers. He presents this as a direction for improvement, not an already completed training system.

Visual input is the other frontier: it still takes time to process. Faster visual models, with broader capabilities such as segmentation, could support additional device behaviors. The next gains therefore depend on both lowering the effort needed to build a specialized model and reducing the time it takes that model to perceive and respond.

## Resources

- [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) — Open-source app for trying local models on Android and iOS, inspecting examples and benchmarking device performance.
- [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) — Google's runtime framework for deploying language models across phones, desktops and edge devices.
- [FunctionGemma model card](https://ai.google.dev/gemma/docs/functiongemma/model_card) — Model documentation covering function-calling evaluations, specialization and measured mobile resource usage.
- [Mobile Actions dataset](https://huggingface.co/datasets/google/mobile-actions) — Google's conversational training and evaluation data for mapping natural-language requests to Android system functions.
- [Fine-tune FunctionGemma for Mobile Actions](https://ai.google.dev/gemma/docs/mobile-actions) — Official tutorial for specializing FunctionGemma 270M and deploying the resulting model into AI Edge Gallery.
- [Gemma 4 on Google AI Edge](https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/) — Launch article explaining memory optimizations, device support and Raspberry Pi and Qualcomm inference measurements.
- [Apple FastVLM](https://github.com/apple/ml-fastVLM) — Official implementation and checkpoints for efficient vision-language models, including the 0.5B variant.
- [Chrome Proofreader API](https://developer.chrome.com/docs/ai/proofreader-api) — Developer-trial documentation for built-in proofreading, including availability checks and hardware requirements.

## Slides

### Why Large? Tiny LMs & Agents on Edge/Robotics

Cormac Brick explores what tiny models can do today and how they could bring intelligence to more devices, beyond expensive robots.

[0:29](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=29s) [Title slide for Cormac Brick’s talk on tiny language models and agents for edge devices and robotics.](https://assets.ai.engineer/sites/aiecode2025/sha256/686e75c7c981a7fe96314395b7a7d514a7d25fca01f0c18ae78bdf108391021a.webp)

Why Large?
Tiny LMs & Agents
on Edge/Robotics

AI Engineer World’s Fair 2026

Cormac Brick
Principal Software Engineer, Google AI Edge

### Introduction

The talk moves from the capabilities and limits of small models to off-the-shelf and custom tiny models, production deployment, and team examples.

[0:36](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=36s) [Agenda covering background, small models, off-the-shelf and custom tiny models, and examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/70fab6eccca192d2b734ac6b2b60fea4a98df22d05878c05fa5bfae47b173e8a.webp)

Introduction

• Background
• Small Models
• Tiny Models — off the shelf
• Tiny Models — custom
• Examples

### Background

Google’s AI Edge team develops tools for deploying models on devices. Its stack places MediaPipe, LiteRT-LM and LiteRT between an app and CPU, GPU or NPU hardware.

[1:08](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=68s) [Google AI Edge team background and a diagram of the app, MediaPipe, LiteRT-LM, LiteRT, and device hardware stack.](https://assets.ai.engineer/sites/aiecode2025/sha256/2aa91234136e260e0e845548ef808176fd19e61a00b3ef57892e50bf677a73c8.webp)

Background

AI Edge Team at Google as tech lead
Developing open source projects (LiteRT-LM, LiteRT, Mediapipe). Making it easy to deploy AI Models across edge devices.
Delivering Edge AI core tech to Google products
Working with Gemma team to ensure their best models run well on lots of devices
Significant focus on small and tiny models

Google AI Edge
Your App
MediaPipe
LiteRT-LM
LiteRT
(FKA TensorFlow Lite)
CPU  GPU  NPU

### Why do edge AI?

Running AI on device offers fast, consistent speed and keeps sensitive data local. It can work without reception and avoids cloud token costs that add up at scale.

[2:26](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=146s) [Four edge AI benefits: latency and user experience, privacy, offline use, and savings.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ef206c94c6af2a4ccd25b091a811333a213bb99ccc10eff6a8466e92dde0ec4.webp)

Why do edge AI?

Latency / UX
Fast, consistent speed

Privacy
Sensitive data stays on device

Offline Use
Reliably available

Savings
No cloud/token costs

### Edge AI Challenges

DRAM cost constrains edge deployment, pushing attention toward quantization and smaller models. Developers also face diverse hardware and a less-studied end of the LLM spectrum.

[3:20](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=200s) [Three edge AI challenges: DRAM cost, diverse target devices, and limited research on smaller models.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0310f2ddad8eb07db97541f29d58137b29917c517c55d0b5abd1e270dbc19e5.webp)

Edge AI Challenges

DRAM Cost
Memory constraints on edge hardware

Target Devices
Wide pool of varying hardware (CPU, GPU, NPU)

Less Studied
Working on a less studied end of the LLM spectrum

### Gemma 4: Smaller Models with Strong Reasoning

Gemma 4 E2B scores 60.0% on MMLU Pro versus Gemma 3 27B’s 67.6%, but slightly exceeds it on GPQA Diamond: 43.4% versus 42.4%.

[6:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=365s) [Bar charts compare Gemma models on MMLU Pro and GPQA Diamond.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf7316740b646a4a2af4ee17742cc970e1ddfc357f12f779563deb628b26cf0c.webp)

Gemma 4 - Smaller models with strong reasoning

MMLU Pro
Advanced general knowledge and reasoning
Gemma 4 31B
Gemma 4 26B A4B
Gemma 4 E4B: 69.4%
Gemma 4 E2B: 60.0%
Gemma 3 27B: 67.6%

GPQA Diamond
PhD-level scientific reasoning
Gemma 4 31B
Gemma 4 26B A4B
Gemma 4 E4B: 58.6%
Gemma 4 E2B: 43.4%
Gemma 3 27B: 42.4%

### Gemma 4: Optimized Memory Footprint

Gemma 4 E2B’s 2,468 MiB file occupies 1,144 MiB in memory for multimodal use and 841 MiB for text-only. With runtime and KV cache, the speaker estimates about 2 GB of active RAM and suggests 4 GB+ device RAM as a rule of thumb.

[6:36](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=396s) [Stacked bars compare Gemma 4 E2B file size with multimodal and text-only memory sizes.](https://assets.ai.engineer/sites/aiecode2025/sha256/720d4458a7a44d5f0786e31759ecbbbe8413fa49d03e4071a24dd8774de17a18.webp)

Gemma 4 - Optimized memory footprint

Gemma4-E2B Model File Size and Memory Footprint

Y-axis ticks: 0, 500, 1000, 1500, 2000, 2500

Model File Size (MiB): 2468
In Memory Size (MiB, Multimodal): 1144
In Memory Size (MiB, text-only): 841

Legend:
embedding / PLE
text embedding
Audio Encoder
Vision Encoder
Other Non Model Metadata
Drafter
Model weights (text only, non-embeddings)

### Gemma 4: E2B Speed

Without MTP, Gemma 4 E2B decodes at 7.6 tk/s on Raspberry Pi 5, 24.2 on Jetson Orin Nano, and 31.7 on Qualcomm’s IQ-8275 EVK. The Qualcomm board prefills at 3,747 tk/s.

[7:36](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=456s) [Device speed table highlights Raspberry Pi 5, Jetson Orin Nano, and Qualcomm IQ-8275 EVK results.](https://assets.ai.engineer/sites/aiecode2025/sha256/70e8d9b393dbd867e6caba85dfac61d6084356a93fdfaf4c93eb03b81f32e248.webp)

Gemma 4 - E2B Speed
(excludes MTP)

Platform + Device | Backend | Prefill (tk/s) | Decode (tk/s)
Android + S26 Ultra | CPU | 557 | 46.9
Android + S26 Ultra | OpenCL | 3,808 | 52.1
Android + S26 Ultra | NPU | 7,463 | 48.1
iOS+ iPhone 17 Pro | CPU | 532 | 25.0
iOS+ iPhone 17 Pro | Metal | 2,878 | 56.5
Linux + NVIDIA (GeForce RTX 4090 & Arm 2.3 & 2.8GHz) | CPU | 260 | 35.0
Linux + NVIDIA (GeForce RTX 4090 & Arm 2.3 & 2.8GHz) | WebGPU | 11,234 | 143.4
macOS + MacBook Pro M4 | CPU | 901 | 41.6
macOS + MacBook Pro M4 | WebGPU | 7,835 | 160.2
Windows + Intel (Raptor Lake + UHD 770) | CPU | 357 | 10.1
Windows + Intel (Raptor Lake + UHD 770) | WebGPU | 472 | 18.6
Raspberry Pi 5 16GB | CPU | 133 | 7.6
Jetson Orin Nano | GPU | 1,142 | 24.2
Qualcomm IQ-8275 EVK | NPU | 3,747 | 31.7

### Community Models

The toolchain supports five community models with listed strengths in local agents, vision and reasoning. Their listed sizes include 0.6B+ and 3B.

[9:15](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=555s) [Table lists five community models with release dates, sizes, and capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e9ef000b69910fe95646762ed1c3d72e72e2b700b6d3766ecb9dc300b21e311.webp)

Community Models

Model | Release | Size | Key Capabilities
MiniCPM5-1B | May 2026 | 1B | Optimized for local agent workflows; InfLLM v2 for long-context.
Ministral-3-3B | Dec 2025 | 3B | Edge-optimized, 0.4B vision encoder, 256k context, agentic skills.
OLMo-2-1B | Dec 2024 | 1B | Fine-tuned for reasoning/math (RLVR). Fast local responses.
Qwen3 | Apr 2025 | 0.6B+ | Hybrid reasoning with "thinking mode". Highly efficient MoE.
VibeThinker-1.5B | Nov 2025 | 1.5B | Top-tier reasoning. High performance.

### Try Small Models

The open-source Google AI Edge Gallery lets you try small models on iOS and Android. Its import dialog configures sampling defaults, feature support, and accelerator compatibility.

[9:31](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=571s) [Resources for small models and Google AI Edge Gallery, with app and repository QR codes and an embedded video.](https://assets.ai.engineer/sites/aiecode2025/sha256/faeebfa8830888d969d8ac89768909250a390450d8278d43a4aaa0ff66f8252d.webp)

Try Small models

Models:
https://huggingface.co/litert-community

App:
‘Google AI Edge Gallery’ on github, app stores

[9:44](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=584s) [A model import dialog appears beside links for trying small models and QR codes for the app.](https://assets.ai.engineer/sites/aiecode2025/sha256/54fe3eaec410bb8e719ee89730bf8f56fd395f5129c3f24f6b5a7faf15df7237.webp)

Try Small models

Models:
https://huggingface.co/litert-community

App:
‘Google AI Edge Gallery’ on github, app stores

Interface text:
Import Model
Default TopK: 5, 64
Default TopP: 0.00, 0.95
Default temperature: 0.00, 1.00
Support image
Support audio
Support tiny garden
Support mobile actions
Support thinking
Compatible accelerators: CPU, GPU, NPU
Cancel
Import

### Try Small Models: Chat Response

Google AI Edge Gallery shows a small model responding on GPU to a question about the meaning of life. The slide links to LiteRT community models; the app is open source.

[10:10](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=610s) [The app displays a submitted question and a model response alongside the small-model resources.](https://assets.ai.engineer/sites/aiecode2025/sha256/bba2d37986d45de321fc1b4d7fba188ad55436b5a75ef563a5aa7148b00003d2.webp)

Try Small models

Models:
https://huggingface.co/litert-community

App:
‘Google AI Edge Gallery’ on github, app stores

Interface text:
4:33
AI Chat
[illegible]
You
what's the meaning of life?

Model on GPU
Okay, the user is asking about the meaning of life. I need to explain it in a way that's both informative and engaging. Let me start by acknowledging the question. The meaning of life is a profound topic, so I should explain it in a way that's understandable.

First, I should mention that the meaning of life is a question that varies

Type prompt...

### Gemma4-E2B Robot Example

The speaker points to a video of two Open Duck Mini v2 robots that read signs and react; the Raspberry Pi version runs much slower than the Jetson Nano version.

[10:16](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=616s) [A robot example slide shows a video thumbnail with a presenter and robot ducks.](https://assets.ai.engineer/sites/aiecode2025/sha256/ddb6396e672960838debe8ed65cc785ed1ac52fa99069986adb2c815f0de2cab.webp)

Gemma4-E2B Robot Example
(raspberryPi, litert)

shout out: Open Duck Mini v2 robots, built by Antoine Pirrone

Gemma Playground: Robot Duck
Google for Developers
Gemma Playground
Robot Duck
Gemma

Interface text:
Watch on YouTube

### But Small Can Also Be Too Big

Small models can still be too large for older devices or responsive interactions, especially when they must share system resources with the rest of an application.

[11:22](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=682s) [Three panels discuss reach, responsiveness, and system health as constraints on model size.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fbe1bfa8c898c0a1bd72e442251d969f9d9a7d17dfd4e2de0d28215d45d0699.webp)

But small can also be too big

Reach
Need to target lower tier devices
e.g. Raspberry Pi, older laptops, watches

Responsiveness
Wide pool of varying hardware
(CPU, GPU, NPU)

System Health
model is be one of many things / not the primary feature

### Tiny Models

Tiny models typically have 50–500 million parameters and can ship natively on consumer devices. The deployment playbook is to use an off-the-shelf model for a fixed task or fine-tune one.

[12:37](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=757s) [Tiny model characteristics and deployment options alongside illustrations of consumer devices.](https://assets.ai.engineer/sites/aiecode2025/sha256/61f0b5fc6227144824ff7324ae73c94d75b103dfe2a7f351f52cad01f7ebcd00.webp)

Tiny Models

• Typically 0.5 - 0.05B parameters
• Easier to ship natively with applications
• In the range of consumer devices ≤ 2G DDR
• Can be very fast, and run on much lower tier hardware
• Playbook: use off the shelf model for fixed tasks, or fine-tune your own

Illustration text: IoT; 5G; 16:04; 23°C; OFF; ON

### Fixed Task tiny models

The slide lists tiny models for speech recognition, vision and embeddings. A FastVLM demo describes a photo on an Android device; the speaker says the 0.5B model uses hardware acceleration.

[13:48](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=828s) [Audio, vision and embedding model examples beside a FastVLM image-description demo.](https://assets.ai.engineer/sites/aiecode2025/sha256/9705d8859b41a5dc13b3268834dfbb54e440aff10323ba988d3d574f5cda241f.webp)

Fixed Task tiny models
(on huggingface)

Audio / ASR
• litert-community/parakeet-tdt-0.6b-v3
• litert-community/parakeet-ctc-0.6b
• litert-community/moonshine-tiny
• litert-community/Qwen3-ASR-0.6B
• litert-community/whisper-tiny

Vision
• litert-community/FastVLM-0.5B

Embeddings
• litert-community/embeddinggemma-300m

Demo prompt: Describe the image in English.
Output should be brief, about 100 words or less.

Demo response: The image depicts a modern, multi-story building with a glass facade, situated in a spacious, open plaza. The building, which appears to be a commercial or office structure, is surrounded by a variety of trees, some of which are bare, suggesting it might be autumn. The plaza is paved with concrete and

Interface text:
FastVLM
FastVLM Demo
LiteRT

### Tiny Models - Fully Custom Fine Tune Example

The Mobile Actions demo feeds a spoken calendar request through speech recognition to a fine-tuned model that outputs function calls. The slide suggests checking a 270m base model’s performance and generating ~10k synthetic samples.

[14:27](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=867s) [Fine-tuning steps and model examples beside a mobile voice-action interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/78aa39563370611d5165a0aee278cf957300433ac56e56ac9ade24f22b63722c.webp)

Tiny Models - Fully Custom Fine Tune Example

Pick a base model
• google/gemma-3-270m
• google/gemma-3-270m-it
• google/functiongemma-270m-it

Check performance: 250 tok/prefill 45 tok/s decode on RPI3

Generate synthetic data (e.g. gemini flash)
include positive / negative samples ~10k Samples

Dataset https://huggingface.co/datasets/google/mobile-actions
Docs https://ai.google.dev/gemma/docs/mobile-actions

Interface text:
Mobile Actions
Control your device with simple commands
[illegible]
add a tennis practice to my calendar for tomorrow
Release to send
Slide up to cancel
Listening...

### Tiny Models - Fully Custom Fine Tune Example

The Mobile Actions demo shows an airport map and a saved calendar event. The fine-tuning recipe uses 270M base models and ~10k synthetic samples; the slide reports 45 tok/s decode on RPI3.

[14:48](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=888s) [Fine-tuning steps beside a phone displaying an airport map result.](https://assets.ai.engineer/sites/aiecode2025/sha256/42fc1c8f686cba9fab3bdd9760f421e49c522e3a6bce1199ecc585eee9b93d7a.webp)

Tiny Models - Fully Custom Fine Tune Example

Pick a base model
• google/gemma-3-270m
• google/gemma-3-270m-it
• google/functiongemma-270m-it

Check performance: 250 tok/prefill 45 tok/s decode on RPI3

Generate synthetic data (e.g. gemini flash)
include positive / negative samples ~10k Samples

Dataset https://huggingface.co/datasets/google/mobile-actions
Docs https://ai.google.dev/gemma/docs/mobile-actions

Interface text:
San Francisco Airport
See similar places
Cargo
Terminal 3
Terminal 2
San Francisco International Airport
4.3 (34,504)
International airport
Directions
Start

[15:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=905s) [Fine-tuning steps alongside a calendar showing a saved event.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e96e1142206ac30ce59074095ede9c6590b40f8fba0feefb43a3feaf0336ca6.webp)

Tiny Models - Fully Custom Fine Tune Example

Pick a base model
• google/gemma-3-270m
• google/gemma-3-270m-it
• google/functiongemma-270m-it

Check performance: 250 tok/prefill 45 tok/s decode on RPI3

Generate synthetic data (e.g. gemini flash)
include positive / negative samples ~10k Samples

Dataset https://huggingface.co/datasets/google/mobile-actions
Docs https://ai.google.dev/gemma/docs/mobile-actions

Interface text:
December
Sat 13
12 PM, 1 PM, 2 PM, 3 PM, 4 PM, 5 PM, 6 PM, 7 PM, 8 PM, 9 PM, 10 PM
Tennis practice
Saved “Tennis practice” on Sat, Dec 13.
Invite via link

### Tiny Models - Fully Custom Fine Tune Example

Pick a base model, check its speed and memory footprint, then fine-tune it with synthetic data. The Mobile Actions dataset supports recreating the FunctionGemma demo; the phone shows a calendar event being edited.

[16:26](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=986s) [Fine-tuning steps alongside a calendar event editor with the keyboard open.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac43e0a39e33e20dd6942a739fdb875c3a96ac331198663fa8413e11630bda81.webp)

Tiny Models - Fully Custom Fine Tune Example

Pick a base model
• google/gemma-3-270m
• google/gemma-3-270m-it
• google/functiongemma-270m-it

Check performance: 250 tok/prefill 45 tok/s decode on RPI3

Generate synthetic data (e.g. gemini flash)
include positive / negative samples ~10k Samples

Dataset https://huggingface.co/datasets/google/mobile-actions
Docs https://ai.google.dev/gemma/docs/mobile-actions

Interface text:
Save
Tennis practice
Event  Task  Birthday
All-day
Sat, Dec 13, 2025  4:00 PM
Sat, Dec 13, 2025  5:00 PM
Pacific Standard Time
Does not repeat
Add people
View schedules
practice  practices  practiced

### Tiny Models in production

AI Edge Eloquent runs voice dictation on device. Its pipeline transcribes speech, then polishes the text using a personal dictionary for relevant names and words.

[17:45](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1065s) [AI Edge Eloquent voice dictation slide showing a speech recognition and text polishing pipeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce28dffafe8c15e6f116a99905d63dcc484f7d268a8ff5dd2a7509a9fda47c34.webp)

Tiny Models in production

AI Edge Eloquent (available on iOS)
premium AI Voice dictation without subscription

Mic Input
Capture user speech
Speech Audio

ASR Engine
Transcribe speech to text
Transcription

Personal Context
Workspace integration (Gmail)

Manual Dictionaries
Add names, jargons, etc.
Download starter dictionary pack

Personal Dictionary
Names, jargons, etc.

[illegible]

Text Polishing Engine
Fix spelling
Filler words removal
False start correction
Grammar correction
etc.

Polished Text
Eloquent UI

Interface text:
18

### Tiny Models in production

AI Edge Eloquent turns speech into text, then polishes it with help from a personal dictionary. The demo shows a shortened result; the speaker says both engines use fine-tuned tiny Gemma models on device.

[18:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1085s) [A voice dictation architecture diagram beside a phone showing a shortened transcript and usage statistics.](https://assets.ai.engineer/sites/aiecode2025/sha256/04029d3da5963e654b9f7468eaecd8b7fa0c8b39812d70dcfb0d00ee5558688f.webp)

Tiny Models in production

AI Edge Eloquent (available on iOS)
premium AI Voice dictation without subscription

Mic Input
Capture user speech
Speech Audio

ASR Engine
Transcribe speech to text
Transcription

Personal Context
Workspace integration (Gmail)
[illegible]

Manual Dictionaries
Add names, jargon, etc.
Download starter dictionary pack

Personal Dictionary
Names, jargon, etc.
[illegible]

Text Polishing Engine
Fix spelling
Filler words removal
False start correction
Grammar correction
etc.
Polished Text

Eloquent UI

Google AI Edge Eloquent
Usage stats
93 words
186 wpm
Stats from your last dictation.

Short
Please review the approved tagline for the Google AI Edge Eloquent app, “Speak naturally, express effectively, no cap,” which cleverly uses “no cap” to mean both “truthfully” and that the app has no usage limits due to its on-device functionality.

Polish
Key points
Formal
Short

Record
History
Dictionaries
Settings

Interface text:
9:41
18

### Takeaways for consumer devices & entry level robotics

Tiny models can reach more devices but mostly need fine-tuning. Tiny VLMs offer general visual intelligence, while robust voice-to-function calling requires synthetic datasets of 10k–10M samples.

[19:32](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1172s) [Takeaways on small language models, tiny vision models, and voice-to-function calling, beside illustrations of connected devices.](https://assets.ai.engineer/sites/aiecode2025/sha256/c26bad13c82f2958b2c42d5216d2633b9acc81815058dcfde86c20c272deabcf.webp)

Takeaways for consumer devices & entry level robotics

• Small LLMs are easy to use, including on NPUs
• Tiny models enable a larger pool of devices but mostly require fine tuning
• Good general visual intelligence with tiny VLMs
• Voice to Function Calling can now be built to be robust, requires investing in synthetic datasets (10k-10M)

Illustration text: IoT, 5G, 16:04, 23°C, OFF, ON

Interface text: 19

### AI Engineer World's Fair

[21:36](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1296s) [AI Engineer World's Fair logo in white on a black background, with the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/1077f4bb00a82b7c7fb77477825ae9c4abb7bc9019fb58789cda50228e4b61d5.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=0s) [upbeat music] Yeah. So yeah, a bit of a change of speed from the last two talks that we're looking at kind of higher-end robots.

[0:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=17s) If we want, uh, for intelligence to get into lots and lots and lots of devices, and not just really expensive robots, we are going to need tiny models. And this talk is about what is the state-of-the-art of tiny models at the moment, what are the things they're good at, um, and what are the things you can go

[0:35](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=35s) start building today. Okay. So yeah, firstly, a bit of background, like, briefly on me and, and the team I work on. Then we're gonna take a look at small models that you may be kind of more familiar with, just kind of explore what they can do, what they can't do, um, yet.

[0:51](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=51s) Um, and then kind of see, "Hey, why do we need even smaller models?" Uh, and then just looking at the state-of-the-art of, uh, tiny models today, and what you need to do to get them into a form where you can deploy them in production to do useful things.

[1:03](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=63s) And yeah, lastly, we've got a couple of examples, uh, that we can look at, uh, from work from our team.

[1:10](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=70s) Okay, um, so me, our work-- I've worked in, um, edge AI for a while. Um, uh, um, these days, I work as a tech lead, um, on the AI Edge team at Google.

[1:21](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=81s) Uh, within the team, the types of things we do, um, are we develop kind of open source projects called, like, LiteRT-LM, LiteRT, MediaPipe, and these make it easy to deploy AI to edge devices.

[1:33](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=93s) Um, we also do a lot of work delivering kind of edge AI core technology to Google's own products, um, some of which would be via tiny models. Um, and then we, we also work with the Gemma team to ensure their models work well on, um, and run well on lots of devices.

[1:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=110s) And then we've a significant focus on small and tiny models, because that's what, um, that's what's useful, uh, for, uh, a lot of kind of mobile phone applications, or if we want to be able to ship a model in-browser, uh, that also has to be really, really small.

[2:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=125s) And generally, our kind of playbook is we develop things for f-first party u- like, for in-house use first, and then if we can figure out a way to share that via an open source package or make those tools available to the wider world, we, we do so.

[2:19](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=139s) And that helps kind of lots of other people, um, uh, build similar types of things using open source technology.

[2:28](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=148s) Okay. So why do edge AI? This is probably, like, um, uh, rela- as opposed to just doing everything in the cloud. Um, you know, it's kind of obvious, but I'll kind of go through it anyway.

[2:37](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=157s) There's, like, kind of latency. You have fast, consistent speed. Privacy, data stays on the device. Offline use, it's kind of reliably available, so that kind of, um, that, that feature that you rely o- rely on on your mobile device will still work even when you don't have reception.

[2:53](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=173s) That can be very helpful. And then savings, um, especially these days if the alternative is to call a, um, even a faster model on the cloud, that will come at a cost, uh, particularly if you're kind of shipping a, an app or, like, a mobile phone app or something in-browser where, you know, the user interaction is at

[3:13](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=193s) kind of a very, very large scale. Then even though those tokens are relatively cheap, you're multiplying it by a large number, and it'll add up quickly.

[3:22](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=202s) So then, um, the main challenges then of deploying AI on the edge is the leftmost one is kind of new, uh, which is DRAM cost. And, um, it's, uh, a really significant constraint that, um...

[3:38](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=218s) And you'll even see some mobile phone manufacturers are putting less DRAM into their devices this year than previously. You'll also see that since, since launch, the cost of a Raspberry Pi 3 sixteen gigabytes has gone up by a factor of, like, two point five X.

[3:53](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=233s) Uh, so DRAM cost is really, really significant. Um, that then has a, like, casts a, a shadow over the rest of this talk, right? Where in order to be able to get AI applications running at the edge, we need to really think a lot about kind of quantization, and we also really need to think about what is

[4:10](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=250s) the smallest possible, um, model we can use for a given task. Um, other challenges are, yeah, there's a, a wider pool of target devices. And yet another challenge is, um, yeah, it's kind of fair to say that a lot of the research, uh, hours that go into LLMs these days are into the much larger models, um, and

[4:31](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=271s) MoE techniques and these types of stuff. Um, and the, the lower end of the LLM spectrum is a lot less studied. Uh, so yeah, these are challenges of deploying on the edge.

[4:42](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=282s) Okay. So small models, and when I say small, I would mean kind of typically maybe kind of one to, like, two or one to four billion parameters. You may find that these are built into the OS.

[4:52](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=292s) There's a version of a small model that ships, um, in Android high-end phones today with AICore. There's a version that ships with Apple, with Apple Intelligence. Um,

[5:02](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=302s) some, some, um, app vendors will ship models this size in their app. We certainly work with some app vendors that do this. Um,

[5:11](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=311s) and for, like, IoT and robotics, you would typically require, like, four, maybe four to eight gigs of DRAM in order to be able to ship this grade of model, which then is an implied cost on the device, right?

[5:24](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=324s) So it then kind of restricts these models to, you know, things like laptops, mobile phones, or kind of higher-end, uh, electronics, and kind of puts it out of reach of maybe a lot of, uh, lower-tier web browsers or the wider kind of IoT and consumer robotics market.

[5:40](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=340s) Um, yeah. And for, for smaller models, yeah, uh, developing smaller models, and we'll look in a while. We do a lot of work to minimize footprint with kind of quantization.

[5:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=350s) Um, and the playbook here is mostly prompting, right? If you want to deliver a particular feature using a smaller model, you can just kind of use zero-shot prompting and get pretty good performance.

[5:58](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=358s) Also, you could use, uh, LoRA adapters. And it's kind of somewhat robust at doing things like kind of function calling and agent skills.

[6:06](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=366s) Okay. So really quick example is like our, you know, um, working with the Gemma team, our favorite go-to example is always Gemma for these kind of things. Uh, so we can see that like the E2B model is, um, pretty capable in terms of reasoning.

[6:19](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=379s) It's certainly like on par, like with a Gemma 3 much larger model from kind of 12 months ago. Um, and yeah, so we now have like much smaller models with...

[6:29](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=389s) That are pretty capable at reasoning, and we get pretty decent answers just with zero-shot prompting for a given task. We've also done lots and lots of work to optimize the memory footprint of that 2 billion parameter model as much as we possibly can.

[6:42](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=402s) Um, so it uses a mix of like 2-bit, 4-bit, and 8-bit quantization, getting it down to like, what is it, like 2.9 bits per weight if you look at the actual weights we need to hold in memory.

[6:52](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=412s) We do other tricks like per-layer embeddings. I won't go into l- like all of the detail here, but end result is we can, you know, you need maybe one, like here it's 841 megabytes for a text-only model in memory just for the weights.

[7:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=425s) And then, you know, maybe by the time you add in the runtime, a, a KV cache, um, uh, footprint, you, you might be up to requiring like 2 gigs of active RAM to be able to run this model.

[7:16](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=436s) Then you account for an OS and the fact that there's other things going on, that's where we get the kind of four, 4 gig plus rule of thumb, uh, for deploying this on a device.

[7:27](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=447s) Um, then in terms of speed, this is using our runtime. This is just a list of devices that we run on. Uh, for the purpose of this talk, we're gonna look more closely at the last three rows of the table, which is if we take that 2 billion parameter model and run it on a Raspberry Pi, um,

[7:44](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=464s) that is... will give about 7.6 tokens per second decode. This is without MTP. If you turn on MTP, that'll get maybe 2X faster depending on the task. Um, if you go to a higher, a more capable device like a Jetson Orin Nano, we can get up to maybe 24, um, tokens per second decode, or maybe even

[8:04](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=484s) faster if you used NVIDIA's own toolchain. This is with our toolchain. Um, we also have done work to port this to a Qualcomm IoT board, which is pretty popular among kind of higher-end, uh, robotics and IoT applications.

[8:16](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=496s) And there, yeah, you can see you can get about like almost 4,000 tokens per second, uh, prefill, uh, 31 tokens per second decode. And that's useful for lots of like almost real-time, um, uh, applications, uh, on an NPU because s- um, with, with these, uh, with Gemma 4 models, like one medium resolution image is like kind of

[8:40](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=520s) 500 tokens. A high resolution image is 1120 tokens. So you could get like, you know, 3 frames per second of high resolu- high resolution tokens, um, uh, going through this model, and have pretty decent decode speed as well.

[8:52](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=532s) So there's lots of, lots of compelling applications you can build with this type of... with a small model. If you're kind of, if you're kind of market or if you're kind of willing to have more expensive hardware and have a more expensive, um, uh, uh, DRAM kind of, uh, line on your kind of bill of materials for

[9:13](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=553s) the device you're building. Uh, yeah, just like our toolchain, we also s- work with other models in the community that are of similar size, and they each have their strengths as well, right?

[9:28](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=568s) Um, so these are some of the other models that we support here. Um, really briefly, I won't go into this in too much detail, but we also... If I can get this to play.

[9:39](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=579s) Um, we also have a- an app that you can use on both iOS and Android. So if you wanna take one of these small models to see how fast it works on a phone, you can, uh, just go straight ahead and do that.

[9:51](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=591s) Um, so it's available on AI Edge Gallery. Also, all of the...

[9:57](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=597s) Why am I getting that buzzing? Uh, all-- the app is also fully open source, so if you wanna see how to build something similar using one of these models or to see how this is using the open source runtime that runs the models, you can see all of that.

[10:10](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=610s) So this is a great way of just getting started and trying small models if this is what you want to do.

[10:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=617s) Okay. This is another example, uh, which I'm not gonna play this video, but you should definitely check it out. This is an example showing a, um, an open source, the Open Duck Mini v2 robot.

[10:28](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=628s) This is one, um, Xavier, one of the, um, engineers in DeepMind built this as a kind of hobby project. Really, really fun. Um, so go check out this YouTube video.

[10:37](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=637s) What you'll, what you'll see is these two robots. One is, uh, one uses the Jetson Nano, one uses the Raspberry Pi. And, um, you'll see that the robot is able to, um, it's, it's able to kind of like read signs like, um, and react to things and kind of nod its head.

[10:55](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=655s) It's also kind of able to take both voice and image input. Um, yeah, and what you'll see is the Jetson Nano one performs, uh, uh, has really good real-time interaction.

[11:04](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=664s) The one based on Raspberry Pi, it works, but it's kind of a lot slower, right? So for some examples, um, for some types of interaction, um, even the kind of best models we have today are, are maybe not meeting a kind of user interaction requirements.

[11:21](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=681s) Uh, but yeah, this is a really fun video, so definitely check it out. So yes, um, so then small models, while, uh, they're great, right? If your product can afford, uh, to use one of these, they're, uh, they're really easy to use because you just need to zero-shot prompt in order to get it to work.

[11:38](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=698s) Um, the Gemma team has done great work in having, you know, low footprint, high capable, um, uh, models that are ready to use, and they're optimized running on all of those devices, uh, you saw earlier.

[11:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=710s) And you know, if, um, yeah, so if, if all of your constraint-- if you can live within those constraints, uh, then great, right? Your journey would stop here, and you would build a feature you would want, right?

[12:02](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=722s) For lots and lots of other things that, that we do in our work and other people that we talk to, you know, we're still at a point where small models are too big because they can't reach like older laptops or kind of more consumer edge devices.

[12:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=737s) Um, the user interaction needs to be more responsive. Um, we also kind of have the reality, and we do have this a lot of times, where the, the model you want to run isn't the main feature in the application.

[12:29](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=749s) It's like one tiny thing in a corner that needs to run while everything else in the system is running. Um, so we also need a, a smaller model for system health as a common, uh, common pattern.

[12:39](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=759s) So then enter kind of tiny models, right? So these are typically in the as small as kind of 50 million parameters. We've deployed models that small, uh, to maybe 500 million parameters.

[12:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=770s) Uh, they're easier to ship nati- natively with applications. They would run on the types of things you see on the right-hand side, um, uh, and would require, you know, maybe less than two gigs of RAM or, or even less than that.

[13:03](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=783s) And they can also be made to run really, really fast. But the playbook to deploying here, it's a little more complicated. Um, so, you know, sometimes there's off-the-shelf models that'll do what you want, and we'll look at those in the next slide.

[13:15](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=795s) Um, or else if that doesn't work, you're gonna be left in a world of kind of fine-tuning a model to achieve a given outcome, but which works very, very well.

[13:24](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=804s) So fixed task models, um, there's a bunch of things around ASR, vision, and embedding models. And if you have a... If you have something, uh... Yeah, so like ASR and vision and embeddings, these are all kind of stock features, and they work really, really well.

[13:38](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=818s) This is an example of Apple FastVLM, uh, which is 0.5 billion parameter model running on a, uh, Android device using hardware acceleration, and you can see it runs really, really fast.

[13:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=830s) So if you needed to kind of add a little bit of visual intelligence to a, um, like an edge device or an IoT device, you know, this class of model is an excellent, uh, is an excellent option to get that kind of first level of visual awareness.

[14:04](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=844s) Or for ASR, um, yeah, there's some kind of strong models listed here as well.

[14:09](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=849s) And, and then lastly, embedding models are great at, um, uh... Yeah, like this is just a text embedding model, which is, yeah, really good at kind of, uh, processing and ma- matching text, which is can be relevant in some cases.

[14:24](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=864s) Okay. Um, but then next scenario is you want to kind of fine-tune a model. Um, so here you can start with, uh, the models I'm citing here are kind of Google developed models.

[14:36](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=876s) Uh, so there's some starting at like 270 million parameters, and Gemma 3 and FunctionGemma. Uh, Gemma 3 is a general purpose model. FunctionGemma is one that has extra pre-training, uh, for function calling patterns.

[14:51](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=891s) So here the performance, if you remember earlier on the Raspberry Pi, our performance was at, uh, mid-single digits tokens per second decode. So here that kind of jumps up to 45 tokens per second because we need to read less, uh, from memory each time.

[15:05](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=905s) And we can fine-tune this to do pretty compelling things. So on the right-hand side, um, this is running a, uh, what we call a mobile actions model. Uh, so this is text in and function calling out.

[15:19](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=919s) Um, this model knows about 10 different output functions and can call them at over 86%, uh, reliability from a given arbitrary text input. And this is for doing common things on a mobile device like, uh, schedule a, uh, calendar or turn on and off Wi-Fi or things like this.

[15:39](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=939s) And it can take like arbitrary free text input and, uh, convert that to appropriate function calling. And for this demo, we've taken another ASR model and put it in front of that, um, which gives kind of voice to function calling as a feature.

[15:53](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=953s) And voice to function calling is pretty key for lots of IoT and edge devices because,

[15:58](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=958s) um, yeah, like smaller devices tend to have, you know, require settings menus, and that user interface can be really, really challenging for lots of people. Uh, so yeah, being able to just talk to something to ask for a given outcome, this is a pretty key capability, and, uh, we can do that reasonably reliably using a fine-tuned small

[16:17](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=977s) model. So the playbook is generally then you pick a base model, you check the performance if the performance and memory footprint are within the range that you want. And then, um, the kind of the harder part is you...

[16:30](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=990s) uh, the playbook we've found works really, really well is we synthetically generate data to fine-tune that model. Um, depending on the model, like, um, there's a dataset we've open sourced here called Mobile Actions, it's available on Hugging Face, that corresponds to this if you want to kind of recreate that same demo yourself and fine-tune FunctionGemma from scratch.

[16:50](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1010s) Um, but we- we've generally found that in the range of 10,000 to 10 million samples of synthetically generated, um, data will be sufficient to fine-tune a smaller model to a really, really high degree of reliability.

[17:07](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1027s) And so for other tasks we've done like things like summarization or proofreading, so something which you could do with a 2 or 4 billion parameter model reasonably reliably. If you're willing to put the time and energy into creating a synthetic dataset and fine-tuning a model, you can achieve a similar, like the same or greater quality with a

[17:27](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1047s) model that is much, much smaller, will work on a much wider set of devices, and will be much, much more responsive. Um, so yeah, and that's the, that's the type of outcome we're seeing now with just fine-tuning a model for a single task.

[17:40](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1060s) And it's a really like... Yeah, we found this is a really good playbook, uh, for deploying at like very wide scale.

[17:47](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1067s) So here's another example in... This is one example in production where we have... This is an app that we've developed for voice dictation without subscription. Um, all of the voice dictation happens locally on device.

[18:02](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1082s) Um, and as well as just doing dictation, it also does, uh, uh, it also does, um... Well, it kind of cleans up ums and uhs, right? If you see on the right-hand side, it's able to clean up text.

[18:13](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1093s) It's also able to do biasing towards kind of words and names that, um, uh, are kind of relevant to you personally, so kind of personalization. The left-hand side kind of shows how we built that application.

[18:25](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1105s) So there's an ASR engine and a text polish engine, and both of these are fine-tuned versions of Tiny Gemma models. And this allows us to take something that would've been a kind of like server-only feature of, you know, where you require a subscription to do highly accurate, uh, voice dictation, and have an app that's just able to

[18:42](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1122s) do that completely offline with very, very good quality. Uh, so this is something you can try on iOS if, if you wanna give this a go today. But, um, uh, yeah, and it just, uh, the backbone of this app is kind of two fine-tuned, uh, small Gemma-based models in the low single digits, hundreds of parameter- million parameters.

[19:01](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1141s) Yeah. We... Also worth, uh, noting is there's also, uh, kind of features in developer preview in Chrome, for example, that kind of summarization and proofread APIs are a feature as built-in APIs in Chrome.

[19:14](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1154s) And delivering those features via tiny models allows, um, the Chrome team to ship them to a much wider set of, uh, Chrome users than would otherwise be possible. Um, yep.

[19:24](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1164s) So that's, uh... We've probably gonna have like one minute for questions. Um, some kind of key takeaways is on the last slide, if I can get there.

[19:33](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1173s) Yeah, so takeaways from consumer devices and entry-level robotics is small LLMs are easy to use, uh, and, and especially on NPUs, uh, they're very, very fast. Uh, tiny models will alla- will enable reach a much, much larger pool of devices.

[19:46](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1186s) And voice-to-function calling, um, can now be built to, to be robust using tiny models. Uh, it just requires kind of investing in an appropriate synthetic data set with enough samples, and then you can fine-tune a model to get really good outcomes.

[19:58](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1198s) Cool. So happy to take one or two questions or, um, if anybody has one. Yeah?

[20:09](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1209s) Sorry, I'm going to plug this out. Yeah, sorry. Sorry. Say again?

[20:27](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1227s) Like broader ambitions of where tiny models can go? Wow. Um,

[20:32](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1232s) I think kind of generalizing voice-to-function calling, um, is a, is a, is one key goal, like making that very easy for lots of people because I think that's a key use case.

[20:43](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1243s) That if we can figure, like, if we can figure out how to make, you know, have, like, an agent generate the synthetic data [laughs] for you, right? Um, if we...

[20:51](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1251s) Like, it's certainly possible to make that journey much easier than it is today and make it available to a lot more people. Um,

[20:59](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1259s) yeah. And, and certainly the, the visual input as well, um, that takes a little bit of time at the moment. There's certainly scope to have faster models there, uh, that can do a wider set of things like kind of segmentation and other things that would enable other use cases.

[21:12](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1272s) Yeah.

[21:12](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1272s) Awesome. Yeah, due to the time, we probably don't have a, a Q&A session for today.

[21:16](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1276s) Um-

[21:16](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1276s) Yeah. But Cormac will stay after the session maybe, and you can ask for more question about the-

[21:22](https://www.youtube.com/watch?v=hacEQHHhu2Q&t=1282s) I'll stay after the session, or you can come grab me downstairs at the DeepMind booth at 4:00. I'll, I'll be there 4:00 to 5:00. Okay. [outro music]
