Why Large? Tiny LMs & Agents on Edge/Robotics

How do you put intelligence into ordinary, inexpensive devices—not just expensive robots? A model that works on a powerful development machine may still be too large to bundle with a mobile app, distribute through a browser or run alongside the rest of a device’s software. The starting question is therefore how little model a useful feature needs.

Cormac Brick introduces himself as a tech lead on Google’s AI Edge team, which builds deployment tools including LiteRT-LM, LiteRT and MediaPipe. The team also supplies technology to Google products and works with the Gemma team on compatibility and performance across devices. The software stack connects applications to execution on CPUs, GPUs and NPUs.

Mobile and browser distribution make model footprint a product constraint. The team’s usual path is to develop a capability for an internal application, then release the tools through open source where feasible. That experience provides two distinct deployment paths: use a small general-purpose model directly, or specialize a much smaller model for a bounded task.

Running inference locally offers several benefits that matter independently:

• Latency: responses can be fast and consistent without a network round trip.
• Privacy: input data stays on the device.
• Offline availability: a feature continues working without reception.
• Operating cost: repeated interactions do not each require a paid cloud inference call.

The last benefit becomes substantial at mobile-app or browser scale. Even inexpensive tokens add up when multiplied by a large population and frequent interactions.

The corresponding constraint is hardware cost, especially DRAM. Brick reports phone manufacturers reducing installed memory and cites a roughly 2.5× Raspberry Pi price increase since launch, though the precise board comparison is unclear. The engineering consequence is straightforward: quantization and task-appropriate model sizing affect the device’s bill of materials, not just an inference benchmark.

Deployment also has to accommodate a heterogeneous collection of devices. Meanwhile, much of the research effort goes toward larger language models and mixture-of-experts architectures, leaving the lowest end of the model-size range less studied. Finding the smallest model that satisfies a task is therefore a development problem in its own right.

Here, a small model means roughly 1–4 billion parameters, with 1–2 billion a common starting range. Models in this tier can arrive through the operating system—Android AICore on high-end phones or Apple Intelligence—or be bundled by an application vendor.

For IoT and robotics, Brick gives a device-memory rule of thumb of 4–8 GB DRAM for this model tier. That pushes deployment toward laptops, phones and higher-end electronics, while excluding many lower-cost devices and browser environments.

The attraction is a comparatively simple development workflow. Quantize to reduce footprint, then try zero-shot prompting for the feature. LoRA adapters offer another route to adaptation, and models in this range can already be reasonably capable at function calling and agent skills. If the hardware budget fits, a custom training pipeline may be unnecessary.

The concrete example is Gemma 4 E2B. Brick describes its reasoning as comparable to a much larger Gemma 3 model from roughly a year earlier, with useful answers obtainable through zero-shot prompting. That capability comes with substantial work to reduce the memory occupied by the model.

The E2B example mixes 2-bit, 4-bit and 8-bit quantization, which Brick reports as about 2.9 bits per resident weight. Per-layer embeddings provide another optimization. The slide distinguishes the model’s file size from its multimodal and text-only in-memory footprints; Brick gives 841 MB for the text-only weights. Those are different accounting boundaries, so a download size and a resident-weight size should not be treated as interchangeable.

Weights are only the first part of the working memory budget. Brick estimates roughly 2 GB of active RAM after adding the runtime and KV cache, then at least 4 GB of device RAM once the operating system and other workloads are included. A model can fit its weights comfortably and still leave too little memory for a healthy application.

The next question is how quickly that model runs. Brick reports the following throughput for the roughly 2-billion-parameter example using the team’s runtime and toolchain:

On Raspberry Pi, Brick says enabling multi-token prediction, or MTP, may roughly double speed depending on the task. He also notes that NVIDIA’s own toolchain may run faster than the team’s Jetson implementation. The Qualcomm board is not identified in the spoken explanation.

Prefill processes the input; decode produces the response. That distinction matters for visual interaction. Brick gives Gemma 4 image budgets of about 500 tokens for a medium-resolution image and 1,120 for a high-resolution image. From the Qualcomm prefill capacity, he estimates roughly three high-resolution frames per second while retaining useful decoding speed. This is a capacity estimate from image-token processing, rather than a measured end-to-end camera application rate.

These speeds make small models attractive for near-real-time applications when the product can afford the processor and DRAM. The runtime also supports similarly sized community models, giving developers alternatives with different strengths within the same hardware tier.

AI Edge Gallery provides a direct way to try small models on iOS and Android and see their speed on a phone. The application is fully open source, so the same experiment also exposes how an app integrates the open-source runtime. It is both a test surface and an implementation to inspect.

The robotics example is Open Duck Mini v2, an open-source hobby project Brick attributes to DeepMind engineer Xavier. He recommends its video without playing it during the talk. He describes two robots, one using a Jetson Nano and the other a Raspberry Pi, that accept voice and image input, read signs, react and nod their heads.

Brick describes good real-time interaction on the Jetson version, while the Raspberry Pi version works but responds much more slowly. That difference exposes a product-level requirement: successfully completing inference does not mean the robot responds quickly enough for the intended interaction.

If a ready-to-prompt small model meets the product’s hardware and interaction constraints, there is no need to go smaller. But older laptops and lower-cost consumer devices may remain out of reach. Sometimes the AI feature is only a small part of an application and must run while everything else continues working. In that setting, lower memory use and faster responses protect overall system health.

Tiny models occupy roughly the 50–500-million-parameter range in this discussion. Brick reports deploying models as small as 50 million parameters. These models are easier to bundle natively with an application and can target devices with less than 2 GB RAM, potentially considerably less. They can also run very quickly, but development becomes more specialized: find an off-the-shelf model that already performs the task, or fine-tune one for the required outcome.

Several fixed-task capabilities are already useful starting points:

• Speech recognition: an ASR model converts audio into text.
• Vision: a compact visual model supplies basic awareness of a scene.
• Embeddings: a text embedding model supports processing and matching text.

For vision, Brick demonstrates Apple FastVLM, a 0.5-billion-parameter model, running on Android with hardware acceleration. The example shows a way to add visual awareness without starting from a larger general-purpose model. ASR and embedding models offer similarly focused building blocks.

For custom tasks, the starting points include Gemma 3 at 270 million parameters and FunctionGemma, which receives additional pretraining for function-calling patterns. Brick reports about 45 decode tokens/s on Raspberry Pi for this tiny-model example, attributing the improvement to reading less memory at each decoding step.

The specialized Mobile Actions model takes text in and produces function calls. Brick reports above 86% reliability for a model covering about ten output functions, including calendar scheduling and Wi-Fi toggling. The published FunctionGemma model card separately reports 85% after specialization, versus 58% before, for identifying and formatting Mobile Actions system calls; it does not establish that its evaluation is identical to the talk’s result. Neither figure should be read as end-to-end speech-command reliability.

The useful interface boundary is a structured request that application code can interpret. For a Wi-Fi request, that boundary can be illustrated with this JSON contract:

{
  "input": "Turn off Wi-Fi",
  "proposed_call": {
    "name": "set_wifi_enabled",
    "arguments": {
      "enabled": false
    }
  }
}

The function name here defines an application interface; producing the call does not itself change the device setting. In the demo, a separate ASR model sits in front of the function caller, making the sequence audio → text → function call.

Voice-to-function calling is particularly useful on small devices whose settings menus are awkward to navigate. A person can ask for an outcome instead of locating the right control. The selected calendar frame shows a populated Tennis practice event form, including date and time fields and a Save button. It is an intermediate state: the form has been populated, but the event is not yet saved.

The fine-tuning workflow begins with the device constraints, before the training investment:

1. Choose a base model. Check that its performance and memory footprint fit the target device.
2. Build synthetic task data. Generate examples of the inputs and outputs the feature needs.
3. Fine-tune for the outcome. Teach the compact model the bounded behavior rather than expecting broad zero-shot capability.

The open-source Mobile Actions dataset provides a starting point for recreating the function-calling example by fine-tuning FunctionGemma.

Across the team’s work, Brick reports using roughly 10,000–10 million synthetic examples to obtain high task reliability through fine-tuning. This broad range reflects different models and tasks, rather than a prescribed dataset size for every feature.

Summarization and proofreading illustrate the trade. A 2–4-billion-parameter model can perform these tasks reasonably reliably with less specialization work. Brick reports that investing in synthetic data and single-task fine-tuning can produce the same or better quality with a much smaller model. The payoff is a smaller footprint, more responsive interaction and access to a wider set of devices—the properties that make deployment at large scale practical.

A production dictation application turns that specialization workflow into a complete feature. Brick describes subscription-free voice dictation that runs entirely on the device. It also removes fillers such as ums and uhs and supports personalization toward words and names relevant to the user.

The architecture separates an ASR engine from a text-polish engine. Both are fine-tuned Tiny Gemma models, described as being in the low hundreds of millions of parameters rather than given exact sizes. The recognition stage produces text; the polish stage cleans it up. Brick describes very good quality from this fully offline application, available to try on iOS, for a feature that would previously have required a server-backed subscription.

The same deployment logic extends to browser features. Brick points to Chrome summarization and proofreading APIs, describing them at the time as developer-preview features. His point is that tiny models can make such capabilities available to a wider hardware population; the preview framing is not a claim of current availability across all Chrome installations.

For consumer devices and entry-level robotics, the deployment choice turns on what the product can afford. Small LLMs are easy to use and can run especially quickly on NPUs. Tiny models reach more devices, but robust voice-to-function calling requires investment in an appropriate synthetic dataset and fine-tuning. The model’s size and the development effort have to be considered together.

Asked about broader ambitions for tiny models, Brick identifies easier, more general voice-to-function calling as a key goal. One possibility is to have an agent generate the synthetic training data, reducing the work required to specialize a model and making the process accessible to more developers. He presents this as a direction for improvement, not an already completed training system.

Visual input is the other frontier: it still takes time to process. Faster visual models, with broader capabilities such as segmentation, could support additional device behaviors. The next gains therefore depend on both lowering the effort needed to build a specialized model and reducing the time it takes that model to perceive and respond.