Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

Which parts of an AI application should run on the device in the user’s hand—and which should call out to another service? That question extends beyond phones to laptops, browsers and small computers. Chintan Parikh, product manager for LiteRT within Google AI Edge, introduces the deployment path, with Weiyi Wang joining for the questions. The focus is Gemma 4 E2B and Gemma 4 E4B, the edge variants intended to move local applications from chat toward agents with reasoning and tool use. Smaller options remain relevant: the Gemma 3 family extends down to 270 million parameters for developers seeking very small models to fine-tune.

A camera filter makes the placement problem concrete. Background replacement during a video call needs to respond as the image changes; sending every interaction through a remote service adds a network dependency to that loop. Sensitive-document summarization makes privacy the priority instead. Poor connectivity makes offline operation valuable, while repeated cloud inference makes token costs part of the design. Local and cloud execution can share the workload: the aim is to choose a useful balance, rather than require every operation to happen in one place.

Memory is the first constraint. Parikh estimates roughly 1–2 GB of RAM for Gemma 4 E2B, depending on configuration and use case. He positions it for voice interfaces, summarization and other low-latency local processing. E4B requires more memory and targets heavier workloads on devices such as laptops or suitably equipped IoT systems. These are quantized deployment choices, so the model name alone does not determine the application’s memory requirement.

The agent capabilities connect that local inference to application behavior:

• Function calling: the model can request tools backed by local APIs or external services. The core inference stays on the device even when a tool reaches outside it.
• Structured output: the model supports structured JSON responses. Parikh contrasts this with trying to obtain structure through prompting alone; the Gemma 4 launch explanation also identifies runtime constrained decoding as the enforcement mechanism. Model support and enforced output validity are distinct layers.
• Thinking mode: Gallery exposes a reasoning display that lets users follow the model’s generated thinking before its answer.
• Hardware support: the models are optimized for deployment across different hardware and platforms, so an application is not tied to one phone configuration.

The edge models are available to download through Hugging Face, including the Gemma 4 E2B LiteRT-LM model. E2B and E4B are distributed under Apache 2.0, giving developers a direct starting point for building their own applications.

Google AI Edge Gallery provides a playground for these capabilities. It includes agent skills, AudioScribe, Ask Image and chat experiences, with sample code developers can inspect and an application they can fork. The demonstrations move from basic local interactions toward voice agents, local workflows and privacy-sensitive tasks.

The first skill augments the agent’s knowledge by querying Wikipedia. An encyclopedia question can therefore lead to a tool request instead of relying entirely on information stored in the model’s weights. This is an existing Gallery skill alongside summarization and Ask Image, and it illustrates the earlier execution split: local inference can still use an external knowledge source.

The next demonstration turns spoken personal information into a journal workflow. The input includes a mood score of nine and a comment about getting eight hours of sleep and looking forward to heading out with Amy. The agent can create entries, summarize them and display sleep trends. A follow-up then asks it to analyze mood over the last seven days. Parikh presents the workflow as running on-device, with the newer reasoning and thinking capabilities supporting the interaction.

For an application, the useful boundary is between understanding the utterance and storing a record. The journal input can be represented in Python as a proposed tool payload, ready for application validation and persistence:

import json

proposed_entry = {
    "mood_score": 9,
    "sleep_hours": 8,
    "comment": (
        "I got eight hours of sleep, and I'm looking forward "
        "to heading out with Amy today."
    ),
}

print(json.dumps(proposed_entry, indent=2))

This captures the entry’s values; it does not supply the historical records needed to answer the seven-day trend question. That follow-up requires the journal skill to work with stored entries, not just the latest utterance.

The next skill starts with a breakfast photo and a request to pair its mood with music. Image understanding supplies context for the music-generation step; after a brief wait, music plays. The locally executed skill combines capabilities, including other models, rather than requiring Gemma itself to synthesize the waveform. Gallery also lets users write skills inside the app, using its included instructions.

An animal-call example extends this into what Parikh describes as a more complex workflow involving multiple apps and users. A previously created and loaded skill takes a prompt and generates sound locally; the demonstration produces a lion’s roar. The app lets users select CPU or GPU execution. At this point in the demonstration, Gallery’s NPU support is described as a future addition, distinct from the runtime integrations discussed later.

Skills are also a sharing unit. Users can publish their own contributions through GitHub, download others’ work, or fork the open-source Gallery app to change the experience itself. The Gallery skills directory provides a starting point for creating a skill, complementing the instructions presented with the demonstrations.

The framework underneath these experiences is LiteRT, Google’s on-device runtime built on the TensorFlow Lite foundation. Parikh reports more than 100,000 apps and billions of active users. Daily interpreter invocations count the inference executions happening through that foundation.

The shared .tflite format preserves the deployment path for existing TensorFlow Lite models. Those models remain runnable in LiteRT, and the format spans Android, iOS, macOS, Linux, Windows, web and IoT. The portability diagram connects the model artifact to CPU, GPU and NPU execution across those platforms.

The LiteRT name also reflects a broader input pipeline: TensorFlow is no longer the only source framework. Developers can bring PyTorch or JAX models, convert supported models into the deployment format, and use sample applications to integrate them. Source-framework flexibility and runtime portability meet at the converted artifact.

Running on one development phone is only the beginning. An application may also need to work on phones that are five or six years old, with different memory limits and accelerators. The deployment stack separates model preparation from runtime integration:

1. Convert the model. LiteRT Torch provides the PyTorch conversion path into .tflite.
2. Quantize where appropriate. Choose a representation that fits the intended device and workload.
3. Select the runtime path. Language models use LiteRT-LM, the name documented for the generative layer; other models use LiteRT directly. The general .tflite conversion path is distinct from packaging a language model with its tokenizer into a .litertlm container.

Model Explorer makes the graph inspectable so developers can decide where quantization or mixed precision is appropriate. It informs those changes rather than performing quantization itself. Parikh then introduces AI Edge Portal as a cloud benchmarking service used by internal and third-party developers to evaluate broad Android deployment. The practical question is which model and compilation recipe will run reliably across the intended fleet, including whether to use ahead-of-time or just-in-time compilation.

CPU and GPU libraries provide the broadly available acceleration paths. Parikh describes Qualcomm and MediaTek NPU integrations as completed, with more partners in progress. Speech recognition, text-to-speech and AR/VR camera updates are motivating workloads because they need repeated, timely inference.

Parikh projects roughly 3–10× better performance from NPU execution, without specifying a workload or baseline. He also expects energy benefits, a separate consideration for continuously running applications. The stack offers both ahead-of-time and on-device compilation options, with documentation to help choose the deployment path.

Parikh reports testing the downloadable Gemma models across Android, iOS, Linux and Raspberry Pi. The most tangible portability example is a small robot assembled that morning. A sign instructs it to move its antenna; two blinking lights indicate inference in progress. LiteRT-LM runs on the Raspberry Pi’s CPU, and after a delay the robot wiggles the Sharpies serving as its antennas. Other playful prompts include a marriage proposal. Parikh explicitly describes the prototype as slow and not yet performance-tuned.

For developers who want a simpler way to try deployment, he also points to a CLI and Python bindings available through the project’s website. These provide another entry point alongside the Gallery application.

The performance discussion gives a sense of the gains Parikh wants developers to investigate on their own hardware:

These are Parikh’s reported figures; the presentation does not establish their full model, hardware and baseline conditions. He points to the Hugging Face listings for quantization and platform measurements. The current model card includes later speculative-decoding updates, so its present-day tables should not be substituted for the figures shown in the recording.

A second comparison concerns runtime performance:

The comparator implementation and benchmark configuration are not specified, limiting how these ratios can guide a deployment decision. The practical entry points remain the same: bring a model from a supported framework, download a prepared model, or begin with Gallery to explore the experience before integrating the runtime.

The first audience question applies local inference to a home security camera: could it recognize a son arriving home without continually paying for cloud processing? Parikh points to phone face unlock as an existing example of local recognition, citing Google’s framework on Google devices and describing Apple’s implementation in terms of Core ML.

The questioner suggests examining frames every two seconds. That is a proposed sampling interval, not a prescribed setting. Parikh says a home-camera system would differ from phone face unlock and would need an authenticity-checking algorithm. The discussion then settles on a concrete architecture: connect a Raspberry Pi to the camera, check frames locally, and message the phone when the relevant person is recognized. The notification crosses the network; the continuous recognition loop stays local.

A question about NVIDIA Orin asks for a direct comparison between LiteRT and TensorRT. The response discusses platform-dependent execution backends, including WebGPU, but no LiteRT-versus-TensorRT numbers are presented. Parikh thinks additional Orin results may be on Hugging Face; that does not establish the requested comparison.

The next question moves from one device to multiple nodes: could models running locally trigger a higher-level agent that processes their outputs? Parikh describes groups exploring separate speaking and thinking agents, with orchestration deciding which work stays local and which runs elsewhere. Health and coaching agents are suggested applications, though he has no concrete implementation to share.

The discussion identifies a simpler routing mechanism already used in practice: a classifier or very small model assesses request complexity before passing work onward. This makes the routing decision itself an edge workload. A small local model can determine that another agent is needed without requiring the local device to perform the entire task.

The final question comes from a developer whose mobile app uses Gemini Live APIs for audio-to-audio interaction. The demonstrated audio-to-text capabilities do not directly replace that experience, so the developer asks what to use instead. Parikh’s answer is conditional: identify a suitable open-weight model, ensure it can be put into the supported file format, and check whether its size is feasible for the device.

The emphasis on Gemma reflects the DeepMind track, not an exclusive restriction of the runtime. Other prepared open-weight models are available through Hugging Face, and developers can convert additional supported models themselves. No specific audio-to-audio replacement is recommended. The next engineering step is to choose a candidate with the required audio behavior, then evaluate its conversion and device fit—the runtime supplies a deployment path, but the model must still supply the capability.