Gemma 4 Deep Dive — Cassidy Hardin, Google DeepMind

How do you give a model useful reasoning and multimodal capabilities while keeping it small enough to run on a phone or laptop? Gemma 4 approaches that problem with a family of models rather than one architecture scaled uniformly. Cassidy Hardin, who introduces herself as a Google DeepMind researcher, presents four models released the previous week: two effective models for local devices, a mixture-of-experts model, and a larger dense model.

The small E2B and E4B models target phones, iPads, and laptops. 26B A4 introduces mixture-of-experts inference to Gemma, while 31B uses a dense architecture. Hardin reports that both larger models ranked among LM Arena’s top six open-source models at launch. That is a historical leaderboard claim; the supplied evidence does not establish its evaluation conditions.

The release also moves Gemma to Apache 2.0. The practical intent is to make the same model family usable throughout development, from initial experiments through deployment, without changing the licensing framework along the way.

The 31B model targets advanced multimodal reasoning. Hardin reports a number-three global Arena ranking and says it outperformed models more than twenty times its size; these are launch-era comparisons, not established superiority across every task. Its more concrete workflow features are a 256K context window, native thinking, function calling, and structured JSON output. Together, those features support autonomous workflows that need to retain context, choose actions, and produce machine-readable results.

The 26B model separates total model capacity from the parameters used in a forward pass. Hardin gives 3.8 billion active parameters here, after saying 3.9 billion in the opening overview; the model card specifies 3.8 billion. Its router selects eight of 128 experts for each inference pass, allowing sparse computation within the larger parameter set.

The effective models instead prioritize on-device multimodality: they accept text, vision, and audio, but produce text only. Hardin describes improvements over Gemma 3 across agentic capabilities, coding, multimodal tasks, and multilingual tasks, without giving individual benchmark scores in this passage. The rest of the walkthrough explains the architecture behind that positioning.

Start with the dense decoder used by 31B and the effective models. Gemma 4 interleaves local and global attention layers, generally at a 5:1 local-to-global ratio; E2B uses 4:1. Local attention restricts each token to a sliding window of preceding tokens. Global attention can reach all preceding tokens, and the final layer is always global.

The local window is 512 tokens in the smaller models and 1,024 tokens in the larger models. Restricting the window reduces the work of local attention while allowing the resulting representations to pass into subsequent layers. Periodic global layers retain access to the full preceding context, so the window is not the model’s total context limit.

Global layers still have to attend across that full context, making their keys and values expensive to retain. Grouped-query attention reduces this burden by letting multiple query heads share key and value heads. Hardin illustrates different grouping strengths for local and global attention:

The wider global heads compensate for some of the capacity lost by reducing the number of independent key/value heads. These grouping ratios should not be treated as a universal configuration for every model: the published E2B configuration specifies eight query heads and one key/value head, while retaining the 256/512 local/global dimensions.

The two attention changes address different costs. Sliding windows reduce how much history local layers inspect; shared key/value heads reduce the state required by attention heads, including those in global layers. Hardin credits the global 8:1 grouping with improved efficiency, but gives no measured memory or latency savings here.

The mixture-of-experts architecture changes the decoder’s feed-forward component. Instead of sending each representation through one dense feed-forward network, it uses a collection of smaller feed-forward networks called experts. A shared expert runs on every pass.

Alongside that shared computation, the router selects eight experts from a pool of 128. The shared expert is three times the size of a regular routed expert in Hardin’s description. The detailed structure is therefore 128 routed experts plus one shared expert, consistent with the model card—not 128 including the shared expert. Only the selected routed experts and the shared expert participate in a given pass, which is the mechanism behind the distinction between total and active parameters.

The effective models make a different distinction between the parameters required to operate the model and its total representational parameters. E2B has 2.3 billion effective parameters and 5.1 billion parameters including embeddings. These are parameter counts, not literal network depth or a measured device-memory footprint. The goal is useful local inference on phones and laptops without requiring calls to a remotely served model.

Per-layer embeddings, or PLE, add a dedicated embedding table at each layer. They do not replace the standard token embedding table: a token ID still maps to an ordinary model-width vector. That vector has 1,536 dimensions in E2B and 2,560 dimensions in E4B.

Each additional PLE table also covers the vocabulary, but holds a smaller representation for each token at that particular layer. Hardin describes storing these tables in flash memory rather than VRAM, addressing a central constraint of local devices. This is the storage strategy described in the talk; whether a deployment keeps those tables outside accelerator memory depends on its implementation. PLE vectors have only 256 dimensions, substantially narrower than the standard token embeddings.

Follow the token hi through the model. Its token identity stays the same, but each layer supplies a different learned PLE vector. Hardin describes 35 layers for E2B and 42 for E4B. At the end of a decoder block, the model looks up the token’s 256-dimensional vector for that layer, then projects it to the model’s full embedding width: 1,536 for E2B or 2,560 for E4B. The model can therefore draw on additional token-specific representations throughout the stack without requiring all of those tables to reside in scarce VRAM under the described storage arrangement.

Hardin attributes stronger performance than prior small Gemma generations to this PLE design. The accompanying diagram completes the mechanism: a selected table row becomes a 256-dimensional vector, passes through a projection, and enters the per-layer embedding component of the decoder block.

The multimodal progression begins with vision in Gemma 3, followed by audio, vision, and text in Gemma 3n. Gemma 4 integrates multimodality from the beginning. Its larger models use a 550-million-parameter vision encoder; E2B and E4B use a smaller 150-million-parameter encoder.

Variable aspect ratios and variable resolutions give developers control over the image resolution and soft-token budget allocated to an input, with five supported options. To see why those choices matter, start with the encoder’s input: an image is divided into 16×16-pixel patches. Each patch is flattened and linearly projected into a patch embedding, with positional information incorporated so the model can interpret the image’s arrangement.

Hardin compares 4×2 and 3×3 patch arrangements. The patch numbered four occupies a different spatial location when the grid shape changes. A sequence index alone cannot tell the model where that patch sits relative to its neighbors; the representation also needs spatial positional information. That is what lets different image shapes remain geometrically meaningful after conversion into a sequence.

Resolution introduces a separate choice. Two images can both have a 3:2 aspect ratio while containing very different amounts of detail. A larger image budget lets the model process more of that detail, but consumes more of the available token budget.

• OCR and spatial object recognition: Allocate a larger budget to preserve the visual detail the task depends on.
• Text-focused applications: Use a smaller image budget when visual detail contributes little to the task.

The budget is therefore a workload decision, not simply an image-format setting.

This changes the handling of images compared with Gemma 3’s pan-and-scan approach. Hardin describes splitting an image into square regions, padding as needed, and passing a single source image as two, three, or four images processed sequentially. Gemma 4 instead accommodates a varying number of patches for the supplied image, supporting its aspect ratio and resolution without relying on that same sequence of square crops.

The language model does not receive one input embedding for every original patch. Gemma 4 pools each 3×3 group of patches into one embedding. A budget of 280 image soft tokens corresponds to a patch budget of 2,520. This is the budget correspondence, not a guarantee that every image produces exactly that many patches. Padding handles image dimensions that do not fit cleanly into 16×16-pixel patches.

The square-image comparison in the talk covers five supported resolution and budget settings. Hardin highlights budgets of 560 and 1,120 soft tokens for object detection and OCR. The same pooling arithmetic lets you compare their patch capacity:

patches_per_soft_token = 3 * 3

for soft_token_budget in (280, 560, 1120):
    patch_budget = soft_token_budget * patches_per_soft_token
    print(f"{soft_token_budget} soft tokens: {patch_budget} patches")

Increasing this budget gives detailed visual tasks more room before pooling compresses the representation for the language model.

The complete path is N image patches → N patch embeddings → N/9 pooled soft tokens → projection to language-model inputs. In the final diagram, the patches pass through GemmaVis ViT; the pooled tokens then pass through Linear Projection and RMSNorm before entering Gemma 4. Patch encoding preserves visual information at the vision stage, while pooling reduces the sequence length that the language model must consume.

Audio support in E2B and E4B targets translation and speech recognition. An audio tokenizer feeds a 305-million-parameter Conformer encoder. Despite the tokenizer terminology, this path supplies continuous audio embeddings rather than discrete audio token IDs.

The input pipeline progressively extracts and compresses acoustic features:

1. Convert raw audio into a Mel-spectrogram.
2. Split the spectrogram into N Mel chunks.
3. Downsample those chunks through two convolutional layers.
4. Pass the resulting N/4 soft-token embeddings into the Conformer.

This reduction happens before the Conformer processes the audio representations.

Hardin relates the Conformer block to the decoder architectures introduced earlier, with an additional convolutional layer. The audio path thus has its own feature extraction and encoding stages before its representations can support the small models’ text responses.

The four-model lineup maps to two broad deployment needs. E2B and E4B target local applications that consume text, vision, and audio. The larger 26B and 31B models target more complex reasoning, agentic workflows, and coding. The architectural differences matter because the best choice depends on whether device constraints, sparse computation, or larger dense capacity dominate the application.

Hardin closes with two ways to begin:

• Download and self-host: The talk names Hugging Face, Kaggle, and Ollama as distribution options for the models.
• Use hosted larger models: The talk names AI Studio and Vertex for access to 31B and 26B, with agentic prototyping and function calling as initial tasks.

Those are the access paths described for this four-model release. Start with the workload you need to test, then choose whether to operate the model yourself or use the hosted route for that experiment.