AI Engineer Europe 2026
Gemma 4 Deep Dive — Cassidy Hardin, Google DeepMind
Read the talk
Inside Gemma 4: Attention, Experts, and Multimodal Inputs
Gemma 4 combines local and global attention, sparse experts, per-layer embeddings, and configurable image processing to serve workloads from phones to reasoning systems.
From a talk by Cassidy Hardin
Before you start: Basic familiarity with transformer layers, attention heads, and token embeddings will help you follow the architecture.
How much model can run locally?
How do you give a model useful reasoning and multimodal capabilities while keeping it small enough to run on a phone or laptop? Gemma 4 approaches that problem with a family of models rather than one architecture scaled uniformly. Cassidy Hardin, who introduces herself as a Google DeepMind researcher, presents four models released the previous week: two effective models for local devices, a mixture-of-experts model, and a larger dense model.
The small E2B and E4B models target phones, iPads, and laptops. 26B A4 introduces mixture-of-experts inference to Gemma, while 31B uses a dense architecture. Hardin reports that both larger models ranked among LM Arena’s top six open-source models at launch. That is a historical leaderboard claim; the supplied evidence does not establish its evaluation conditions.
The release also moves Gemma to Apache 2.0. The practical intent is to make the same model family usable throughout development, from initial experiments through deployment, without changing the licensing framework along the way.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning capacity and active computation
The 31B model targets advanced multimodal reasoning. Hardin reports a number-three global Arena ranking and says it outperformed models more than twenty times its size; these are launch-era comparisons, not established superiority across every task. Its more concrete workflow features are a 256K context window, native thinking, function calling, and structured JSON output. Together, those features support autonomous workflows that need to retain context, choose actions, and produce machine-readable results.
The 26B model separates total model capacity from the parameters used in a forward pass. Hardin gives 3.8 billion active parameters here, after saying 3.9 billion in the opening overview; the model card specifies 3.8 billion. Its router selects eight of 128 experts for each inference pass, allowing sparse computation within the larger parameter set.
The effective models instead prioritize on-device multimodality: they accept text, vision, and audio, but produce text only. Hardin describes improvements over Gemma 3 across agentic capabilities, coding, multimodal tasks, and multilingual tasks, without giving individual benchmark scores in this passage. The rest of the walkthrough explains the architecture behind that positioning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Local windows, global context, and shared keys
Start with the dense decoder used by 31B and the effective models. Gemma 4 interleaves local and global attention layers, generally at a 5:1 local-to-global ratio; E2B uses 4:1. Local attention restricts each token to a sliding window of preceding tokens. Global attention can reach all preceding tokens, and the final layer is always global.
The local window is 512 tokens in the smaller models and 1,024 tokens in the larger models. Restricting the window reduces the work of local attention while allowing the resulting representations to pass into subsequent layers. Periodic global layers retain access to the full preceding context, so the window is not the model’s total context limit.
Global layers still have to attend across that full context, making their keys and values expensive to retain. Grouped-query attention reduces this burden by letting multiple query heads share key and value heads. Hardin illustrates different grouping strengths for local and global attention:
| Attention in the illustration | Queries per shared key/value head | Key/value head dimension |
|---|---|---|
| Local | 2 | 256 |
| Global | 8 | 512 |
The wider global heads compensate for some of the capacity lost by reducing the number of independent key/value heads. These grouping ratios should not be treated as a universal configuration for every model: the published E2B configuration specifies eight query heads and one key/value head, while retaining the 256/512 local/global dimensions.
The two attention changes address different costs. Sliding windows reduce how much history local layers inspect; shared key/value heads reduce the state required by attention heads, including those in global layers. Hardin credits the global 8:1 grouping with improved efficiency, but gives no measured memory or latency savings here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One shared expert and eight routed experts
The mixture-of-experts architecture changes the decoder’s feed-forward component. Instead of sending each representation through one dense feed-forward network, it uses a collection of smaller feed-forward networks called experts. A shared expert runs on every pass.
Alongside that shared computation, the router selects eight experts from a pool of 128. The shared expert is three times the size of a regular routed expert in Hardin’s description. The detailed structure is therefore 128 routed experts plus one shared expert, consistent with the model card—not 128 including the shared expert. Only the selected routed experts and the shared expert participate in a given pass, which is the mechanism behind the distinction between total and active parameters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Effective size and flash-backed embeddings
The effective models make a different distinction between the parameters required to operate the model and its total representational parameters. E2B has 2.3 billion effective parameters and 5.1 billion parameters including embeddings. These are parameter counts, not literal network depth or a measured device-memory footprint. The goal is useful local inference on phones and laptops without requiring calls to a remotely served model.
Per-layer embeddings, or PLE, add a dedicated embedding table at each layer. They do not replace the standard token embedding table: a token ID still maps to an ordinary model-width vector. That vector has 1,536 dimensions in E2B and 2,560 dimensions in E4B.
Each additional PLE table also covers the vocabulary, but holds a smaller representation for each token at that particular layer. Hardin describes storing these tables in flash memory rather than VRAM, addressing a central constraint of local devices. This is the storage strategy described in the talk; whether a deployment keeps those tables outside accelerator memory depends on its implementation. PLE vectors have only 256 dimensions, substantially narrower than the standard token embeddings.
Follow the token hi through the model. Its token identity stays the same, but each layer supplies a different learned PLE vector. Hardin describes 35 layers for E2B and 42 for E4B. At the end of a decoder block, the model looks up the token’s 256-dimensional vector for that layer, then projects it to the model’s full embedding width: 1,536 for E2B or 2,560 for E4B. The model can therefore draw on additional token-specific representations throughout the stack without requiring all of those tables to reside in scarce VRAM under the described storage arrangement.
Hardin attributes stronger performance than prior small Gemma generations to this PLE design. The accompanying diagram completes the mechanism: a selected table row becomes a 256-dimensional vector, passes through a projection, and enters the per-layer embedding component of the decoder block.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Image patches need spatial positions
The multimodal progression begins with vision in Gemma 3, followed by audio, vision, and text in Gemma 3n. Gemma 4 integrates multimodality from the beginning. Its larger models use a 550-million-parameter vision encoder; E2B and E4B use a smaller 150-million-parameter encoder.
Variable aspect ratios and variable resolutions give developers control over the image resolution and soft-token budget allocated to an input, with five supported options. To see why those choices matter, start with the encoder’s input: an image is divided into 16×16-pixel patches. Each patch is flattened and linearly projected into a patch embedding, with positional information incorporated so the model can interpret the image’s arrangement.
Hardin compares 4×2 and 3×3 patch arrangements. The patch numbered four occupies a different spatial location when the grid shape changes. A sequence index alone cannot tell the model where that patch sits relative to its neighbors; the representation also needs spatial positional information. That is what lets different image shapes remain geometrically meaningful after conversion into a sequence.
Resolution introduces a separate choice. Two images can both have a 3:2 aspect ratio while containing very different amounts of detail. A larger image budget lets the model process more of that detail, but consumes more of the available token budget.
- OCR and spatial object recognition: Allocate a larger budget to preserve the visual detail the task depends on.
- Text-focused applications: Use a smaller image budget when visual detail contributes little to the task.
The budget is therefore a workload decision, not simply an image-format setting.
This changes the handling of images compared with Gemma 3’s pan-and-scan approach. Hardin describes splitting an image into square regions, padding as needed, and passing a single source image as two, three, or four images processed sequentially. Gemma 4 instead accommodates a varying number of patches for the supplied image, supporting its aspect ratio and resolution without relying on that same sequence of square crops.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Nine patches become one soft token
The language model does not receive one input embedding for every original patch. Gemma 4 pools each 3×3 group of patches into one embedding. A budget of 280 image soft tokens corresponds to a patch budget of 2,520. This is the budget correspondence, not a guarantee that every image produces exactly that many patches. Padding handles image dimensions that do not fit cleanly into 16×16-pixel patches.
The square-image comparison in the talk covers five supported resolution and budget settings. Hardin highlights budgets of 560 and 1,120 soft tokens for object detection and OCR. The same pooling arithmetic lets you compare their patch capacity:
python
patches_per_soft_token = 3 * 3
for soft_token_budget in (280, 560, 1120):
patch_budget = soft_token_budget * patches_per_soft_token
print(f"{soft_token_budget} soft tokens: {patch_budget} patches")
Increasing this budget gives detailed visual tasks more room before pooling compresses the representation for the language model.
The complete path is N image patches → N patch embeddings → N/9 pooled soft tokens → projection to language-model inputs. In the final diagram, the patches pass through GemmaVis ViT; the pooled tokens then pass through Linear Projection and RMSNorm before entering Gemma 4. Patch encoding preserves visual information at the vision stage, while pooling reduces the sequence length that the language model must consume.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From raw audio to continuous embeddings
Audio support in E2B and E4B targets translation and speech recognition. An audio tokenizer feeds a 305-million-parameter Conformer encoder. Despite the tokenizer terminology, this path supplies continuous audio embeddings rather than discrete audio token IDs.
The input pipeline progressively extracts and compresses acoustic features:
- Convert raw audio into a Mel-spectrogram.
- Split the spectrogram into N Mel chunks.
- Downsample those chunks through two convolutional layers.
- Pass the resulting N/4 soft-token embeddings into the Conformer.
This reduction happens before the Conformer processes the audio representations.
Hardin relates the Conformer block to the decoder architectures introduced earlier, with an additional convolutional layer. The audio path thus has its own feature extraction and encoding stages before its representations can support the small models’ text responses.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the workload, then the hosting path
The four-model lineup maps to two broad deployment needs. E2B and E4B target local applications that consume text, vision, and audio. The larger 26B and 31B models target more complex reasoning, agentic workflows, and coding. The architectural differences matter because the best choice depends on whether device constraints, sparse computation, or larger dense capacity dominate the application.
Hardin closes with two ways to begin:
- Download and self-host: The talk names Hugging Face, Kaggle, and Ollama as distribution options for the models.
- Use hosted larger models: The talk names AI Studio and Vertex for access to 31B and 26B, with agentic prototyping and function calling as initial tasks.
Those are the access paths described for this four-model release. Start with the workload you need to test, then choose whether to operate the model yourself or use the hosted route for that experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Ollama's Gemma 4 library entry for local model access.
Further reading
- Gemma 4 E2B model configurationRepository
Inspect the E2B checkpoint's attention schedule, embedding dimensions, and vision and audio configuration.
Updates since the talk
- Gemma 4 model cardDocumentation
Model specifications, supported modalities, evaluation results, image budgets, and prompting guidance, including the later 12B variant.
The team's subsequent report explains architecture, multimodal encoders, training, evaluations, and image preprocessing.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hi, everyone.
- 0:15
My name is Cassidy, and I'm a researcher at Google DeepMind. Today, I'm really excited to share with you some of the technical improvements and architecture that we have with Gemma 4.
- 0:28
Last week, we launched Gemma 4, which is the latest addition to our family of open source models. Gemma 4 brought incredible improvements at, at a scale that has not been seen before.
- 0:41
We have a family of very small models with incredible performance, setting a new precedent for what's possible with small open source models.
- 0:51
Gemma 4 comes in four sizes. We have two smaller effective models, which are geared towards on-device applications. These models have been adapted and, and improved in order to provide incredible performance at a small scale, which are able to run locally on phones, iPads, and laptops.
- 1:11
We have two larger models, starting with a 26B mixture of experts model, which is the first ever Gemma MoE. This model has been adapted to have incredible performance while only requiring three point nine billion active parameters.
- 1:26
And our largest model is our 31B dense. This has insane performance, a huge improvement upon what existed within Gemma 3, at a new precedent that hasn't been seen before.
- 1:39
Taking a look at our larger models, our 31B and our 26B, these models have both ranked in the top six of all open source models on the LM, the LM arena.
- 1:54
One of the most exciting improvements and things that we've launched alongside Gemma 4 is the move to an Apache 2.0 license. This was deliberately done in order to make our models more accessible for the everyday developer.
- 2:06
You should easily be able to integrate Gemma into your life cycle of development through initial testing all the way to deployment and building within the Gemma i- in, in, within the Gemma universe.
- 2:18
Now, let's take a little look at what each of these models are and some of the use cases that we've adapted this for. Starting with our 31B dense model.
- 2:27
This is a state-of-the-art multimodal model, which has been purposely built for advanced reasoning. This model ranked number three on the global arena for the AI leaderboard. This is outperforming models over twenty times its size.
- 2:42
This is a huge improvement. The 31B has a 256K context length, which has been purpose-built for autonomous workflows with native support for thinking, function calling, and structured JSON outputs.
- 2:59
We also have a slightly smaller 26B. This 26B is the first edition of a mixture of experts model into the Gemma family. Only requiring three point eight billion parameters during any forward pass, this model is small and efficient.
- 3:16
Utilizing a total of a hundred and twenty-eight experts while only requiring eight experts during any inference, this is efficient for running while still maintaining some of the incredible performance that we saw with our 31B.
- 3:30
On the smaller side, we introduced two effective models. These models are geared towards on-device applications with the additional support of audio. These are vision, text, and image... vision, text, and audio input models while remaining being text-only output models.
- 3:47
Similarly, we have our effective 2B model. Across a variety of benchmarks, these models are incredible. Looking at our performance across agentic capabilities, coding, multimodal, multilingual, we've truly set a new frontier for what's capable with the Gemma models.
- 4:07
This is significantly outperforming everything we had with the Gemma 3 family of models.
- 4:13
Now, let's take a look at what's actually new in Gemma 4, and what have we done and how have we actually been able to achieve this incredible performance, starting on the architecture side.
- 4:24
We have our standard dense model. This is our 31B, as well as our smaller effective 2B and 4B models. We have our standard decoder block. What we've done with Gemma 4 is we've made several improvements within attention.
- 4:38
We've introduced a five-to-one ratio of interleaving local to global layers with our smaller effective 2B having a four-to-one ratio. This means that within our local layers, we have a sliding window of how many tokens we're attending to.
- 4:53
And lastly, with our global layers, we've now ensured that the last layer is always a global layer, meaning that our last layer is attending to all preceding tokens.
- 5:03
In practice, what this looks like is our global layers are attending to every token that is preceded within this, whereas our local models are only attending to a specific number of preceding tokens.
- 5:15
In our smaller models, we have a sliding window of five hundred and twelve tokens, while in our larger models, we have a sliding window of one thousand and twenty-four tokens.
- 5:25
This sliding window has provided significant improvements in the efficiency and optimizations of our local layers while still maintaining passing through information to the preceding layers.
- 5:37
However, our global layers remain to be quite expensive. Despite this interleaving of local and global layers, all of our global layers are still required to attend to all preceding tokens, which makes it quite memory-intensive and expensive to run.
- 5:51
And this is where we've looked into introducing grouped-query attention. Within our local layers, we group together two queries to share the same key and value heads. However, in our global layers, we're grouping together eight queries sharing the same key and value heads.
- 6:09
Since reducing the number of key and value heads can have a big impact on performance, we've doubled the length of the key value heads within our global layers to have a length of 512 as opposed to 256 as used within the local layers.
- 6:24
This grouped-query attention has provided significant performance improvements without massive memory cost and inference increases in the cost of being able to serve these models. This ratio of eight queries to one key value across all of our models has provided significant improvements in the efficiency of our global layers.
- 6:48
These attention changes were present across all of our models, but we also introduced a new architecture with Gemma 4, and this is our MoE.
- 6:57
Our MoE has one shared router expert with a total of 128 total experts with eight activated experts on each forward pass. All of these experts are small fed forward neural networks.
- 7:11
Similar to the architecture that we had with our dense model, we've now replaced that standard fed forward neural network with an MoE. In practice, what this looks like is having our constant shared expert, which is activated on every pass of the model.
- 7:27
This shared expert is three times the size of our nor-- of our regular experts. We then have 128 experts, each of which-- eight of which are selected by the router during any pass of the model.
- 7:42
And lastly, on an architecture side, we get into our dense effective models. So what does it mean when we're saying our model is effectively 2B, effectively 4B? This is where we're looking at the difference in the number of parameters which are required to operate the model as opposed to the total number of representational parameters present.
- 8:02
Our 2B is effectively two point three billion parameters, while having a representational depth of five point one billion parameters. These models have been specifically designated and optimized in order to have the best on-device performance.
- 8:17
These are designed to run on phones, run on laptops without requiring expensive API calls to models served somewhere else in the world.
- 8:25
These advancements were made possible through PLE. This is our Per Layer Embeddings, where within each of our layers, we now have a dedicated embedding table.
- 8:36
Before digging into exactly how our per layer, layer embedding table looks, let's take a look at how the entire token embedding layer works. This hasn't been replaced within our effective models.
- 8:48
We still have your standard embedding table where you're looking at the mapping of an ID for a token in toward, towards its embedding vector. In our E2B, we have an embedding vector size of 1,536, and in our larger E4B, we have an embedding vector size of 2,560.
- 9:07
This is where we're storing the vector embeddings of each of these tokens.
- 9:13
Now, we also have a per-embedding table-- a per-layer embedding table. This is where similarly, we have our entire vocabulary size and have an embedding representation for each of these tokens, but now we also have one of these for each of the layers.
- 9:27
The big advancement that comes here is the fact that we store our PLE, our Per Layer Embedding table, in flash memory as opposed to VRAM. VRAM is one of the largest constraints on on-device.
- 9:38
It's where you quickly run out of memory in phones and laptops. So by requiring that we no longer need to store this additional embedding table in VRAM, and we can store it in flash memory, we're able to get incredible improvements on the inference side without having an expensive cost of this additional storage and memory.
- 9:57
The big difference in our PLE embedding table as opposed to the standard embedding table is our embedding dimension now is only 256. So this is significantly reduced from the size of the full model.
- 10:10
For each token, for example, hi, we now have an embedding for this token at each separate layer within the model. And as you progress through the layers of the models, 35 for the 2B and 42 for our larger model, you'll see the progression and improvements in the embedding representation for each of these tokens at the next subsequential
- 10:30
layer. And how does this work in practice? Now, at the end of our decoder block, we're able to look up the per-layer embedding for each of our tokens. This is where we're able to look up the 256 dimension and project this up to the full embedding size that's expected for each of our models.
- 10:49
Ultimately, these improvements with PLE allow for our E2B and our E4B to be significantly outperforming prior generations of Gemma small models.
- 11:02
In addition to everything that we've done on an architecture side, we've also really set a new frontier for what's possible with multimodality. In Gemma 3, we introduced Vision for the first time.
- 11:14
We added Vision in Gemma 3, adding support across all of our sizes, and in the Gemma 3N model, we launched open-sourced audio, vision, and text model for the first time.
- 11:24
This really paved the way for Gemma 4 being natively multimodal models. Multimodal has been integrated from the very beginning across all of these models with quite incredible performance.
- 11:36
Starting on the Vision side, our 31B model u-- and 26B model both use a 550 million parameter vision encoder. Our Effective 2B and Effective 4B have a smaller, compact, more designed encoder of 150 million parameters.
- 11:54
We've made big strides and improvements from Gemma 3 by the introduction of variable aspect ratios and variable resolutions. What this means in practice for you as developers is you now have a choice when you're running a Gemma model to select the resolution and the soft token budget that you want to allocate for images.
- 12:12
This is available in five different resolutions across all of our models.
- 12:19
Let's take a second to revisit how our vision encoder works. What we want to understand here is we get an initial image, and this image is split into patches, patches of 16 by 16 pixels.
- 12:31
These patches are then flattened and linear projected up into patch embeddings. These embeddings are then transformed to account for their positional encodings, and this is what's ultimately showed to our model.
- 12:44
With this in mind, let's take a look back at variable aspect. We can have images of different variable aspects where we have an image of four by two and an image of three by three.
- 12:55
What's important to under- Next. Okay, what's important to understand here now is patch four is now in a completely different position. If we're encoding both of these images in the same way, we need our model to understand that patch four in the one on the left is in the second row, it's right below our first, versus patch
- 13:16
four in the image on the right is now at the very end. So what's been a critical development here is that we need to ensure that their spatial positional encoding is also passed through to our model.
- 13:28
In addition to variable aspect ratios, we've also had to introduce variable resolution. This is where, despite the fact that both of our images have a ratio of three to two, they're both very different resolutions.
- 13:42
We have a higher resolution on the right with a lower resolution on the left, and this is where we introduce the variable resolutions where you can select how many images and token-- how many tokens you want to allocate to each image.
- 13:56
This is critical because it allows you to determine how much of your token budget you want to be spent on images. For tasks such as OCR, spatial object recognition, you want to allocate a much higher budget to ensure that you're processing high quality, high resolution images.
- 14:12
If your applications are purely text-based and you're not using some of the multimodal capabilities, you can allocate a much smaller and low-lower token budget.
- 14:22
What's important to understand here is this is a huge improvement from Gemma 3. In Gemma 3, we had to introduce pan and scan, and this is where you would give an image of variable resolutions, variable aspect ratios, and we split it into a sequence of squares and padded whatever else was necessary.
- 14:39
Then your one image was passed to the model as two, three, four different images, which were each processed sequentially. Now, with these advancements across variable resolution and variable aspect ratios, we're now able to process images with varying number of patches, depending upon the actual image which was provided by the user.
- 15:00
The next thing to understand here is, how do we actually take these patches of images and patch-- and pass this through to the model? This is where we take three by three grids of patches, and they become a single embedding, and this single embedding is then passed forward to a model.
- 15:16
So if you're running a Gemma model with a token budget of two hundred and eighty, this actually equates to two thousand five hundred and twenty different patches which will be constructed from each of your images.
- 15:27
For anything that doesn't fit within a sequence of 16 by 16 patches or 16 by 16 pixels for each of the patches, we'll provide additional padding.
- 15:39
For an example of square images across the five different soft token budgets and resolutions which are supported with Gemma, here's an instance of the resolutions, patches, and the pooled elements embeddings which would exist at each of these different sizes.
- 15:54
And this is where you really see that if you're doing something such as object detection, OCR, you really want to be running with a higher quality resolution and token budget, and this is where we have five sixty and eleven twenty available for these use cases.
- 16:11
And pulling this all together, we start with an image of N patches. These patches are each turned into N patch embeddings. We then pool this together to produce N over nine soft tokens, and these soft tokens are linear projected up to what is actually showed to our models.
- 16:31
Now for the audio side. Audio has been added into the E2B and the E4B with the goal of being able to support translation and speak rec-- speech recognition. This is made possible through the combination of an audio tokenizer and a conformer.
- 16:47
This is a three hundred and five million parameter conformer which processes as an audio encoder, processing embeddings rather than tokens.
- 16:58
On the audio tokenizer side, we start with raw audio. This raw audio is run through a Mel-spectrogram, which is able to process fea-features out of the raw audio files.
- 17:09
This Mel-spectrogram is then split into N Mel chunks, which are downsampled through two convolutional layers. This ultimately outputs N over four soft tokens. These audio embeddings are what's passed forward into the conformer.
- 17:25
Our conformer follows a similar architecture to what we've already seen across the dense models and the MoE architecture, although this time we're adding in a convolutional layer.
- 17:39
Returning back to where we started today, the Gemma family of models has four models which we've released to you externally last week. We have two smaller models which have been geared towards on-device applications with support for text, vision, and audio.
- 17:54
Our two larger models are designed for more complex and reasoning tasks with support for agentic workflows and coding. And that's what brings us to what can you do with Gemma today, and how can you get started?
- 18:07
There's two main options for getting started with Gemma. You have the option to download and self-host all of our models. These are available across Hugging Face, Kaggle, and Ollama.
- 18:17
We also have cloud-hosted options for our larger models, our 31B and our 26B, and these can be accessible across AI Studio and Vertex. This is what's gonna allow you to immediately get started with prototyping, building agentic workflows, and testing out function calling in these larger models.
- 18:37
Thank you, and I'm happy to answer any questions about Gemma. [applause] [outro music]