AI Engineer World's Fair 2024
Unveiling the latest Gemma model advancements
Read the talk
Gemma 2: Practical model sizes, familiar tools, and room to customize
Gemma’s launch-day story connects safety and deployment choices with specialized models, multimodal checkpoints, and the performance claims behind Gemma 2.
From a talk by Kathleen Kenealy
Before you start: Familiarity with language-model inference, fine-tuning, and parameter counts will help; no knowledge of Gemma is required.
What can developers build when they have the model?
How do you put stronger language models into the hands of developers and researchers who want to build on them? That was the starting problem for Gemma. Kathleen Kenealy, the Gemma team’s technical lead at Google DeepMind, describes the project’s goal as enabling community projects, research, and experimentation. The first models launched in February 2024; the work already emerging around them provided the backdrop for the next releases.
Gemma extends a longer sequence of contributions across the machine-learning stack: Transformer research, SentencePiece, and BERT; TPU hardware; and frameworks including TensorFlow and JAX. The family brings that work into lightweight models built from the research and technology behind Gemini. Kenealy uses the term “open source”; more precisely, these are open models distributed under Gemma-specific licensing terms. Access to the models creates opportunities to inspect, adapt, and deploy them, while their license remains a separate consideration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Safety during training, flexibility during deployment
Safety enters the development process before the final evaluation. Kenealy describes three stages:
- Manually inspect training datasets for quality and safety.
- Evaluate safety during early experiments and ablations, using the results to help select training methods.
- Evaluate the final models using the safety evaluations applied to Gemini.
These are development practices, not a guarantee that every customized application will remain safe. Kenealy offers a broader assurance about customized models, but the Gemma 2 model card calls for monitoring during fine-tuning and safeguards suited to the application.
The next design objective is performance relative to size: making a smaller model useful enough to compete with larger alternatives. That objective sits alongside portability. Gemma targets TPUs, GPUs, and local devices, with support across TensorFlow, JAX, Keras, PyTorch, Ollama, and Transformers. These options let a developer choose an execution environment without first committing to a single framework.
Model access and broad distribution complete the picture. The practical promise is that developers can customize Gemma using tools they already prefer, rather than build their application around one hosted interface. Performance matters, but so do the ability to obtain the weights and the freedom to decide where the application runs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Language, code, and recurrent inference
The initial family grew along three distinct directions:
| Model | Purpose |
|---|---|
| Gemma 1.0 | Foundational language models |
| CodeGemma | Specialization for code generation and evaluation |
| RecurrentGemma | An architecture targeting more efficient inference, especially at long contexts |
Kenealy presents CodeGemma as a fine-tuned extension of Gemma 1.0. RecurrentGemma changes the architecture instead: although described here as a state-space model, its Griffin architecture combines local attention with linear recurrences. The distinction matters because it pursues inference efficiency through the model’s structure, rather than only through specialization of its training.
All three branches had received updates by the time of the talk. Gemma 1.1 improved instruction following in chat, CodeGemma received further code-performance improvements, and RecurrentGemma expanded from its original 2B size to include a 9B model. These are separate changes: better conversational behavior, stronger code capabilities, and another model size for recurrent inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
PaliGemma connects vision to a language decoder
Multimodality was the most requested addition from the community. PaliGemma addresses it by combining a SigLIP vision encoder with a Gemma 1.0 text decoder. That pairing supports visual question answering, image and video captioning, object detection, and segmentation. The same broad vision-and-language foundation can therefore be adapted to tasks that ask for descriptions, answers, or information about objects.
Checkpoint choice determines the starting point for that adaptation:
- Pretrained weights: A foundation for task-specific fine-tuning.
- Fine-tuned variants: Models already targeted toward particular tasks, including the detection and segmentation uses Kenealy describes.
- Transfer checkpoints: Models specialized for academic benchmarks.
The talk calls PaliGemma a 2B model. That number identifies its Gemma language component; the complete original vision-language model is 3B, as clarified in the later PaliGemma paper. Pretrained weights should be treated as a tuning starting point, not as interchangeable with a checkpoint already adapted to the intended task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Gemma 2 fits into existing development workflows
Gemma 2 arrived on the day of the presentation, following work informed by community feedback about Gemma 1.0 and 1.1. The launch introduced 9B and 27B models. Kenealy described both as outperforming models two to three times larger, a claim about the launch evaluations rather than a promise of superiority on every task.
Adoption was meant to preserve the developer’s existing workflow. Gemma 2 used the same tooling and broad framework ecosystem as Gemma 1, with Kenealy describing a switch requiring only a few lines of code. Expanded documentation covered inference as well as advanced, efficient fine-tuning.
Fine-tuning was also a model-development objective. The team studied how core modeling decisions affected downstream customization, aiming to make the resulting models easier to adapt. The talk does not specify those architectural or training choices; its emphasis is on making customization an expected use of the model, supported from launch.
For immediate experimentation, the launch offered Gemma 2 27B in Google AI Studio: select the model and begin trying prompts before setting up a local inference environment. This was the hosted entry point available at the June 2024 launch, not a statement about today’s AI Studio model menu.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Benchmarks and human preference answer different questions
Kenealy reports that Gemma 2 27B performed in the same ballpark as Llama 3 70B and exceeded Grok on many benchmarks. The presentation does not provide benchmark names or scores for that comparison. Its useful claim is competitive capability at a smaller parameter count, not uniform dominance across evaluations.
Academic benchmarks are only one part of the evaluation because they may not predict how a model behaves in an application. Kenealy reports that extensive human evaluations strongly preferred Gemma over other open models, including larger ones. She also describes Gemma 27B as the top open model of its size, ranking above named Llama, Nemotron, Grok, and Claude 3 entries. These are launch-time claims; the talk supplies neither preference percentages nor enough comparator-version detail to reconstruct the ranking.
In that LMSYS comparison, Kenealy identifies Yi-Large as the only model she classifies as open that ranked above Gemma 27B. That classification needs care: the later Gemma 2 technical report marks Yi-Large as non-open, and its later leaderboard snapshot is not the conference-day ranking. The practical motivation for discussing human preference remains clear: chat quality needs evaluation beyond academic test scores.
Against earlier Gemma releases, the reported improvements include instruction following, creativity, and factuality. Those dimensions give developers more specific behaviors to examine than a single overall rank: whether the model follows a requested format, produces useful alternatives, and keeps its answers grounded in facts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From trying a model to sharing an application
The Gemma Cookbook provides the next step beyond a prompt experiment. At the time of the talk, Kenealy described 20 recipes ranging from introductory examples to advanced applications. The repository was also accepting pull requests, giving developers a way to contribute their own applications and make them available to the wider community.
Research support accompanied the release. Kenealy encouraged applications for GCP credits from projects of any size; these were applications for support, not guaranteed allocations. The launch program had an August 9, 2024 application deadline. Guides, tutorials, and Colabs covered inference, fine-tuning, and evaluation across frameworks, while downloadable models supported work outside the hosted environment. Discord and other social channels provided ways to discuss what developers were building.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Closing the gap for open-model research
Kenealy closes with the reason she had spent nearly two years on Gemma. As an academic researcher, she had been disappointed by the gap between open foundational language models and rapidly improving proprietary systems. Gemma was an opportunity to build something she wished had been available during that work: stronger models that researchers and developers could use as the basis for their own projects.
The request is not simply to try the release. It is to report what works, what falls short, and what the team should improve next—through GitHub, Hugging Face, social channels, or direct conversation. Community feedback had already helped shape Gemma 2; the applications and research built with it would provide the next set of concrete problems to solve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Griffin-based model implementations with JAX and PyTorch sampling examples and a fine-tuning tutorial.
Overview of coding models for completion, generation and instruction-following applications.
Practical notebooks and application examples for Gemma, with archived examples and contribution guidance.
Further reading
- Gemma 2 launch announcementArticle
The June 2024 launch overview covering model sizes, supported frameworks, AI Studio access and research support.
Architecture, training methods, benchmark comparisons and human evaluations for Gemma 2.
Explains how SigLIP and Gemma form a vision-language model for transfer across diverse tasks.
- Gemma 2 model cardDocumentation
Training information, evaluation metrics, safety findings and limitations for Gemma 2.
Updates since the talk
Guidance on model sizes, pretrained and tuned checkpoints, with links to inference and fine-tuning tutorials.
Read the complete timestamped transcript
- 0:00
[upbeat music] My name is Kathleen Kenealy.
- 0:15
I'm a research engineer at Google DeepMind, and as was just mentioned, I'm the technical lead of the Gemma team. Before I get started, I just wanted to say, uh, how awesome it is to get to be here with you all today.
- 0:30
When we were building Gemma, our North Star, the, the thing we were most excited about, was building something to empower and accelerate the amazing work being done by the open source community.
- 0:43
And since we launched our first models in February, I have been absolutely blown away by the incredible projects and research and, and innovations that have been, already been built on top of Gemma.
- 0:58
Um, so I'm particularly excited to be here with so many developers today, and especially delighted to unveil the latest advancements and additions to the Gemma model family. So without further ado, we'll get started.
- 1:15
As many of you probably know, Google has been a pioneer in publications of AI and ML research for the past decade, including publishing some of the key research that has sparked recent innovations we've seen in AI, research like the Transformer, SentencePiece, BERT, to name a few.
- 1:38
Google DeepMind has really continued this tradition and is actively working to share our research for the world to validate and examine and build upon. But Google's support of the open community for AI and ML is not just limited to publishing research.
- 1:56
We've also been doing work to support ML across the entire technical stack for a long time, from hardware breakthroughs of TPUs, um, which I imagine is especially relevant for this crowd [laughs] and this track, um, all the way to an evolution in ML frameworks from TensorFlow to JAX.
- 2:15
Throughout all of this, open development has been especially critical for Google. Our ability to collaborate with the open source community has helped us all discover more, innovate faster, and really push the limits of what AI is capable of.
- 2:34
So this long history of support of the open source community leads us to today and to Google's latest investment in open models, Gemma.
- 2:45
Gemma is Google DeepMind's family of open source, lightweight, state-of-the-art models, which we build from the same research and technology used to create the Gemini models. I'm so sorry. I think that's my phone going off during this talk [laughs].
- 3:01
Please feel free to rummage through that bag. Wow, lesson learned that even the speaker needs to remember to silence her cell phone. All right, back to Gemma. There are a couple of key advantages of the Gemma models that I want to highlight today.
- 3:18
The first is that Gemma models were built to be responsible from des- by design. I can tell you from personal experience that from day zero of developing a Gemma model, safety is a top priority.
- 3:32
That means we are manually inspecting data sets to make sure that we are not only training on the highest quality data, but also the safest data we can. This means that we are evaluating our models for safety, starting with our earliest experimentation and ablations so that we are selecting training methodolies- methodologies that we know will result in
- 3:55
a safer model. And at the end of our development, our final models are evaluated against the same rigorous state-of-the-art safety evaluations that we evaluate Gemini models against. And we really do this to make sure that no matter where or how you deploy a Gemma model, you can count on the fact that you will have a trustworthy and
- 4:18
responsible AI application. No matter how you've customized Gemma models, you can trust that it will be a responsible model.
- 4:27
Gemma models also achieve unparalleled breakthrough performance for models of their scale, including outperforming significantly larger models. But we'll get to more on that very shortly.
- 4:43
We also designed the Gemma models to be highly extensible so that you can use a Gemma model wherever and however you want. This means they're optimized for TPUs and GPUs, as well as for use on your local device.
- 4:57
They're supported across many frameworks: TensorFlow, JAX, Keras, PyTorch, Ollama, Transformers. You name it, Gemma is probably there.
- 5:07
And finally, the real power of the Gemma models comes from their open access and open license. That, period, that's what's powerful about Gemma. We put state-of-the-art technology into your hands so you can decide what the next wave of innovation looks like.
- 5:26
When we decided to launch the Gemma models, we wanted to make sure that we could meet developers exactly where they are, which is why Gemma models are available anywhere and everywhere you can find an open model.
- 5:40
I will not list all of the frameworks on this slide, but this is only a fraction of the places where you can find Gemma models today. This means you can use Gemma how you need it, when you need it, with the tools that you prefer for development.
- 5:57
Since our initial launch back in February, we've added a couple of different variants to the Gemma model family. We, of course, have our initial models, Gemma 1.0, which are our foundational LLMs.
- 6:10
We also released shortly after that CodeGemma.
- 6:13
Which are the Gemma 1.0 models fine-tuned for improved performance on code generation and code evaluation. And one variant that I am particularly excited about is Recurrent Gemma, which is a novel architecture, a state space model, that's designed for faster and more efficient inference, especially at long contexts.
- 6:35
We've also updated all of these [laughs] models since their initial release. We now have Gemma 1.1, which is better at instruction following in chat. We've updated CodeGemma to have even more improved code performance, and we now have Recurrent Gemma at not only the original 2B size, but also at a nine billion parameter size.
- 6:58
So there's a lot going on [laughs] in the Gemma model family, and I'm especially excited to tell you about our two most recent launches. Um, the first one is actually our most highly requested feature since day zero of launch, and that was multimodality.
- 7:19
So we launched PaliGemma. PaliGemma... [applause] Ooh, thank you. I appreciate it. [laughs]
- 7:26
This is why I love the open source community, truly the most passionate developers that there are. PaliGemma is a combination of the SigLIP vision encoder combined with the Gemma 1.0 text dec- decoder.
- 7:40
This combination allows us to do a variety of image text sort of tasks and capabilities, including question answering, image and video captioning, object detection, and object segmentation. The model comes in a couple of different variants.
- 7:57
It's currently only available at the 2B size. We have pre-trained weights that are available that can be fine-tuned for specific tasks. We have a couple of different fine-tuned variants as well that are already targeted towards things like object detection and object segmentation.
- 8:13
And we also have transfer checkpoints that are models that are specialized to target a couple of academic benchmarks.
- 8:23
Up until this morning, that was our latest release, but I'm very excited to be here today with you guys because it is Gemma 2 launch day. Woo. [laughs] [applause]
- 8:37
Wow, thanks. We have been working very hard on these models since Gemma 1.0 launch date. We tried to do as much as we could to gather feedback from the community to learn where the 1.0 and 1.1 models fell short and what we could do to make them better.
- 8:56
And so we created Gemma 2. Gemma 2 comes in both a nine billion parameter size and a 27 billion parameter size. Both models are, without a doubt, the most performant of their size, and both models also outperform models that are even two to three times larger than these base models.
- 9:21
But Gemma 2 isn't just powerful. It's designed to easily integrate into the workflows that you already have existing. So Gemma 2 uses all of the same tools, all of the same frameworks as Gemma 1, which means if you've already started developing with Gemma 1, you can, with only a couple of lines of code, automatically switch to using
- 9:42
the Gemma 2 models and have increased performance and, um, more power behind your applications. We also have the same broad framework compatibility. Again, TensorFlow, JAX, Transformers, Ollama, all of the ones I previously named, we have them for Gemma 2 as well.
- 10:02
We also have significantly improved documentation. We have more guides, more tutorials so that we can coach you through how to get started, not only with inference, but with advanced and efficient fine-tuning from day zero.
- 10:17
And finally, we really wanted to target fine-tuning as one of the key capabilities of these models. We did extensive research into how our core modeling decisions impact users' ability to do downstream fine-tuning.
- 10:33
So we believe these models are inc- going to be incredibly easy to fine-tune, so you can customize them to whatever your use case may be. In addition, to make it especially easy to get started using Gemma 2 models, we have made the 27B model available in Google AI Studios.
- 10:53
This means you can go to the AI Studio homepage and select Gemma 2 now, if you wanted to, and start playing around with prompts right away. You shouldn't have to do anything except come up with an idea for how you wanna push the limits of our model.
- 11:09
I am, I am especially excited to see what you all end up doing with AI Studios and Gemma. Um, and we have a couple of different ways for you to let us know what you're building, which I'll get to down the road.
- 11:22
Um, but if you have ideas, I'll be here all day and wanna hear what you're doing with the Gemma models.
- 11:29
But let's dive a little bit more into performance. We are incredibly proud of the models that we've made. As I mentioned, they are, without a doubt, the best, most performant models of their size and are also competitive with models two to three times larger.
- 11:46
So our 27B model is, has performance in the same ballpark as Llama 3 70B and outperforms Groq models on many benchmarks by a fairly significant margin [laughs] in some cases.
- 12:02
Um, but I think academic benchmarks are only part of the way that we evaluate Gemma models. Sometimes these benchmarks are not always indicative of w- how a model will perform once it's in your hands.
- 12:15
So we've done extensive human evaluations as well, where we find that the Gemma models are- Consistently, heavily preferred to other open models, including larger open models. Um, and I'm also proud to say that the Gemma 27B model is currently the number one open model of its size, and it currently outranks Llama
- 12:40
370B, Nemotron 340B, Grok, Claude 3, many, many other models as well. Um... [audience applauding] Thank you. Wow, you guys are very supportive. [laughs] I appreciate it.
- 12:56
Um, the only other open model of any size that outperforms the Gemma 27B model is the E large model on, on LMSES. Um, so we expect that you should have some fun playing around with it, especially for chat applications.
- 13:13
We found in our evaluations that the Gemma 2 models are even better at instruction following. They're even more creative. They're better at factuality, better all around than the Gemma 1.0 and 1.1 models.
- 13:28
The other important thing that I wanna make sure to highlight from our most recent launch is the Gemma Cookbook. Cur- the Gemma Cookbook is available on GitHub now and contains 20 different recipes of ranging from easy to very advanced applications of how to use the Gemma models.
- 13:46
And the thing that I am most excited about is the Gemma Cookbook is currently accepting pull requests. So this is a great opportunity to share with us what you're building with the Gemma models, and so we can help share it with the rest of the world.
- 14:03
And of course, I have to say, we also wouldn't mind if you starred the repository. [audience laughing] Come, go take a look and tell us what you're building with Gemma.
- 14:11
So there are a couple of different ways you can get started with the Gemma 2 models. Of course, I just mentioned the cookbook. You can also apply to get, uh, GCP credits to accelerate your research using Gemma 2.
- 14:25
We have a lot of funding available to support research. I would really encourage you to fill out an application regardless of how small or big your project is. We also, as I mentioned, have significantly improved documentation.
- 14:41
We have many guides, tutorials, collabs across every framework, so you can get started doing inference, fine-tuning, and evaluation with Gemma 2 models. You can download them anywhere open models are available, and please chat with us on Discord or other social media channels so we can learn more about what you're building.
- 15:03
And that's about all from me today. I am so excited to see what you all build with Gemma. I have been working on this project for almost two years now and started working on this project because I, as a researcher in academia, was disappointed to see how far behind open foundational LLMs were
- 15:28
compared to the rapid improvements we were seeing in proprietary models. So this is something that's very near and dear to my heart and that I wish I had had when I was actively part of the open source community.
- 15:42
So I'm very excited to see the projects and the research that you all do with these models. Please engage with us on social media, on GitHub, on Hugging Face, here at the event, and let us know what you think of the models.
- 15:56
Let us know what you think we can do better for next time. And thank you all very much. Really appreciate your time. [audience applauding] [outro music]