Machine-generated source transcript; recognition errors remain. Timestamps and subtitles use the selected source recording’s clock, not clip-relative time. Section links use cited source moments; sections without a verified moment return to the talk start.
7:35:58So today I thought I'll take you through a little journey of how we post train Gemini 4.
7:36:03And our journey begins by people calling us not a frontier model.
7:36:09So here's an example. You know, people say Gemini is cooked, but GCP is cooking.
7:36:15For all intents and purposes, we believe DeepMind is no longer a frontier lab.
7:36:20Ethan Malik, who we respect a lot, he says in his own nice way, pretty long gap between launches is what he calls us.
7:36:30And I'm here to tell you we're not dead.
7:36:34We're here.
7:36:37And we have launched Gemini 4 Argon.
7:36:42We announced Carbon and we launched Carbon all within the last 28 days.
7:36:47So we're on a run here, and that's not it.
7:36:54In the process, we actually garnered an apology from Dylan.
7:36:59He says official apology to Google.
7:37:03They actually cooked. They're not cooked.
7:37:06Ethan says one of the most significant aspects of Gemini 4
7:37:10is the speed at which we're launching things.
7:37:13So we've gone 236 days without having a frontier model,
7:37:17and we've had two frontier models in 28 days.
7:37:21That's a 10x improvement in our speed of shipping.
7:37:24And, you know, people call us Google kind of cooked,
7:37:29but Google's not cooked, so it's good to see that sentiment.
7:37:33It's glad to be on the other end.
7:37:35It's also good to be on the receiving end of the bad comments,
7:37:38because it pushes you to do better.
7:37:41I want to start by playing a little video of a bunch of things that
7:37:46Karvin has built. If we can cue the video, please.
7:37:52There we go. Go back. There we go.
7:38:01Did that work? No.
7:38:08Let me see if going to the laptop is going to fix it.
7:38:50Kudos to the model for building that video itself.
7:38:54So you saw great examples of the model doing much better at 3D, model building assets using
7:39:00blender there's lots of games in there the real world use cases model processing multi-modal
7:39:06documents but you know it's not just the demos carbon's actually number one on a bunch of real
7:39:14world customer use cases and one of the things that we're really excited about is you know it's
7:39:19not just really good on these benchmarks it's actually much cheaper so on cursor bench it's
7:39:27It's the number one model today, and it's 3.3 times cheaper than the second best model.
7:39:32On Perplexity's Wander, which is a benchmark for deep research queries, it's the best model.
7:39:39You saw competitor models come out.
7:39:439x cheaper than the second best model there.
7:39:46And on AutomationBench, which is Zapier's benchmark, again, we're number one, while
7:39:52being two times cheaper per task.
7:39:56and that's not it.
7:39:58I mean, today we announced
7:40:00a new element to our Gemini 4 series, and it's not a surprise. It's hydrogen. So, you know, you might have seen the launch earlier today.
7:40:12Gemini 4 Hydrogen is where we're trying to bring
7:40:15the Gemini 4 intelligence to everyone.
7:40:18It's fast, it's dependable,
7:40:20it's launched on all surfaces,
7:40:23on anti-gravity, on the Gemini API,
7:40:25in the Gemini app, and on search.
7:40:28And, you know, I have a graph that shows the A index
7:40:32which many of you might be familiar with,
7:40:35and time on the x-axis.
7:40:37You can see it's one of the fastest models
7:40:40for that level of intelligence.
7:40:42and to give you a sense of how far you know gemini has come 3.8 flash on terminal bench 4
7:40:50was about 20 percent and hydrogen is about 60 percent so we're three times better performance
7:40:58while being three times faster right so that's no easy task and we're you know really happy to
7:41:06bring that to you in your hands so you can build things with it.
7:41:11And that completes, you know, our Gemini 4 series.
7:41:15So we have Gemini 4 Carbon, a frontier model for deep frontier intelligence, Gemini 4 Hydrogen,
7:41:22which is fast for everyday execution, and we'll have more elements.
7:41:27You know, of course, there are many elements left in the periodic table, and we have a
7:41:31lot of work to do to exhaust them.
7:41:33but we're glad to put it out there for you all to test it
7:41:37and for you all to play with it.
7:41:40Now, I think enough marketing with the models.
7:41:43I was told this is a deeply technical audience,
7:41:45so I want to talk a little bit about
7:41:48why post-training these frontier models is hard
7:41:51and what goes into that.
7:41:53Let me start by saying, you know,
7:41:54AI writes about 75% of Google's new code today,
7:41:58And this is up from less than 25% a few years ago.
7:42:04And there are 100,000 people using anti-gravity writing code inside Google every single day.
7:42:14Now, the complexity of the tasks that people do have also increased exponentially in the last couple of years.
7:42:20We started with, you know, models just completing chats, which took seconds,
7:42:25to writing small functions with the minutes.
7:42:29You can submit a PR and resolve an issue.
7:42:31It takes about an hour, let's say.
7:42:33But now people are doing days of real engineering tasks with the model.
7:42:38People have swarms of agents solving real open problems with the model.
7:42:42We have engineers across Google using anti-gravity
7:42:46and our Gemini 4 models to solve tasks over multiple days.
7:42:51Now how do we go about post-training Gemini for real engineering tasks?
7:42:56We have to start with hard tasks
7:42:58So we really need to source hard problems for the model
7:43:02Grading of these models is actually a very hard problem
7:43:06So you want to grade like an experienced code reviewer would grade
7:43:10You want to grade like an experienced engineer would grade a solution
7:43:14Large-scale RL is how we train the model
7:43:16and we want to dog food our models
7:43:18with our 100,000 engineer community
7:43:20and repeat the loop.
7:43:22So any failures in the dog food
7:43:23become tasks for the next iteration of the loop.
7:43:28So I want to start by going into a little bit more detail
7:43:31of each of these sections.
7:43:32So let's start with the hard tasks.
7:43:35So you might have heard, you know,
7:43:36as models are getting better,
7:43:38lots of benchmarks are getting saturated.
7:43:40And it's not just the benchmarks that get saturated.
7:43:43Our training data gets saturated too.
7:43:45So here is a graph that shows
7:43:47for three different iterations of the Gemini 4 models
7:43:51how much the share of learnable prompts has gone down.
7:43:54So let's say we had X percent of learnable prompts
7:43:57for the first iteration.
7:43:58By the time we finish and train the next one,
7:44:00the prompts are too easy for the second one.
7:44:02We finish that and train the third one,
7:44:04the prompts are too easy for the third one.
7:44:06So what's the problem with training on easy tasks?
7:44:09Easy tasks give you zero learning signal, right?
7:44:11So you have a prompt.
7:44:12When all of your rollouts are going to be correct,
7:44:15There's no variance. There's no RL signal.
7:44:17So you really need to focus on getting hard tasks.
7:44:20And you really need to be faster at getting hard tasks
7:44:23than the models are getting better.
7:44:27So how do we go about it?
7:44:28We first have to identify these easy tasks and prune them.
7:44:32So we cut these simple Sweeband-style tasks from the mixture
7:44:36that are too easy for the models.
7:44:39We expand into much harder Swee tasks, for example,
7:44:43that are multi-file repos.
7:44:45We go from unit test or binary correctness to rubric-based grading.
7:44:51And rubrics give you dense signal.
7:44:53They're not just 0, 1.
7:44:54They can be more gradients within the criteria.
7:45:02Here are some examples of hard tasks that we train on.
7:45:05So we look at multi-file repos and we extract commits from them.
7:45:10From those commits, we have models automatically extract rubrics.
7:45:15We don't actually necessitate unit tests being involved.
7:45:19We can actually generate rubrics on the fly
7:45:22based on these repos.
7:45:23Another example, a great example of tasks,
7:45:27you try to teach the model to build CLI tools from scratch.
7:45:32So you start with an empty folder and a specification.
7:45:35The specification says, hey, you have to build this tool.
7:45:38It's called grep.
7:45:39These are the parameters of grep.
7:45:41These are the flags.
7:45:42This is the functionality.
7:45:43and you let the model build the entire tool from scratch.
7:45:47And you know what the functionality for grep is going to be,
7:45:50so you have a hidden test suite that you can evaluate the binary on.
7:45:54The last example is production codebase TIFFs.
7:45:58So you look at real usage of the model inside your production codebase
7:46:03and extract useful tasks out of them.
7:46:08Now let me talk a little bit about the graders.
7:46:13So agents, you know, they find any way,
7:46:16you're training them to pass the test,
7:46:18so they will find any way to pass the test.
7:46:21But if the test pass, that doesn't necessarily mean
7:46:23that they've solved the problem.
7:46:25There are examples where the agent will bypass the checker.
7:46:28There are examples where the agent will game the benchmark.
7:46:32There are examples where the agent will fake the evidence.
7:46:35You know, it writes the fake evidence into a temp file.
7:46:39It will delete any evidence of it faking it, right?
7:46:43So you have to be smarter about how you judge these models,
7:46:46especially as they become more capable.
7:46:49Now, this is what I meant by grading like a reviewer.
7:46:53So if you've done a code review, you have a very strict reviewer,
7:46:56you're going to know they're not just going to think about the correctness of the solution.
7:47:00They're actually going to think about the style.
7:47:02They're going to think about maintainability.
7:47:05They're going to think of many different aspects.
7:47:07So what we do is we move from unit test-based correctness to rubric-based.
7:47:13When these are generated from real commits, the rubrics also provide dense learning signal, not just 0-1 signal.
7:47:20And we penalize any sort of solution that is looking for a workaround.
7:47:26So here is an example of a rubric where the criteria is, here are the edge cases that need to pass.
7:47:34This is an example of hygiene that needs to be maintained.
7:47:37These are constraints around verification that need to be satisfied.
7:47:43Now, it's not just about the rubrics needed for grading.
7:47:47We also need to be careful about the graders not being gamed.
7:47:52And you really need to harden the environment that the grader is in.
7:47:55So we think about putting the graders in an isolated sandbox.
7:48:00So we remove git history and git remote so the answers don't leak.
7:48:04The test sets are held out.
7:48:05So only when the agent is done and submits the solution
7:48:09do we actually reveal the test.
7:48:11we've had instances where the agent will fake it and say,
7:48:14hey, I'm done, and the tests come into play,
7:48:16and then it'll look at the tests and resubmit the solution.
7:48:18So you don't want that.
7:48:20We also want to be much more detailed with our scoring.
7:48:23So we take the full diff, and we look at the reference,
7:48:26and we look at multiple aspects.
7:48:27It's not just the correctness.
7:48:28We also want to look at the code quality.
7:48:30We also want to look at the scope of the change that was done.
7:48:36And then finally, we want to penalize any sort of cheating attempt
7:48:41by the agent while being careful not to police the thoughts.
7:48:45So any cheating or hampering of the environment
7:48:48gets a very negative RL reward.
7:48:52And all of this is detected just based on the actions
7:48:55that the model is taking, not the thoughts.
7:48:58So a good example of how we have handled cheating
7:49:01is, you know, Carbon, for instance,
7:49:03has 0% cheating rate on Terminal Bench 4.
7:49:06Our previous models had a much higher cheating rate.
7:49:08and it's about 5x less adversarial than argon.
7:49:11So every generation of the model,
7:49:13we're trying to make it much more aligned
7:49:15so it's not trying to cheat.
7:49:18Now finally, I think I want to talk about model behavior
7:49:22a little bit and modeling honesty specifically.
7:49:27Now, agents, they always want to be helpful
7:49:30and sometimes they fake helpfulness.
7:49:32You might have seen this in your usage of the agents.
7:49:35So sometimes they'll claim that tests have passed
7:49:37when they've obviously not passed.
7:49:40They'll declare done,
7:49:42but obviously they're not done.
7:49:44They'll make edits that they'll claim to have made edits,
7:49:47but they've not made those edits.
7:49:48Or they'll invent facts and APIs and whatnot.
7:49:52Or they will give you a wall of text
7:49:54that has so many numbers
7:49:55that is going to be impossible for you to verify.
7:49:57We've gone through all of these examples.
7:50:02Now, here is an example of fabrication rates in Gemini.
7:50:06So we, on an internal eval, started with 8% fabrication in that hard eval.
7:50:13We cut that down dramatically to about 3% for hydrogen, and carbon is half of that,
7:50:20is about 1.5%, but we're striving to get this down to zero.
7:50:23Like you don't want your agents fabricating anything, right?
7:50:26But they try to fabricate because they want to be helpful.
7:50:30And you really have to try hard to align them to be helpful, but not by faking it.
7:50:37How do we reduce the fabrication?
7:50:40Believe it or not, you have to clean your data, right?
7:50:43So we filter out lots of training trajectories where there is, you know,
7:50:48not just intended fabrication, but unintended fabrication.
7:50:52There are lots of claims that are not fully supported.
7:50:56They're not verified.
7:50:57We ground in the full trajectory.
7:50:59So if the agent says, I've completed the task
7:51:01and it has a final response,
7:51:03we actually look at that response
7:51:05and see all of the actions that the agent had taken
7:51:08in the trajectory to get to that final solution,
7:51:10and we look at the steps that were taken.
7:51:12So you really need to, when the agent says, I'm done,
7:51:15and gives you a summary, that needs to be grounded
7:51:17in the actions that were taken.
7:51:19And we want to reward honesty.
7:51:21So, if there's a hard problem, it's better for the agent to say, hey, this is my partial progress.
7:51:27It's a hard problem. I'm done here. This is my partial solution.
7:51:32Rather than faking the last step and saying, hey, I've solved the problem. This is done.
7:51:39Finally, shipping in weeks. So, how are we able to ship so rapidly?
7:51:45And the biggest reason for that is our internal team of 100,000 plus engineers at Google using
7:51:52the model for realistic tasks.
7:51:54You know, many of you might know Google is a mono repo.
7:51:57And what is a mono repo?
7:51:59The entire company is one code base, right?
7:52:03And I don't know how to measure this, but I think it might be one of the largest mono
7:52:07repos in the world.
7:52:08Now, that's a really complex code base.
7:52:11and we have the best people in the world
7:52:15using a really complex code base
7:52:18to do really complex tasks.
7:52:20So every time we have a new candidate from post-training,
7:52:23we actually serve that model for our users.
7:52:27We run blind battles with them
7:52:29through multiple different candidates.
7:52:31We let users pick the best model that they want.
7:52:35We also run Elyse,
7:52:37where under the hood we're routing people
7:52:40to experimental models and get signal and feedback from that and we take all
7:52:46of these signals and friction back into training the next candidate. So this is a
7:52:51cycle that helps us you know rapidly iterate really train on hard problems
7:52:56and as the models get better the engineers are going to push the model in
7:53:00interesting ways. This is also why you know we've gone from releasing in weeks
7:53:08to releasing in quarters to releasing in weeks.
7:53:12So in the Gemini 3 era,
7:53:14we've had launches every two to three months.
7:53:16But now we've had six releases in the last 16 weeks.
7:53:21And we're very excited to have you all try carbon and hydrogen
7:53:26and Gemini 4.
7:53:29And we're excited for us to see what you all do with it.
7:53:34And we're not done.
7:53:35So we have a lot more elements to go through.
7:53:38With that, you know, thank you.
At 07:46:00, Johnson separates multi-file repository commits, from-scratch CLI tools, and production codebase diffs. Each has a different route to a verifiable result.