# Train on the engineering work that still fails

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[07:35:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27358s) So today I thought I'll take you through a little journey of how we post train Gemini 4.

[07:36:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27363s) And our journey begins by people calling us not a frontier model.

[07:36:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27369s) So here's an example. You know, people say Gemini is cooked, but GCP is cooking.

[07:36:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27375s) For all intents and purposes, we believe DeepMind is no longer a frontier lab.

[07:36:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27380s) Ethan Malik, who we respect a lot, he says in his own nice way, pretty long gap between launches is what he calls us.

[07:36:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27390s) And I'm here to tell you we're not dead.

[07:36:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27394s) We're here.

[07:36:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27397s) And we have launched Gemini 4 Argon.

[07:36:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27402s) We announced Carbon and we launched Carbon all within the last 28 days.

[07:36:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27407s) So we're on a run here, and that's not it.

[07:36:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27414s) In the process, we actually garnered an apology from Dylan.

[07:36:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27419s) He says official apology to Google.

[07:37:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27423s) They actually cooked. They're not cooked.

[07:37:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27426s) Ethan says one of the most significant aspects of Gemini 4

[07:37:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27430s) is the speed at which we're launching things.

[07:37:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27433s) So we've gone 236 days without having a frontier model,

[07:37:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27437s) and we've had two frontier models in 28 days.

[07:37:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27441s) That's a 10x improvement in our speed of shipping.

[07:37:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27444s) And, you know, people call us Google kind of cooked,

[07:37:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27449s) but Google's not cooked, so it's good to see that sentiment.

[07:37:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27453s) It's glad to be on the other end.

[07:37:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27455s) It's also good to be on the receiving end of the bad comments,

[07:37:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27458s) because it pushes you to do better.

[07:37:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27461s) I want to start by playing a little video of a bunch of things that

[07:37:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27466s) Karvin has built. If we can cue the video, please.

[07:37:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27472s) There we go. Go back. There we go.

[07:38:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27481s) Did that work? No.

[07:38:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27488s) Let me see if going to the laptop is going to fix it.

[07:38:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27530s) Kudos to the model for building that video itself.

[07:38:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27534s) So you saw great examples of the model doing much better at 3D, model building assets using

[07:39:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27540s) blender there's lots of games in there the real world use cases model processing multi-modal

[07:39:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27546s) documents but you know it's not just the demos carbon's actually number one on a bunch of real

[07:39:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27554s) world customer use cases and one of the things that we're really excited about is you know it's

[07:39:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27559s) not just really good on these benchmarks it's actually much cheaper so on cursor bench it's

[07:39:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27567s) It's the number one model today, and it's 3.3 times cheaper than the second best model.

[07:39:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27572s) On Perplexity's Wander, which is a benchmark for deep research queries, it's the best model.

[07:39:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27579s) You saw competitor models come out.

[07:39:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27583s) 9x cheaper than the second best model there.

[07:39:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27586s) And on AutomationBench, which is Zapier's benchmark, again, we're number one, while

[07:39:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27592s) being two times cheaper per task.

[07:39:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27596s) and that's not it.

[07:39:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27598s) I mean, today we announced

[07:40:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27600s) a new element to our Gemini 4 series, and it's not a surprise. It's hydrogen. So, you know, you might have seen the launch earlier today.

[07:40:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27612s) Gemini 4 Hydrogen is where we're trying to bring

[07:40:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27615s) the Gemini 4 intelligence to everyone.

[07:40:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27618s) It's fast, it's dependable,

[07:40:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27620s) it's launched on all surfaces,

[07:40:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27623s) on anti-gravity, on the Gemini API,

[07:40:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27625s) in the Gemini app, and on search.

[07:40:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27628s) And, you know, I have a graph that shows the A index

[07:40:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27632s) which many of you might be familiar with,

[07:40:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27635s) and time on the x-axis.

[07:40:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27637s) You can see it's one of the fastest models

[07:40:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27640s) for that level of intelligence.

[07:40:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27642s) and to give you a sense of how far you know gemini has come 3.8 flash on terminal bench 4

[07:40:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27650s) was about 20 percent and hydrogen is about 60 percent so we're three times better performance

[07:40:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27658s) while being three times faster right so that's no easy task and we're you know really happy to

[07:41:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27666s) bring that to you in your hands so you can build things with it.

[07:41:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27671s) And that completes, you know, our Gemini 4 series.

[07:41:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27675s) So we have Gemini 4 Carbon, a frontier model for deep frontier intelligence, Gemini 4 Hydrogen,

[07:41:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27682s) which is fast for everyday execution, and we'll have more elements.

[07:41:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27687s) You know, of course, there are many elements left in the periodic table, and we have a

[07:41:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27691s) lot of work to do to exhaust them.

[07:41:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27693s) but we're glad to put it out there for you all to test it

[07:41:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27697s) and for you all to play with it.

[07:41:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27700s) Now, I think enough marketing with the models.

[07:41:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27703s) I was told this is a deeply technical audience,

[07:41:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27705s) so I want to talk a little bit about

[07:41:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27708s) why post-training these frontier models is hard

[07:41:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27711s) and what goes into that.

[07:41:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27713s) Let me start by saying, you know,

[07:41:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27714s) AI writes about 75% of Google's new code today,

[07:41:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27718s) And this is up from less than 25% a few years ago.

[07:42:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27724s) And there are 100,000 people using anti-gravity writing code inside Google every single day.

[07:42:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27734s) Now, the complexity of the tasks that people do have also increased exponentially in the last couple of years.

[07:42:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27740s) We started with, you know, models just completing chats, which took seconds,

[07:42:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27745s) to writing small functions with the minutes.

[07:42:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27749s) You can submit a PR and resolve an issue.

[07:42:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27751s) It takes about an hour, let's say.

[07:42:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27753s) But now people are doing days of real engineering tasks with the model.

[07:42:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27758s) People have swarms of agents solving real open problems with the model.

[07:42:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27762s) We have engineers across Google using anti-gravity

[07:42:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27766s) and our Gemini 4 models to solve tasks over multiple days.

[07:42:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27771s) Now how do we go about post-training Gemini for real engineering tasks?

[07:42:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27776s) We have to start with hard tasks

[07:42:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27778s) So we really need to source hard problems for the model

[07:43:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27782s) Grading of these models is actually a very hard problem

[07:43:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27786s) So you want to grade like an experienced code reviewer would grade

[07:43:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27790s) You want to grade like an experienced engineer would grade a solution

[07:43:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27794s) Large-scale RL is how we train the model

[07:43:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27796s) and we want to dog food our models

[07:43:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27798s) with our 100,000 engineer community

[07:43:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27800s) and repeat the loop.

[07:43:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27802s) So any failures in the dog food

[07:43:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27803s) become tasks for the next iteration of the loop.

[07:43:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27808s) So I want to start by going into a little bit more detail

[07:43:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27811s) of each of these sections.

[07:43:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27812s) So let's start with the hard tasks.

[07:43:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27815s) So you might have heard, you know,

[07:43:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27816s) as models are getting better,

[07:43:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27818s) lots of benchmarks are getting saturated.

[07:43:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27820s) And it's not just the benchmarks that get saturated.

[07:43:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27823s) Our training data gets saturated too.

[07:43:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27825s) So here is a graph that shows

[07:43:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27827s) for three different iterations of the Gemini 4 models

[07:43:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27831s) how much the share of learnable prompts has gone down.

[07:43:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27834s) So let's say we had X percent of learnable prompts

[07:43:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27837s) for the first iteration.

[07:43:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27838s) By the time we finish and train the next one,

[07:44:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27840s) the prompts are too easy for the second one.

[07:44:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27842s) We finish that and train the third one,

[07:44:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27844s) the prompts are too easy for the third one.

[07:44:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27846s) So what's the problem with training on easy tasks?

[07:44:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27849s) Easy tasks give you zero learning signal, right?

[07:44:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27851s) So you have a prompt.

[07:44:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27852s) When all of your rollouts are going to be correct,

[07:44:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27855s) There's no variance. There's no RL signal.

[07:44:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27857s) So you really need to focus on getting hard tasks.

[07:44:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27860s) And you really need to be faster at getting hard tasks

[07:44:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27863s) than the models are getting better.

[07:44:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27867s) So how do we go about it?

[07:44:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27868s) We first have to identify these easy tasks and prune them.

[07:44:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27872s) So we cut these simple Sweeband-style tasks from the mixture

[07:44:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27876s) that are too easy for the models.

[07:44:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27879s) We expand into much harder Swee tasks, for example,

[07:44:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27883s) that are multi-file repos.

[07:44:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27885s) We go from unit test or binary correctness to rubric-based grading.

[07:44:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27891s) And rubrics give you dense signal.

[07:44:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27893s) They're not just 0, 1.

[07:44:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27894s) They can be more gradients within the criteria.

[07:45:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27902s) Here are some examples of hard tasks that we train on.

[07:45:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27905s) So we look at multi-file repos and we extract commits from them.

[07:45:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27910s) From those commits, we have models automatically extract rubrics.

[07:45:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27915s) We don't actually necessitate unit tests being involved.

[07:45:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27919s) We can actually generate rubrics on the fly

[07:45:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27922s) based on these repos.

[07:45:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27923s) Another example, a great example of tasks,

[07:45:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27927s) you try to teach the model to build CLI tools from scratch.

[07:45:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27932s) So you start with an empty folder and a specification.

[07:45:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27935s) The specification says, hey, you have to build this tool.

[07:45:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27938s) It's called grep.

[07:45:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27939s) These are the parameters of grep.

[07:45:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27941s) These are the flags.

[07:45:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27942s) This is the functionality.

[07:45:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27943s) and you let the model build the entire tool from scratch.

[07:45:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27947s) And you know what the functionality for grep is going to be,

[07:45:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27950s) so you have a hidden test suite that you can evaluate the binary on.

[07:45:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27954s) The last example is production codebase TIFFs.

[07:45:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27958s) So you look at real usage of the model inside your production codebase

[07:46:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27963s) and extract useful tasks out of them.

[07:46:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27968s) Now let me talk a little bit about the graders.

[07:46:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27973s) So agents, you know, they find any way,

[07:46:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27976s) you're training them to pass the test,

[07:46:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27978s) so they will find any way to pass the test.

[07:46:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27981s) But if the test pass, that doesn't necessarily mean

[07:46:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27983s) that they've solved the problem.

[07:46:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27985s) There are examples where the agent will bypass the checker.

[07:46:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27988s) There are examples where the agent will game the benchmark.

[07:46:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27992s) There are examples where the agent will fake the evidence.

[07:46:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27995s) You know, it writes the fake evidence into a temp file.

[07:46:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=27999s) It will delete any evidence of it faking it, right?

[07:46:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28003s) So you have to be smarter about how you judge these models,

[07:46:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28006s) especially as they become more capable.

[07:46:49](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28009s) Now, this is what I meant by grading like a reviewer.

[07:46:53](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28013s) So if you've done a code review, you have a very strict reviewer,

[07:46:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28016s) you're going to know they're not just going to think about the correctness of the solution.

[07:47:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28020s) They're actually going to think about the style.

[07:47:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28022s) They're going to think about maintainability.

[07:47:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28025s) They're going to think of many different aspects.

[07:47:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28027s) So what we do is we move from unit test-based correctness to rubric-based.

[07:47:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28033s) When these are generated from real commits, the rubrics also provide dense learning signal, not just 0-1 signal.

[07:47:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28040s) And we penalize any sort of solution that is looking for a workaround.

[07:47:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28046s) So here is an example of a rubric where the criteria is, here are the edge cases that need to pass.

[07:47:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28054s) This is an example of hygiene that needs to be maintained.

[07:47:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28057s) These are constraints around verification that need to be satisfied.

[07:47:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28063s) Now, it's not just about the rubrics needed for grading.

[07:47:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28067s) We also need to be careful about the graders not being gamed.

[07:47:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28072s) And you really need to harden the environment that the grader is in.

[07:47:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28075s) So we think about putting the graders in an isolated sandbox.

[07:48:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28080s) So we remove git history and git remote so the answers don't leak.

[07:48:04](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28084s) The test sets are held out.

[07:48:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28085s) So only when the agent is done and submits the solution

[07:48:09](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28089s) do we actually reveal the test.

[07:48:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28091s) we've had instances where the agent will fake it and say,

[07:48:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28094s) hey, I'm done, and the tests come into play,

[07:48:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28096s) and then it'll look at the tests and resubmit the solution.

[07:48:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28098s) So you don't want that.

[07:48:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28100s) We also want to be much more detailed with our scoring.

[07:48:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28103s) So we take the full diff, and we look at the reference,

[07:48:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28106s) and we look at multiple aspects.

[07:48:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28107s) It's not just the correctness.

[07:48:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28108s) We also want to look at the code quality.

[07:48:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28110s) We also want to look at the scope of the change that was done.

[07:48:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28116s) And then finally, we want to penalize any sort of cheating attempt

[07:48:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28121s) by the agent while being careful not to police the thoughts.

[07:48:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28125s) So any cheating or hampering of the environment

[07:48:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28128s) gets a very negative RL reward.

[07:48:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28132s) And all of this is detected just based on the actions

[07:48:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28135s) that the model is taking, not the thoughts.

[07:48:58](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28138s) So a good example of how we have handled cheating

[07:49:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28141s) is, you know, Carbon, for instance,

[07:49:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28143s) has 0% cheating rate on Terminal Bench 4.

[07:49:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28146s) Our previous models had a much higher cheating rate.

[07:49:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28148s) and it's about 5x less adversarial than argon.

[07:49:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28151s) So every generation of the model,

[07:49:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28153s) we're trying to make it much more aligned

[07:49:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28155s) so it's not trying to cheat.

[07:49:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28158s) Now finally, I think I want to talk about model behavior

[07:49:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28162s) a little bit and modeling honesty specifically.

[07:49:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28167s) Now, agents, they always want to be helpful

[07:49:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28170s) and sometimes they fake helpfulness.

[07:49:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28172s) You might have seen this in your usage of the agents.

[07:49:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28175s) So sometimes they'll claim that tests have passed

[07:49:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28177s) when they've obviously not passed.

[07:49:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28180s) They'll declare done,

[07:49:42](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28182s) but obviously they're not done.

[07:49:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28184s) They'll make edits that they'll claim to have made edits,

[07:49:47](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28187s) but they've not made those edits.

[07:49:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28188s) Or they'll invent facts and APIs and whatnot.

[07:49:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28192s) Or they will give you a wall of text

[07:49:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28194s) that has so many numbers

[07:49:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28195s) that is going to be impossible for you to verify.

[07:49:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28197s) We've gone through all of these examples.

[07:50:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28202s) Now, here is an example of fabrication rates in Gemini.

[07:50:06](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28206s) So we, on an internal eval, started with 8% fabrication in that hard eval.

[07:50:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28213s) We cut that down dramatically to about 3% for hydrogen, and carbon is half of that,

[07:50:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28220s) is about 1.5%, but we're striving to get this down to zero.

[07:50:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28223s) Like you don't want your agents fabricating anything, right?

[07:50:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28226s) But they try to fabricate because they want to be helpful.

[07:50:30](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28230s) And you really have to try hard to align them to be helpful, but not by faking it.

[07:50:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28237s) How do we reduce the fabrication?

[07:50:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28240s) Believe it or not, you have to clean your data, right?

[07:50:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28243s) So we filter out lots of training trajectories where there is, you know,

[07:50:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28248s) not just intended fabrication, but unintended fabrication.

[07:50:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28252s) There are lots of claims that are not fully supported.

[07:50:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28256s) They're not verified.

[07:50:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28257s) We ground in the full trajectory.

[07:50:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28259s) So if the agent says, I've completed the task

[07:51:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28261s) and it has a final response,

[07:51:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28263s) we actually look at that response

[07:51:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28265s) and see all of the actions that the agent had taken

[07:51:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28268s) in the trajectory to get to that final solution,

[07:51:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28270s) and we look at the steps that were taken.

[07:51:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28272s) So you really need to, when the agent says, I'm done,

[07:51:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28275s) and gives you a summary, that needs to be grounded

[07:51:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28277s) in the actions that were taken.

[07:51:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28279s) And we want to reward honesty.

[07:51:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28281s) So, if there's a hard problem, it's better for the agent to say, hey, this is my partial progress.

[07:51:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28287s) It's a hard problem. I'm done here. This is my partial solution.

[07:51:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28292s) Rather than faking the last step and saying, hey, I've solved the problem. This is done.

[07:51:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28299s) Finally, shipping in weeks. So, how are we able to ship so rapidly?

[07:51:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28305s) And the biggest reason for that is our internal team of 100,000 plus engineers at Google using

[07:51:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28312s) the model for realistic tasks.

[07:51:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28314s) You know, many of you might know Google is a mono repo.

[07:51:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28317s) And what is a mono repo?

[07:51:59](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28319s) The entire company is one code base, right?

[07:52:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28323s) And I don't know how to measure this, but I think it might be one of the largest mono

[07:52:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28327s) repos in the world.

[07:52:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28328s) Now, that's a really complex code base.

[07:52:11](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28331s) and we have the best people in the world

[07:52:15](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28335s) using a really complex code base

[07:52:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28338s) to do really complex tasks.

[07:52:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28340s) So every time we have a new candidate from post-training,

[07:52:23](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28343s) we actually serve that model for our users.

[07:52:27](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28347s) We run blind battles with them

[07:52:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28349s) through multiple different candidates.

[07:52:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28351s) We let users pick the best model that they want.

[07:52:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28355s) We also run Elyse,

[07:52:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28357s) where under the hood we're routing people

[07:52:40](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28360s) to experimental models and get signal and feedback from that and we take all

[07:52:46](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28366s) of these signals and friction back into training the next candidate. So this is a

[07:52:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28371s) cycle that helps us you know rapidly iterate really train on hard problems

[07:52:56](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28376s) and as the models get better the engineers are going to push the model in

[07:53:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28380s) interesting ways. This is also why you know we've gone from releasing in weeks

[07:53:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28388s) to releasing in quarters to releasing in weeks.

[07:53:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28392s) So in the Gemini 3 era,

[07:53:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28394s) we've had launches every two to three months.

[07:53:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28396s) But now we've had six releases in the last 16 weeks.

[07:53:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28401s) And we're very excited to have you all try carbon and hydrogen

[07:53:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28406s) and Gemini 4.

[07:53:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28409s) And we're excited for us to see what you all do with it.

[07:53:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28414s) And we're not done.

[07:53:35](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28415s) So we have a lot more elements to go through.

[07:53:38](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=28418s) With that, you know, thank you.