1 07:35:58,220 --> 07:36:02,940 So today I thought I'll take you through a little journey of how we post train Gemini 4. 2 07:36:03,360 --> 07:36:09,000 And our journey begins by people calling us not a frontier model. 3 07:36:09,680 --> 07:36:15,320 So here's an example. You know, people say Gemini is cooked, but GCP is cooking. 4 07:36:15,820 --> 07:36:20,400 For all intents and purposes, we believe DeepMind is no longer a frontier lab. 5 07:36:20,400 --> 07:36:29,600 Ethan Malik, who we respect a lot, he says in his own nice way, pretty long gap between launches is what he calls us. 6 07:36:30,860 --> 07:36:34,520 And I'm here to tell you we're not dead. 7 07:36:34,980 --> 07:36:35,860 We're here. 8 07:36:37,320 --> 07:36:41,880 And we have launched Gemini 4 Argon. 9 07:36:42,480 --> 07:36:47,560 We announced Carbon and we launched Carbon all within the last 28 days. 10 07:36:47,560 --> 07:36:53,400 So we're on a run here, and that's not it. 11 07:36:54,500 --> 07:36:58,880 In the process, we actually garnered an apology from Dylan. 12 07:36:59,380 --> 07:37:02,260 He says official apology to Google. 13 07:37:03,320 --> 07:37:05,400 They actually cooked. They're not cooked. 14 07:37:06,640 --> 07:37:10,640 Ethan says one of the most significant aspects of Gemini 4 15 07:37:10,640 --> 07:37:13,100 is the speed at which we're launching things. 16 07:37:13,100 --> 07:37:17,720 So we've gone 236 days without having a frontier model, 17 07:37:17,920 --> 07:37:20,840 and we've had two frontier models in 28 days. 18 07:37:21,040 --> 07:37:23,960 That's a 10x improvement in our speed of shipping. 19 07:37:24,740 --> 07:37:28,640 And, you know, people call us Google kind of cooked, 20 07:37:29,020 --> 07:37:33,220 but Google's not cooked, so it's good to see that sentiment. 21 07:37:33,460 --> 07:37:34,760 It's glad to be on the other end. 22 07:37:35,000 --> 07:37:38,840 It's also good to be on the receiving end of the bad comments, 23 07:37:38,860 --> 07:37:40,280 because it pushes you to do better. 24 07:37:41,820 --> 07:37:46,180 I want to start by playing a little video of a bunch of things that 25 07:37:46,660 --> 07:37:50,460 Karvin has built. If we can cue the video, please. 26 07:37:52,260 --> 07:37:54,620 There we go. Go back. There we go. 27 07:38:01,400 --> 07:38:02,500 Did that work? No. 28 07:38:08,280 --> 07:38:11,580 Let me see if going to the laptop is going to fix it. 29 07:38:50,760 --> 07:38:54,360 Kudos to the model for building that video itself. 30 07:38:54,540 --> 07:39:00,840 So you saw great examples of the model doing much better at 3D, model building assets using 31 07:39:00,840 --> 07:39:06,320 blender there's lots of games in there the real world use cases model processing multi-modal 32 07:39:06,320 --> 07:39:14,860 documents but you know it's not just the demos carbon's actually number one on a bunch of real 33 07:39:14,860 --> 07:39:19,940 world customer use cases and one of the things that we're really excited about is you know it's 34 07:39:19,940 --> 07:39:27,680 not just really good on these benchmarks it's actually much cheaper so on cursor bench it's 35 07:39:27,680 --> 07:39:32,040 It's the number one model today, and it's 3.3 times cheaper than the second best model. 36 07:39:32,520 --> 07:39:39,460 On Perplexity's Wander, which is a benchmark for deep research queries, it's the best model. 37 07:39:39,620 --> 07:39:42,660 You saw competitor models come out. 38 07:39:43,420 --> 07:39:45,940 9x cheaper than the second best model there. 39 07:39:46,200 --> 07:39:52,340 And on AutomationBench, which is Zapier's benchmark, again, we're number one, while 40 07:39:52,340 --> 07:39:55,140 being two times cheaper per task. 41 07:39:56,640 --> 07:39:58,060 and that's not it. 42 07:39:58,440 --> 07:40:00,000 I mean, today we announced 43 07:40:00,120 --> 07:40:12,220 a new element to our Gemini 4 series, and it's not a surprise. It's hydrogen. So, you know, you might have seen the launch earlier today. 44 07:40:12,220 --> 07:40:15,980 Gemini 4 Hydrogen is where we're trying to bring 45 07:40:15,980 --> 07:40:18,220 the Gemini 4 intelligence to everyone. 46 07:40:18,480 --> 07:40:20,680 It's fast, it's dependable, 47 07:40:20,980 --> 07:40:23,140 it's launched on all surfaces, 48 07:40:23,140 --> 07:40:25,240 on anti-gravity, on the Gemini API, 49 07:40:25,580 --> 07:40:27,800 in the Gemini app, and on search. 50 07:40:28,880 --> 07:40:32,820 And, you know, I have a graph that shows the A index 51 07:40:32,820 --> 07:40:35,180 which many of you might be familiar with, 52 07:40:35,420 --> 07:40:37,500 and time on the x-axis. 53 07:40:37,720 --> 07:40:40,240 You can see it's one of the fastest models 54 07:40:40,240 --> 07:40:42,100 for that level of intelligence. 55 07:40:42,220 --> 07:40:50,760 and to give you a sense of how far you know gemini has come 3.8 flash on terminal bench 4 56 07:40:50,760 --> 07:40:58,740 was about 20 percent and hydrogen is about 60 percent so we're three times better performance 57 07:40:58,740 --> 07:41:06,140 while being three times faster right so that's no easy task and we're you know really happy to 58 07:41:06,140 --> 07:41:09,660 bring that to you in your hands so you can build things with it. 59 07:41:11,060 --> 07:41:15,600 And that completes, you know, our Gemini 4 series. 60 07:41:15,840 --> 07:41:22,400 So we have Gemini 4 Carbon, a frontier model for deep frontier intelligence, Gemini 4 Hydrogen, 61 07:41:22,400 --> 07:41:27,300 which is fast for everyday execution, and we'll have more elements. 62 07:41:27,480 --> 07:41:31,080 You know, of course, there are many elements left in the periodic table, and we have a 63 07:41:31,080 --> 07:41:32,680 lot of work to do to exhaust them. 64 07:41:33,260 --> 07:41:37,020 but we're glad to put it out there for you all to test it 65 07:41:37,020 --> 07:41:38,380 and for you all to play with it. 66 07:41:40,760 --> 07:41:43,240 Now, I think enough marketing with the models. 67 07:41:43,720 --> 07:41:45,920 I was told this is a deeply technical audience, 68 07:41:45,920 --> 07:41:48,040 so I want to talk a little bit about 69 07:41:48,040 --> 07:41:51,680 why post-training these frontier models is hard 70 07:41:51,680 --> 07:41:52,740 and what goes into that. 71 07:41:53,040 --> 07:41:54,600 Let me start by saying, you know, 72 07:41:54,660 --> 07:41:58,700 AI writes about 75% of Google's new code today, 73 07:41:58,700 --> 07:42:03,880 And this is up from less than 25% a few years ago. 74 07:42:04,560 --> 07:42:12,540 And there are 100,000 people using anti-gravity writing code inside Google every single day. 75 07:42:14,340 --> 07:42:20,380 Now, the complexity of the tasks that people do have also increased exponentially in the last couple of years. 76 07:42:20,560 --> 07:42:25,880 We started with, you know, models just completing chats, which took seconds, 77 07:42:25,880 --> 07:42:28,740 to writing small functions with the minutes. 78 07:42:29,060 --> 07:42:31,740 You can submit a PR and resolve an issue. 79 07:42:31,880 --> 07:42:33,340 It takes about an hour, let's say. 80 07:42:33,600 --> 07:42:38,240 But now people are doing days of real engineering tasks with the model. 81 07:42:38,480 --> 07:42:42,480 People have swarms of agents solving real open problems with the model. 82 07:42:42,700 --> 07:42:46,640 We have engineers across Google using anti-gravity 83 07:42:46,640 --> 07:42:49,900 and our Gemini 4 models to solve tasks over multiple days. 84 07:42:51,580 --> 07:42:56,100 Now how do we go about post-training Gemini for real engineering tasks? 85 07:42:56,580 --> 07:42:58,600 We have to start with hard tasks 86 07:42:58,600 --> 07:43:02,020 So we really need to source hard problems for the model 87 07:43:02,640 --> 07:43:06,220 Grading of these models is actually a very hard problem 88 07:43:06,220 --> 07:43:10,280 So you want to grade like an experienced code reviewer would grade 89 07:43:10,280 --> 07:43:13,440 You want to grade like an experienced engineer would grade a solution 90 07:43:14,320 --> 07:43:16,520 Large-scale RL is how we train the model 91 07:43:16,520 --> 07:43:18,540 and we want to dog food our models 92 07:43:18,540 --> 07:43:20,600 with our 100,000 engineer community 93 07:43:20,600 --> 07:43:22,040 and repeat the loop. 94 07:43:22,200 --> 07:43:23,840 So any failures in the dog food 95 07:43:23,840 --> 07:43:26,760 become tasks for the next iteration of the loop. 96 07:43:28,480 --> 07:43:31,160 So I want to start by going into a little bit more detail 97 07:43:31,160 --> 07:43:32,200 of each of these sections. 98 07:43:32,260 --> 07:43:33,740 So let's start with the hard tasks. 99 07:43:35,260 --> 07:43:36,700 So you might have heard, you know, 100 07:43:36,780 --> 07:43:37,880 as models are getting better, 101 07:43:38,260 --> 07:43:40,160 lots of benchmarks are getting saturated. 102 07:43:40,560 --> 07:43:43,020 And it's not just the benchmarks that get saturated. 103 07:43:43,280 --> 07:43:45,180 Our training data gets saturated too. 104 07:43:45,180 --> 07:43:47,760 So here is a graph that shows 105 07:43:47,760 --> 07:43:50,640 for three different iterations of the Gemini 4 models 106 07:43:51,120 --> 07:43:54,720 how much the share of learnable prompts has gone down. 107 07:43:54,860 --> 07:43:57,000 So let's say we had X percent of learnable prompts 108 07:43:57,000 --> 07:43:57,900 for the first iteration. 109 07:43:58,240 --> 07:44:00,320 By the time we finish and train the next one, 110 07:44:00,500 --> 07:44:02,540 the prompts are too easy for the second one. 111 07:44:02,660 --> 07:44:04,260 We finish that and train the third one, 112 07:44:04,360 --> 07:44:05,940 the prompts are too easy for the third one. 113 07:44:06,460 --> 07:44:09,460 So what's the problem with training on easy tasks? 114 07:44:09,560 --> 07:44:11,600 Easy tasks give you zero learning signal, right? 115 07:44:11,680 --> 07:44:12,460 So you have a prompt. 116 07:44:12,580 --> 07:44:15,020 When all of your rollouts are going to be correct, 117 07:44:15,180 --> 07:44:17,080 There's no variance. There's no RL signal. 118 07:44:17,300 --> 07:44:20,660 So you really need to focus on getting hard tasks. 119 07:44:20,820 --> 07:44:23,560 And you really need to be faster at getting hard tasks 120 07:44:23,560 --> 07:44:25,100 than the models are getting better. 121 07:44:27,300 --> 07:44:28,760 So how do we go about it? 122 07:44:28,960 --> 07:44:32,720 We first have to identify these easy tasks and prune them. 123 07:44:32,780 --> 07:44:36,800 So we cut these simple Sweeband-style tasks from the mixture 124 07:44:36,800 --> 07:44:38,760 that are too easy for the models. 125 07:44:39,000 --> 07:44:42,880 We expand into much harder Swee tasks, for example, 126 07:44:43,040 --> 07:44:44,380 that are multi-file repos. 127 07:44:45,180 --> 07:44:51,160 We go from unit test or binary correctness to rubric-based grading. 128 07:44:51,340 --> 07:44:52,960 And rubrics give you dense signal. 129 07:44:53,140 --> 07:44:54,520 They're not just 0, 1. 130 07:44:54,740 --> 07:45:00,500 They can be more gradients within the criteria. 131 07:45:02,380 --> 07:45:05,080 Here are some examples of hard tasks that we train on. 132 07:45:05,180 --> 07:45:10,320 So we look at multi-file repos and we extract commits from them. 133 07:45:10,580 --> 07:45:14,400 From those commits, we have models automatically extract rubrics. 134 07:45:15,180 --> 07:45:18,900 We don't actually necessitate unit tests being involved. 135 07:45:19,100 --> 07:45:22,360 We can actually generate rubrics on the fly 136 07:45:22,360 --> 07:45:23,420 based on these repos. 137 07:45:23,760 --> 07:45:26,000 Another example, a great example of tasks, 138 07:45:27,120 --> 07:45:31,880 you try to teach the model to build CLI tools from scratch. 139 07:45:32,040 --> 07:45:34,760 So you start with an empty folder and a specification. 140 07:45:35,120 --> 07:45:38,160 The specification says, hey, you have to build this tool. 141 07:45:38,160 --> 07:45:39,140 It's called grep. 142 07:45:39,340 --> 07:45:41,600 These are the parameters of grep. 143 07:45:41,680 --> 07:45:42,360 These are the flags. 144 07:45:42,480 --> 07:45:43,280 This is the functionality. 145 07:45:43,280 --> 07:45:46,720 and you let the model build the entire tool from scratch. 146 07:45:47,140 --> 07:45:50,040 And you know what the functionality for grep is going to be, 147 07:45:50,060 --> 07:45:53,640 so you have a hidden test suite that you can evaluate the binary on. 148 07:45:54,380 --> 07:45:58,820 The last example is production codebase TIFFs. 149 07:45:58,920 --> 07:46:03,940 So you look at real usage of the model inside your production codebase 150 07:46:03,940 --> 07:46:06,080 and extract useful tasks out of them. 151 07:46:08,220 --> 07:46:11,360 Now let me talk a little bit about the graders. 152 07:46:13,420 --> 07:46:16,520 So agents, you know, they find any way, 153 07:46:16,740 --> 07:46:18,540 you're training them to pass the test, 154 07:46:18,620 --> 07:46:20,740 so they will find any way to pass the test. 155 07:46:21,000 --> 07:46:23,700 But if the test pass, that doesn't necessarily mean 156 07:46:23,700 --> 07:46:24,840 that they've solved the problem. 157 07:46:25,160 --> 07:46:28,100 There are examples where the agent will bypass the checker. 158 07:46:28,700 --> 07:46:31,940 There are examples where the agent will game the benchmark. 159 07:46:32,600 --> 07:46:35,400 There are examples where the agent will fake the evidence. 160 07:46:35,400 --> 07:46:39,560 You know, it writes the fake evidence into a temp file. 161 07:46:39,560 --> 07:46:42,420 It will delete any evidence of it faking it, right? 162 07:46:43,280 --> 07:46:46,020 So you have to be smarter about how you judge these models, 163 07:46:46,060 --> 07:46:47,780 especially as they become more capable. 164 07:46:49,780 --> 07:46:53,060 Now, this is what I meant by grading like a reviewer. 165 07:46:53,180 --> 07:46:56,380 So if you've done a code review, you have a very strict reviewer, 166 07:46:56,560 --> 07:47:00,180 you're going to know they're not just going to think about the correctness of the solution. 167 07:47:00,480 --> 07:47:02,460 They're actually going to think about the style. 168 07:47:02,660 --> 07:47:04,700 They're going to think about maintainability. 169 07:47:05,060 --> 07:47:06,620 They're going to think of many different aspects. 170 07:47:07,500 --> 07:47:13,200 So what we do is we move from unit test-based correctness to rubric-based. 171 07:47:13,280 --> 07:47:19,960 When these are generated from real commits, the rubrics also provide dense learning signal, not just 0-1 signal. 172 07:47:20,280 --> 07:47:25,200 And we penalize any sort of solution that is looking for a workaround. 173 07:47:26,100 --> 07:47:33,760 So here is an example of a rubric where the criteria is, here are the edge cases that need to pass. 174 07:47:34,140 --> 07:47:37,060 This is an example of hygiene that needs to be maintained. 175 07:47:37,380 --> 07:47:40,540 These are constraints around verification that need to be satisfied. 176 07:47:43,500 --> 07:47:47,600 Now, it's not just about the rubrics needed for grading. 177 07:47:47,960 --> 07:47:51,900 We also need to be careful about the graders not being gamed. 178 07:47:52,220 --> 07:47:55,680 And you really need to harden the environment that the grader is in. 179 07:47:55,860 --> 07:47:59,800 So we think about putting the graders in an isolated sandbox. 180 07:48:00,020 --> 07:48:03,960 So we remove git history and git remote so the answers don't leak. 181 07:48:04,200 --> 07:48:05,880 The test sets are held out. 182 07:48:05,880 --> 07:48:09,400 So only when the agent is done and submits the solution 183 07:48:09,400 --> 07:48:11,220 do we actually reveal the test. 184 07:48:11,220 --> 07:48:14,120 we've had instances where the agent will fake it and say, 185 07:48:14,240 --> 07:48:16,540 hey, I'm done, and the tests come into play, 186 07:48:16,600 --> 07:48:18,640 and then it'll look at the tests and resubmit the solution. 187 07:48:18,820 --> 07:48:19,700 So you don't want that. 188 07:48:20,620 --> 07:48:23,240 We also want to be much more detailed with our scoring. 189 07:48:23,420 --> 07:48:25,880 So we take the full diff, and we look at the reference, 190 07:48:26,020 --> 07:48:27,080 and we look at multiple aspects. 191 07:48:27,220 --> 07:48:28,280 It's not just the correctness. 192 07:48:28,300 --> 07:48:29,960 We also want to look at the code quality. 193 07:48:30,280 --> 07:48:33,840 We also want to look at the scope of the change that was done. 194 07:48:36,800 --> 07:48:41,060 And then finally, we want to penalize any sort of cheating attempt 195 07:48:41,060 --> 07:48:44,840 by the agent while being careful not to police the thoughts. 196 07:48:45,580 --> 07:48:48,880 So any cheating or hampering of the environment 197 07:48:48,880 --> 07:48:51,340 gets a very negative RL reward. 198 07:48:52,100 --> 07:48:55,320 And all of this is detected just based on the actions 199 07:48:55,320 --> 07:48:57,280 that the model is taking, not the thoughts. 200 07:48:58,120 --> 07:49:01,460 So a good example of how we have handled cheating 201 07:49:01,460 --> 07:49:03,200 is, you know, Carbon, for instance, 202 07:49:03,460 --> 07:49:06,120 has 0% cheating rate on Terminal Bench 4. 203 07:49:06,320 --> 07:49:08,460 Our previous models had a much higher cheating rate. 204 07:49:08,460 --> 07:49:11,400 and it's about 5x less adversarial than argon. 205 07:49:11,580 --> 07:49:13,040 So every generation of the model, 206 07:49:13,100 --> 07:49:15,020 we're trying to make it much more aligned 207 07:49:15,020 --> 07:49:16,320 so it's not trying to cheat. 208 07:49:18,540 --> 07:49:22,220 Now finally, I think I want to talk about model behavior 209 07:49:22,220 --> 07:49:25,860 a little bit and modeling honesty specifically. 210 07:49:27,640 --> 07:49:30,240 Now, agents, they always want to be helpful 211 07:49:30,240 --> 07:49:32,280 and sometimes they fake helpfulness. 212 07:49:32,480 --> 07:49:35,300 You might have seen this in your usage of the agents. 213 07:49:35,480 --> 07:49:37,820 So sometimes they'll claim that tests have passed 214 07:49:37,820 --> 07:49:39,960 when they've obviously not passed. 215 07:49:40,420 --> 07:49:41,620 They'll declare done, 216 07:49:42,360 --> 07:49:43,620 but obviously they're not done. 217 07:49:44,220 --> 07:49:47,020 They'll make edits that they'll claim to have made edits, 218 07:49:47,100 --> 07:49:48,340 but they've not made those edits. 219 07:49:48,580 --> 07:49:52,220 Or they'll invent facts and APIs and whatnot. 220 07:49:52,580 --> 07:49:54,480 Or they will give you a wall of text 221 07:49:54,480 --> 07:49:55,460 that has so many numbers 222 07:49:55,460 --> 07:49:57,340 that is going to be impossible for you to verify. 223 07:49:57,980 --> 07:50:00,000 We've gone through all of these examples. 224 07:50:02,080 --> 07:50:05,900 Now, here is an example of fabrication rates in Gemini. 225 07:50:06,160 --> 07:50:12,880 So we, on an internal eval, started with 8% fabrication in that hard eval. 226 07:50:13,580 --> 07:50:20,180 We cut that down dramatically to about 3% for hydrogen, and carbon is half of that, 227 07:50:20,340 --> 07:50:23,400 is about 1.5%, but we're striving to get this down to zero. 228 07:50:23,400 --> 07:50:26,460 Like you don't want your agents fabricating anything, right? 229 07:50:26,800 --> 07:50:30,740 But they try to fabricate because they want to be helpful. 230 07:50:30,920 --> 07:50:35,980 And you really have to try hard to align them to be helpful, but not by faking it. 231 07:50:37,500 --> 07:50:39,280 How do we reduce the fabrication? 232 07:50:40,120 --> 07:50:43,220 Believe it or not, you have to clean your data, right? 233 07:50:43,600 --> 07:50:48,300 So we filter out lots of training trajectories where there is, you know, 234 07:50:48,360 --> 07:50:52,580 not just intended fabrication, but unintended fabrication. 235 07:50:52,580 --> 07:50:55,860 There are lots of claims that are not fully supported. 236 07:50:56,060 --> 07:50:56,920 They're not verified. 237 07:50:57,580 --> 07:50:59,380 We ground in the full trajectory. 238 07:50:59,760 --> 07:51:01,720 So if the agent says, I've completed the task 239 07:51:01,720 --> 07:51:03,140 and it has a final response, 240 07:51:03,560 --> 07:51:05,700 we actually look at that response 241 07:51:05,700 --> 07:51:08,080 and see all of the actions that the agent had taken 242 07:51:08,080 --> 07:51:10,280 in the trajectory to get to that final solution, 243 07:51:10,280 --> 07:51:12,660 and we look at the steps that were taken. 244 07:51:12,880 --> 07:51:15,560 So you really need to, when the agent says, I'm done, 245 07:51:15,720 --> 07:51:17,900 and gives you a summary, that needs to be grounded 246 07:51:17,900 --> 07:51:19,120 in the actions that were taken. 247 07:51:19,740 --> 07:51:21,400 And we want to reward honesty. 248 07:51:21,400 --> 07:51:27,660 So, if there's a hard problem, it's better for the agent to say, hey, this is my partial progress. 249 07:51:27,780 --> 07:51:31,660 It's a hard problem. I'm done here. This is my partial solution. 250 07:51:32,100 --> 07:51:36,020 Rather than faking the last step and saying, hey, I've solved the problem. This is done. 251 07:51:39,880 --> 07:51:43,980 Finally, shipping in weeks. So, how are we able to ship so rapidly? 252 07:51:45,100 --> 07:51:52,240 And the biggest reason for that is our internal team of 100,000 plus engineers at Google using 253 07:51:52,240 --> 07:51:53,780 the model for realistic tasks. 254 07:51:54,360 --> 07:51:57,380 You know, many of you might know Google is a mono repo. 255 07:51:57,740 --> 07:51:59,040 And what is a mono repo? 256 07:51:59,320 --> 07:52:02,840 The entire company is one code base, right? 257 07:52:03,320 --> 07:52:07,320 And I don't know how to measure this, but I think it might be one of the largest mono 258 07:52:07,320 --> 07:52:08,140 repos in the world. 259 07:52:08,600 --> 07:52:11,840 Now, that's a really complex code base. 260 07:52:11,840 --> 07:52:15,480 and we have the best people in the world 261 07:52:15,480 --> 07:52:18,040 using a really complex code base 262 07:52:18,040 --> 07:52:19,660 to do really complex tasks. 263 07:52:20,000 --> 07:52:22,740 So every time we have a new candidate from post-training, 264 07:52:23,040 --> 07:52:26,800 we actually serve that model for our users. 265 07:52:27,100 --> 07:52:29,540 We run blind battles with them 266 07:52:29,540 --> 07:52:31,100 through multiple different candidates. 267 07:52:31,100 --> 07:52:35,220 We let users pick the best model that they want. 268 07:52:35,520 --> 07:52:37,100 We also run Elyse, 269 07:52:37,220 --> 07:52:40,020 where under the hood we're routing people 270 07:52:40,020 --> 07:52:46,040 to experimental models and get signal and feedback from that and we take all 271 07:52:46,040 --> 07:52:51,740 of these signals and friction back into training the next candidate. So this is a 272 07:52:51,740 --> 07:52:56,600 cycle that helps us you know rapidly iterate really train on hard problems 273 07:52:56,600 --> 07:53:00,460 and as the models get better the engineers are going to push the model in 274 07:53:00,460 --> 07:53:07,280 interesting ways. This is also why you know we've gone from releasing in weeks 275 07:53:08,100 --> 07:53:11,860 to releasing in quarters to releasing in weeks. 276 07:53:12,040 --> 07:53:14,280 So in the Gemini 3 era, 277 07:53:14,520 --> 07:53:16,620 we've had launches every two to three months. 278 07:53:16,900 --> 07:53:20,700 But now we've had six releases in the last 16 weeks. 279 07:53:21,940 --> 07:53:26,900 And we're very excited to have you all try carbon and hydrogen 280 07:53:26,900 --> 07:53:28,640 and Gemini 4. 281 07:53:29,100 --> 07:53:33,800 And we're excited for us to see what you all do with it. 282 07:53:34,000 --> 07:53:35,060 And we're not done. 283 07:53:35,060 --> 07:53:37,680 So we have a lot more elements to go through. 284 07:53:38,120 --> 07:53:39,740 With that, you know, thank you.