1 02:21:16,800 --> 02:21:18,950 and harness and the less code we need. 2 02:21:18,960 --> 02:21:21,110 And if we look at um what we have been 3 02:21:21,120 --> 02:21:22,870 doing at Google deepmind for the last 6 4 02:21:22,880 --> 02:21:25,910 months. So at May we had Google IO IO 5 02:21:25,920 --> 02:21:29,030 and today it's uh November 11 and we 6 02:21:29,040 --> 02:21:30,710 have been shipping relentlessly. 7 02:21:30,720 --> 02:21:32,950 Basically just earlier today we 8 02:21:32,960 --> 02:21:35,110 announced our first decision model with 9 02:21:35,120 --> 02:21:38,230 Gemini decision one. Yesterday we made 10 02:21:38,240 --> 02:21:40,389 Gemini 4 carbon available and then if we 11 02:21:40,399 --> 02:21:41,830 just look at the last six months we 12 02:21:41,840 --> 02:21:43,510 basically ship now models every few 13 02:21:43,520 --> 02:21:46,389 weeks. Um as already said Gemini Carbon 14 02:21:46,399 --> 02:21:48,550 is now available for you. So if you are 15 02:21:48,560 --> 02:21:50,389 a Google AI ultra subscriber, you can 16 02:21:50,399 --> 02:21:51,910 use it in anti-gravity, you can use it 17 02:21:51,920 --> 02:21:53,830 in the Gemini app, you can use it in AI 18 02:21:53,840 --> 02:21:56,469 studio. Uh carbon is number one on the 19 02:21:56,479 --> 02:21:59,110 cursor bench uh eval and also number one 20 02:21:59,120 --> 02:22:01,110 on the perlexity benchmark and 21 02:22:01,120 --> 02:22:03,190 internally at Google. We have been using 22 02:22:03,200 --> 02:22:06,150 it already a lot and much more exciting 23 02:22:06,160 --> 02:22:08,150 is that I'm happy to share that we 24 02:22:08,160 --> 02:22:10,550 launched the Gemini decisions API 25 02:22:10,560 --> 02:22:13,190 roughly one hour ago with our new Gemini 26 02:22:13,200 --> 02:22:15,590 decision one model. So it's a similar 27 02:22:15,600 --> 02:22:18,070 model to Chef. You can provide all of 28 02:22:18,080 --> 02:22:20,230 the existing Gemini inputs, meaning it's 29 02:22:20,240 --> 02:22:22,630 multimodal. So you can provide text, 30 02:22:22,640 --> 02:22:25,349 audio, images, video, and also PDFs 31 02:22:25,359 --> 02:22:27,750 natively, and then provide up to 1,000 32 02:22:27,760 --> 02:22:29,910 questions in a single request. The 33 02:22:29,920 --> 02:22:31,910 average end to-end latency, including 34 02:22:31,920 --> 02:22:33,910 all of the network overhead, is between 35 02:22:33,920 --> 02:22:37,270 150 to 250 milliseconds, and it costs 10 36 02:22:37,280 --> 02:22:40,630 cents per 1 million token. So how do 37 02:22:40,640 --> 02:22:43,510 models get good at work? And that's a 38 02:22:43,520 --> 02:22:46,469 rough overview on how post- training 39 02:22:46,479 --> 02:22:48,790 works. So we have a task. We have our 40 02:22:48,800 --> 02:22:50,630 model. The model has a certain set of 41 02:22:50,640 --> 02:22:52,150 tools and then we have some kind of 42 02:22:52,160 --> 02:22:54,230 compute environment. And at the end we 43 02:22:54,240 --> 02:22:55,830 have some verifier. And the model tries 44 02:22:55,840 --> 02:22:57,670 to basically solve that task to use 45 02:22:57,680 --> 02:22:59,590 different tools. And then based on the 46 02:22:59,600 --> 02:23:01,270 score from the verifier, we update the 47 02:23:01,280 --> 02:23:03,750 model. But no one is telling the model 48 02:23:03,760 --> 02:23:06,469 which tools to use. And basically it 49 02:23:06,479 --> 02:23:10,070 only receives that reward feedback. So 50 02:23:10,080 --> 02:23:12,150 if we look at two examples on how a 51 02:23:12,160 --> 02:23:14,230 model can do a certain tasks with tools, 52 02:23:14,240 --> 02:23:16,309 we have the path A where we have 53 02:23:16,319 --> 02:23:19,110 dedicated tools for file system control. 54 02:23:19,120 --> 02:23:21,510 So you all have probably used uh coding 55 02:23:21,520 --> 02:23:23,750 agents and you have seen a read uh file 56 02:23:23,760 --> 02:23:26,230 call or an edit file call. So that's one 57 02:23:26,240 --> 02:23:28,230 way of doing it and the other way of 58 02:23:28,240 --> 02:23:30,150 doing it would be just using a single 59 02:23:30,160 --> 02:23:31,750 inline bash script for all of the 60 02:23:31,760 --> 02:23:34,230 changes. So on the left side we roughly 61 02:23:34,240 --> 02:23:36,150 need around 10 turns to make the same 62 02:23:36,160 --> 02:23:39,670 changes to a coding um repository 63 02:23:39,680 --> 02:23:41,750 compared to the two ch uh turns on the 64 02:23:41,760 --> 02:23:43,910 right side just using bash and it also 65 02:23:43,920 --> 02:23:45,990 reflects in the amount of tokens we 66 02:23:46,000 --> 02:23:49,349 need. So the left path takes around 10x 67 02:23:49,359 --> 02:23:51,349 more tokens than the life uh the right 68 02:23:51,359 --> 02:23:54,150 path and it's doing the exact same thing 69 02:23:54,160 --> 02:23:56,469 uh at the end for the model to achieve 70 02:23:56,479 --> 02:23:59,510 that task. So in my experiment every 71 02:23:59,520 --> 02:24:02,230 roughly 6 months I try to build a new 72 02:24:02,240 --> 02:24:04,790 coding agent harness from scratch and I 73 02:24:04,800 --> 02:24:06,710 always ask myself okay how much of the 74 02:24:06,720 --> 02:24:08,790 harness do I really need to achieve the 75 02:24:08,800 --> 02:24:11,429 same tasks and what is overhead to the 76 02:24:11,439 --> 02:24:13,750 model and we no longer need. So if we 77 02:24:13,760 --> 02:24:16,710 look at the coding agent harness which 78 02:24:16,720 --> 02:24:20,230 um from 2025 we had a roughly around 10 79 02:24:20,240 --> 02:24:22,790 to 15 to 20 tools. We had dedicated 80 02:24:22,800 --> 02:24:25,990 tools for um file editing. We had 81 02:24:26,000 --> 02:24:28,790 dedicated tools for planning, for to-do 82 02:24:28,800 --> 02:24:31,030 lists and also we had a lot of dedicated 83 02:24:31,040 --> 02:24:33,110 rules in our system prompt which told 84 02:24:33,120 --> 02:24:36,150 the model what to do and what not to do 85 02:24:36,160 --> 02:24:38,309 and that worked of course but since then 86 02:24:38,319 --> 02:24:39,990 we made a lot of changes. So in my 87 02:24:40,000 --> 02:24:42,710 experiments uh first one was done 88 02:24:42,720 --> 02:24:46,150 roughly now two years ago uh in February 89 02:24:46,160 --> 02:24:47,830 where we did not really had agents right 90 02:24:47,840 --> 02:24:49,349 we had workflows we basically had 91 02:24:49,359 --> 02:24:51,750 predefined steps rock applications back 92 02:24:51,760 --> 02:24:53,670 then were very popular we had a planner 93 02:24:53,680 --> 02:24:55,429 we had search we reflected on whatever 94 02:24:55,439 --> 02:24:57,110 was found and then we generated an 95 02:24:57,120 --> 02:24:59,270 answer then we started to see our first 96 02:24:59,280 --> 02:25:01,510 agents we had many many different tools 97 02:25:01,520 --> 02:25:04,710 for all sorts of fine cular actions um 98 02:25:04,720 --> 02:25:06,469 and then came the coding agents where we 99 02:25:06,479 --> 02:25:09,270 had those roughly 10 to 20 tools 100 02:25:09,280 --> 02:25:11,670 And now in my last experiment I tried to 101 02:25:11,680 --> 02:25:13,750 see okay what's going to happen if I 102 02:25:13,760 --> 02:25:16,790 provide only one tool to my agent and 103 02:25:16,800 --> 02:25:19,190 the outcome is was pretty exciting to 104 02:25:19,200 --> 02:25:21,910 me. So this one tool agent works the 105 02:25:21,920 --> 02:25:23,670 same way as every other agent except 106 02:25:23,680 --> 02:25:26,389 that it only has one tool which is bash 107 02:25:26,399 --> 02:25:29,030 and the model can use that tool for any 108 02:25:29,040 --> 02:25:31,270 sorts of task and the bash tool is very 109 02:25:31,280 --> 02:25:33,349 simple. So it runs in a se a se separate 110 02:25:33,359 --> 02:25:35,990 clean subshell and it supports the 111 02:25:36,000 --> 02:25:38,150 command argument. I also added a 112 02:25:38,160 --> 02:25:40,309 description so that it's nice and easier 113 02:25:40,319 --> 02:25:41,990 to display to the user what the model is 114 02:25:42,000 --> 02:25:43,590 actually doing and then like some async 115 02:25:43,600 --> 02:25:45,030 parameters. So basically when the model 116 02:25:45,040 --> 02:25:47,910 runs a bash command uh the parameter 117 02:25:47,920 --> 02:25:49,990 tells the model okay if it is uh the 118 02:25:50,000 --> 02:25:51,990 command longer running than this um 119 02:25:52,000 --> 02:25:55,190 milliseconds it's being um that the tool 120 02:25:55,200 --> 02:25:57,910 call resolves and the model receives uh 121 02:25:57,920 --> 02:26:00,150 a local file handle where it can look up 122 02:26:00,160 --> 02:26:01,670 later okay if the process is still 123 02:26:01,680 --> 02:26:03,510 running that's very interesting if you 124 02:26:03,520 --> 02:26:05,750 do like all sorts of like uh web 125 02:26:05,760 --> 02:26:08,070 development for example and the single 126 02:26:08,080 --> 02:26:09,670 batch tool actually works for all of the 127 02:26:09,680 --> 02:26:11,270 things we are using Asian so we can have 128 02:26:11,280 --> 02:26:13,429 longunning work where we can start on 129 02:26:13,439 --> 02:26:15,910 the foreground and then automatically 130 02:26:15,920 --> 02:26:17,590 move it to the background. The model 131 02:26:17,600 --> 02:26:19,990 receives some file handle and since it 132 02:26:20,000 --> 02:26:22,230 has bash available, if we store it to a 133 02:26:22,240 --> 02:26:24,469 file system, it can read the logs which 134 02:26:24,479 --> 02:26:26,950 are being streamed. If the agent itself 135 02:26:26,960 --> 02:26:30,389 is a CLI, it can call itself using bash. 136 02:26:30,399 --> 02:26:33,030 So there's practically no need to create 137 02:26:33,040 --> 02:26:34,790 sub aent tools for it because it can 138 02:26:34,800 --> 02:26:36,790 invoke itself. And then of course we 139 02:26:36,800 --> 02:26:39,190 don't need dedicated planning or memory 140 02:26:39,200 --> 02:26:41,030 tools because we have our file system 141 02:26:41,040 --> 02:26:43,030 available. We can provide some context 142 02:26:43,040 --> 02:26:44,469 to the model. So the model can keep 143 02:26:44,479 --> 02:26:46,469 track of its own work with the file 144 02:26:46,479 --> 02:26:49,110 system and using the bash tools. In 145 02:26:49,120 --> 02:26:51,510 addition of course to our tools we have 146 02:26:51,520 --> 02:26:54,309 system prompts and similar to tools uh 147 02:26:54,319 --> 02:26:57,190 system prompts evolved over time. Um 148 02:26:57,200 --> 02:26:59,270 every dot here is roughly around 100 149 02:26:59,280 --> 02:27:01,190 tokens and you can see that coding 150 02:27:01,200 --> 02:27:03,429 agents mid2025 151 02:27:03,439 --> 02:27:06,950 had a lot of context we needed to send 152 02:27:06,960 --> 02:27:09,270 in every request. Of course, this also 153 02:27:09,280 --> 02:27:11,349 improved and similar story here is like, 154 02:27:11,359 --> 02:27:13,270 okay, what do I really need to provide 155 02:27:13,280 --> 02:27:16,070 to my agent to make it successful and 156 02:27:16,080 --> 02:27:18,550 work and what of all of the system 157 02:27:18,560 --> 02:27:20,389 information and details we had in the 158 02:27:20,399 --> 02:27:22,790 past are now basically part of the 159 02:27:22,800 --> 02:27:26,150 model. So, how much of the harness helps 160 02:27:26,160 --> 02:27:29,030 and how much is in the way and to find 161 02:27:29,040 --> 02:27:31,190 that out I was running a lot of 162 02:27:31,200 --> 02:27:32,870 different evaluations of a lot of 163 02:27:32,880 --> 02:27:34,469 different agent harnesses with a lot of 164 02:27:34,479 --> 02:27:36,469 different tools. And one example I like 165 02:27:36,479 --> 02:27:38,630 to show here which was done with Gemini 166 02:27:38,640 --> 02:27:41,990 4 carbon from an eval task from deep sv 167 02:27:42,000 --> 02:27:44,630 is like okay what happens if I provide 168 02:27:44,640 --> 02:27:47,670 my existing coding agent tools with read 169 02:27:47,680 --> 02:27:51,190 write search crap and my bash only tool 170 02:27:51,200 --> 02:27:53,110 and I think you can pretty clearly see 171 02:27:53,120 --> 02:27:55,830 that the top line is taking a lot of 172 02:27:55,840 --> 02:27:58,230 more tokens a lot of more turns and a 173 02:27:58,240 --> 02:28:00,710 lot of more times to achieve the same 174 02:28:00,720 --> 02:28:03,990 task and what stood out really well is 175 02:28:04,000 --> 02:28:06,550 that Okay, reading is done in parallel 176 02:28:06,560 --> 02:28:08,070 in the beginning, but as soon as we 177 02:28:08,080 --> 02:28:10,550 start going into making code changes, we 178 02:28:10,560 --> 02:28:13,270 see that same behavior that the model is 179 02:28:13,280 --> 02:28:16,469 making turn by turn a single edit, which 180 02:28:16,479 --> 02:28:18,950 of course consumes a lot of time and a 181 02:28:18,960 --> 02:28:23,270 lot of tokens. So how does this look at 182 02:28:23,280 --> 02:28:26,550 a protoscope? So I have my own eval set 183 02:28:26,560 --> 02:28:29,910 of around 350 tasks for coding knowledge 184 02:28:29,920 --> 02:28:33,429 work uh and terminal commands all run 185 02:28:33,439 --> 02:28:36,230 with Gemini 4 argan here and four 186 02:28:36,240 --> 02:28:38,070 different agent harnesses. One is the 187 02:28:38,080 --> 02:28:40,389 bash only harness and then okay with 188 02:28:40,399 --> 02:28:43,590 bash we still have that issue on how can 189 02:28:43,600 --> 02:28:46,630 I read images right if I'm I cannot cut 190 02:28:46,640 --> 02:28:49,349 screenshot.png it would return a binary 191 02:28:49,359 --> 02:28:51,349 and it will fill my context and the 192 02:28:51,359 --> 02:28:53,830 model cannot see the image. I added uh a 193 02:28:53,840 --> 02:28:55,830 view media tool which basically returns 194 02:28:55,840 --> 02:28:58,230 the image in a way the model can see. 195 02:28:58,240 --> 02:29:00,550 Then we have uh two different agent 196 02:29:00,560 --> 02:29:02,950 harnesses with structure tools uh for 197 02:29:02,960 --> 02:29:05,670 file system operations. And if we just 198 02:29:05,680 --> 02:29:08,870 look at the score column, all of them 199 02:29:08,880 --> 02:29:11,110 look roughly identical, right? So if we 200 02:29:11,120 --> 02:29:14,389 would just evaluate our model in ancient 201 02:29:14,399 --> 02:29:18,150 harness looking at the score, it would 202 02:29:18,160 --> 02:29:20,070 basically not really matter which one of 203 02:29:20,080 --> 02:29:21,670 those we would pick. Of course, there's 204 02:29:21,680 --> 02:29:23,830 a slight variance in the scores, but I 205 02:29:23,840 --> 02:29:27,349 only ran uh each task one time. So, 206 02:29:27,359 --> 02:29:29,110 might be just some noise or maybe 207 02:29:29,120 --> 02:29:30,630 something else. So, you would probably 208 02:29:30,640 --> 02:29:32,950 need to run a few more samples per per 209 02:29:32,960 --> 02:29:35,590 harness, but the score looks roughly the 210 02:29:35,600 --> 02:29:38,950 same. So, how would we now know which 211 02:29:38,960 --> 02:29:41,510 harness should I choose? If we just look 212 02:29:41,520 --> 02:29:44,550 all at all of the evol being shown, it's 213 02:29:44,560 --> 02:29:46,710 always mostly about score. But if we 214 02:29:46,720 --> 02:29:49,110 look at the token column or even more at 215 02:29:49,120 --> 02:29:51,510 a dollar per task, there's a picture 216 02:29:51,520 --> 02:29:54,309 which is changing. So the token per task 217 02:29:54,319 --> 02:29:56,950 is getting lower. Interestingly, if we 218 02:29:56,960 --> 02:29:59,429 just use bash and then also the cash hit 219 02:29:59,439 --> 02:30:01,590 rate increases. So if we just compare 220 02:30:01,600 --> 02:30:04,710 like the bash media as the the baseline, 221 02:30:04,720 --> 02:30:07,830 the tasks are almost 2x cheaper compared 222 02:30:07,840 --> 02:30:11,429 to more complex more tools agent 223 02:30:11,439 --> 02:30:14,389 harnesses. And we would not see it if we 224 02:30:14,399 --> 02:30:18,309 just look at the score. And to show you 225 02:30:18,319 --> 02:30:20,550 a few examples, I prepared some demos. 226 02:30:20,560 --> 02:30:23,349 So here I'm um using the PI harness, 227 02:30:23,359 --> 02:30:25,670 which makes it super simple to basically 228 02:30:25,680 --> 02:30:28,389 define what tools you want to use. I 229 02:30:28,399 --> 02:30:31,670 have the the same prompt for 3.5 flash, 230 02:30:31,680 --> 02:30:34,230 which we released roughly 6 months ago, 231 02:30:34,240 --> 02:30:36,550 uh for all tools. So like whatever pi 232 02:30:36,560 --> 02:30:38,870 ships out of the box. Then we have it 233 02:30:38,880 --> 02:30:41,830 for uh bash only and then we have it for 234 02:30:41,840 --> 02:30:45,429 carbon and uh sorry for carbon with the 235 02:30:45,439 --> 02:30:47,590 default tools and then with with um bash 236 02:30:47,600 --> 02:30:50,950 only. This runs now since we only have 237 02:30:50,960 --> 02:30:54,150 roughly around 20 minutes. I recorded 238 02:30:54,160 --> 02:30:56,150 all of that and clicked that very nice 239 02:30:56,160 --> 02:30:58,070 accelerate button here. And what we can 240 02:30:58,080 --> 02:31:01,349 see on the right side is that the same 241 02:31:01,359 --> 02:31:04,790 paradigma basically applies. when models 242 02:31:04,800 --> 02:31:07,349 get better, they roughly need less 243 02:31:07,359 --> 02:31:10,230 tools. So with 3.5 flash, we see with 244 02:31:10,240 --> 02:31:12,309 the default tools uh compared to bash 245 02:31:12,319 --> 02:31:14,230 only, we had the same amount of model 246 02:31:14,240 --> 02:31:15,910 calls. So we had the same amount of tool 247 02:31:15,920 --> 02:31:17,510 calls and we had roughly the same amount 248 02:31:17,520 --> 02:31:20,309 of tokens. And if we look at the traces, 249 02:31:20,319 --> 02:31:23,510 we can see for bash only, we have some 250 02:31:23,520 --> 02:31:25,670 cut commands to read different files. We 251 02:31:25,680 --> 02:31:27,510 have some inline scripts to make uh 252 02:31:27,520 --> 02:31:30,070 changes and then like some git changes. 253 02:31:30,080 --> 02:31:32,150 If we look at the default tools, we can 254 02:31:32,160 --> 02:31:34,630 see roughly the same. We had uh read 255 02:31:34,640 --> 02:31:36,950 file commands. We used the edit tool and 256 02:31:36,960 --> 02:31:38,870 then had some git commands to make those 257 02:31:38,880 --> 02:31:42,150 changes. If we now look at carbon with 258 02:31:42,160 --> 02:31:44,389 the default tools, it already used way 259 02:31:44,399 --> 02:31:48,150 less uh tool calls and model calls. So 260 02:31:48,160 --> 02:31:51,110 it it's basically doing work in parallel 261 02:31:51,120 --> 02:31:52,950 but also trying to reduce it. So we had 262 02:31:52,960 --> 02:31:55,990 for example the still same for write 263 02:31:56,000 --> 02:31:58,389 read calls and also the edit calls. But 264 02:31:58,399 --> 02:32:00,630 what starts to get interesting if we are 265 02:32:00,640 --> 02:32:03,190 going to look into bash and in bash we 266 02:32:03,200 --> 02:32:06,309 had even less tool calls and model calls 267 02:32:06,319 --> 02:32:08,550 and the interesting part here is that 268 02:32:08,560 --> 02:32:10,790 the model writes a single Python script 269 02:32:10,800 --> 02:32:13,429 to make all of that changes in one go 270 02:32:13,439 --> 02:32:15,590 and no longer needs to call multiple 271 02:32:15,600 --> 02:32:19,510 tools. So if we go further what does it 272 02:32:19,520 --> 02:32:21,670 means in terms for for carbon here and 273 02:32:21,680 --> 02:32:24,230 how bash pays off is that uh the better 274 02:32:24,240 --> 02:32:27,110 the model gets or the more reasoning and 275 02:32:27,120 --> 02:32:29,750 like capability it gets bash helps the 276 02:32:29,760 --> 02:32:32,230 model to be able to chain more commands 277 02:32:32,240 --> 02:32:35,270 in the single call for carbon it was 278 02:32:35,280 --> 02:32:39,270 roughly 2.2 to 3.7 uh commands per bash 279 02:32:39,280 --> 02:32:41,830 call and for flash it was only 1.3 to 280 02:32:41,840 --> 02:32:45,670 1.8 eight. So we are seeing up to 3x 281 02:32:45,680 --> 02:32:50,309 less tokens uh by carbon to achieve the 282 02:32:50,319 --> 02:32:52,870 same task compared to having the same 283 02:32:52,880 --> 02:32:57,030 tools we need to use. So two more 284 02:32:57,040 --> 02:32:59,349 examples to look at it as well is like 285 02:32:59,359 --> 02:33:01,670 we have on the left side carbon with the 286 02:33:01,680 --> 02:33:03,750 default tools. So we see our coding 287 02:33:03,760 --> 02:33:06,790 agent tools with write with edit and we 288 02:33:06,800 --> 02:33:09,429 see okay it does some parallel work in 289 02:33:09,439 --> 02:33:11,510 turn two some parallel work in turn 290 02:33:11,520 --> 02:33:13,750 three and four and then we have all of 291 02:33:13,760 --> 02:33:15,510 the changes and on the right side where 292 02:33:15,520 --> 02:33:17,830 it just has the bash tool it uses Python 293 02:33:17,840 --> 02:33:20,150 to write an inline script to make all of 294 02:33:20,160 --> 02:33:23,349 that changes at the same time with much 295 02:33:23,359 --> 02:33:26,389 less token and the same counts for if we 296 02:33:26,399 --> 02:33:29,830 compare Gemini 3.5 flesh to Gemini 4 297 02:33:29,840 --> 02:33:32,950 carbon flesh with bash only is not 298 02:33:32,960 --> 02:33:35,349 chaining those commands. So here it it's 299 02:33:35,359 --> 02:33:36,870 not really it doesn't matter if I'm 300 02:33:36,880 --> 02:33:39,910 having a a retool a write tool edit tool 301 02:33:39,920 --> 02:33:41,830 and a bash tool or if I have just a bash 302 02:33:41,840 --> 02:33:44,950 tool because 3.5 flesh used to use bash 303 02:33:44,960 --> 02:33:48,150 for single turn command so it had a 304 02:33:48,160 --> 02:33:50,070 single turn for reading it had a single 305 02:33:50,080 --> 02:33:51,990 turn for writing but if we look at 306 02:33:52,000 --> 02:33:54,230 carbon which in here in this example is 307 02:33:54,240 --> 02:33:57,030 the exact same task it needs less weight 308 02:33:57,040 --> 02:33:59,190 turns so the reading is not needed to 309 02:33:59,200 --> 02:34:01,670 read all of the different files and uh 310 02:34:01,680 --> 02:34:03,510 completely it just uses rip trap and 311 02:34:03,520 --> 02:34:05,270 like some intelligent searching to find 312 02:34:05,280 --> 02:34:07,030 where we need to make changes and then 313 02:34:07,040 --> 02:34:09,510 it writes a single Python inline bash 314 02:34:09,520 --> 02:34:11,830 command to make all of those edits in 315 02:34:11,840 --> 02:34:15,429 one go saving us a lot of tokens. So 316 02:34:15,439 --> 02:34:17,830 what does it bring us? Models will keep 317 02:34:17,840 --> 02:34:20,630 eating the harness and I'm pretty sure 318 02:34:20,640 --> 02:34:23,190 that 2027 will also be a lot more about 319 02:34:23,200 --> 02:34:25,429 how many tokens do we need to solve a 320 02:34:25,439 --> 02:34:28,230 task. Even for the few uh improvements 321 02:34:28,240 --> 02:34:30,630 we made between Argan and Carbon is the 322 02:34:30,640 --> 02:34:32,790 the better model needs less tokens per 323 02:34:32,800 --> 02:34:34,469 task which of course is very exciting 324 02:34:34,479 --> 02:34:36,710 for all of us because then we can use it 325 02:34:36,720 --> 02:34:39,670 more and also it matters a lot what the 326 02:34:39,680 --> 02:34:41,750 harness provides and we will might not 327 02:34:41,760 --> 02:34:44,469 see it if we just look at the accuracy. 328 02:34:44,479 --> 02:34:46,469 It's also very important that we start 329 02:34:46,479 --> 02:34:49,190 measuring the token efficiency because 330 02:34:49,200 --> 02:34:51,110 harnesses can look the same on accuracy 331 02:34:51,120 --> 02:34:53,110 because our models are now very good. So 332 02:34:53,120 --> 02:34:55,830 if your harness is more complex to use 333 02:34:55,840 --> 02:34:57,830 the model will work around it will find 334 02:34:57,840 --> 02:34:59,990 solutions around it but needs more turns 335 02:35:00,000 --> 02:35:02,150 and more tokens to achieve the same 336 02:35:02,160 --> 02:35:04,389 results and then also what comes 337 02:35:04,399 --> 02:35:07,110 afterwards. So I will try to do the same 338 02:35:07,120 --> 02:35:09,750 experiment 6 months from now and 339 02:35:09,760 --> 02:35:13,750 currently the trend we see slowly um 340 02:35:13,760 --> 02:35:16,150 rising at least on like social media and 341 02:35:16,160 --> 02:35:18,469 like the industry is that we might go 342 02:35:18,479 --> 02:35:21,590 back to some code mode tool that we 343 02:35:21,600 --> 02:35:23,830 instead of writing a bash commands that 344 02:35:23,840 --> 02:35:26,630 we write TypeScript or Python directly 345 02:35:26,640 --> 02:35:29,510 that MCP will come back. I'm a big fan 346 02:35:29,520 --> 02:35:31,510 of MCP and like they are making great 347 02:35:31,520 --> 02:35:35,030 changes to make it more uh easy easily 348 02:35:35,040 --> 02:35:38,070 usable especially uh interesting for all 349 02:35:38,080 --> 02:35:40,790 of that code mode work is uh MCP servers 350 02:35:40,800 --> 02:35:42,389 now return allow you to return 351 02:35:42,399 --> 02:35:44,469 structured output so the model can write 352 02:35:44,479 --> 02:35:46,389 better scripts to take more actions in 353 02:35:46,399 --> 02:35:48,469 the single turn and it's also allows us 354 02:35:48,479 --> 02:35:50,710 to do some better proxy around the 355 02:35:50,720 --> 02:35:52,630 security because bash calls are very 356 02:35:52,640 --> 02:35:55,110 hard to understand but if I have some 357 02:35:55,120 --> 02:35:58,070 abstractive layer in typescript I can uh 358 02:35:58,080 --> 02:36:00,389 watch all of the commands being um 359 02:36:00,399 --> 02:36:03,270 executed and make sure those are secure 360 02:36:03,280 --> 02:36:07,349 and for you build to delete your code 361 02:36:07,359 --> 02:36:09,910 before the model does. give your model 362 02:36:09,920 --> 02:36:12,950 uh to a computer, not a toolbox. And 363 02:36:12,960 --> 02:36:14,550 when a new model ships, I definitely 364 02:36:14,560 --> 02:36:17,510 recommend you um trying to think a 365 02:36:17,520 --> 02:36:19,190 little bit outside of the box, be more 366 02:36:19,200 --> 02:36:21,349 ambitious, try to really see, okay, what 367 02:36:21,359 --> 02:36:23,429 if I remove my complete harness and just 368 02:36:23,439 --> 02:36:25,830 give it one tool and then also please 369 02:36:25,840 --> 02:36:28,230 scan the QR code. You will get free 370 02:36:28,240 --> 02:36:30,469 credits. Uh we will send them out after 371 02:36:30,479 --> 02:36:32,790 the AI engineering event to try out 372 02:36:32,800 --> 02:36:34,870 Carbon and also to try out the new 373 02:36:34,880 --> 02:36:37,270 Gemini decisions API. So if you have 374 02:36:37,280 --> 02:36:39,270 some time today, I really can recommend 375 02:36:39,280 --> 02:36:41,510 checking out the new decisions model 376 02:36:41,520 --> 02:36:45,898 carbon and thank you for your time.