Thinking Deeper in Gemini

[upbeat music] [audience cheering] Hi, everybody.

Uh, yeah, my name is Jack. I'm a researcher at Google, and I'm the tech lead of Thinking within Gemini. And I'm gonna give a brief deep dive into Thinking from the research perspective, uh, within Gemini.

So, I'm going to kind of give this talk in three stages. One is to give a research motivation of why we actually are excited about Thinking in terms of unblocking bottlenecks towards intelligence.

And I'm gonna give a kind of, uh, give a few examples of how often discovering the most prescient bottlenecks, uh, in kind of our current, uh, models, uh, our most advanced systems.

How often if you can just identify the crucial kind of, uh, issues and shortcomings, you often will then find a solution. And there's a reason how that is linked to Thinking.

I'm then going to talk, um, a little bit more, um, just pragmatically about what is Thinking in Gemini, why is it interesting to developers. And I think you-- someone is...

Okay. The slides are still not here. We did do a rehearsal this morning where the slides were there. [laughing] But yeah, keynote speaker slide. Yeah. Someone's... I can see someone.

Yeah. It's the keynote speakers folder. [laughing] Jack Rae.

I think it's under keynote speaker. That one. [laughing] Um, anyway, um, it's gonna come up soon. You are close, um, person. [laughing] Um, yeah. But, um, and then I'm also gonna talk a little bit about what's next.

Ah, I'm just... Sorry, I'm just watching. You...

There you go. Nice one. [laughing] Yeah. That's great. Okay. The slides will appear. Thank you, whoever is coordinator. Apologies, I didn't know what happened. Um, and then I'm just t-gonna talk a bit about what's next.

So Logan did a great job of kinda giving a, uh, an incredible overview of Gemini as a whole ecosystem, everything that's going on. Uh, I'm gonna really be focusing on, on kind of what we're excited about in, in the reasoning space.

So with intelligence bottlenecks, uh, we're kind of... The, the message of this section is really about, uh, progress. So progress has really been marked by identifying key bottlenecks towards intelligence and then solving them.

And, uh, I'm gonna kinda give some examples throughout history. I'm gonna actually rewind the clock to 1948. Claude Shannon, he invents the language model mathematical theory of communication. He builds a language model, a 2-gram, using a, a textbook of word statistics that was hand-calculated, and he samples from it, and he kind of marvels at the samples.

He feels like, "These are, these are getting pretty good. They're a lot better than unigram character, this 2-gram word model." But, uh, kind of he remarks like, "I think this would be better if we could really, like, make a better language model and scale up this current method."

So he really wanted to just scale up the n-gram. That was the bottleneck. Like, small amount of data, very s- you know, elementary statistics. And, and unfortunately for Claude Shannon, kind of the solution was pretty hard.

He needed the digitalization of human knowledge, and he needed modern computing to be able to aggregate these statistics at scale. So, you know, that wasn't so easy for him to solve.

He had it a bit more tricky. But fast-forward a few decades. At Google, uh, in, in the 2000s, uh, my colleagues such as, uh, Jeff Dean are training n-gram language models over trillions of tokens.

These are powering, at the time, the most sophisticated speech recognition and translation systems. Uh, and, and a lot of progress has been made. But their bottleneck was actually, uh, with these systems, was that these n-gram language models were very restricted to short context.

And they were because, um, there was an ex-exponential storage cost with, uh, context length, and there wasn't really a way around that with, with just sticking with n-grams. The solution was the early kind of, uh, introduction of deep learning in 2010, uh, with, uh, the introduction of recurrent, uh, neural l- uh, language models.

So recurrent neural networks apply to modeling text. Where the recurrent neural networks could avoid this problem by, uh, storing compressed representation of the past into the state of a neural network, and they could now start to model beyond a 5-gram sentences or even paragraphs, and this was a massive kind of, uh, uh, step change and improvement.

However, a couple of years later, people would notice even there there was a bottleneck. So, uh, the recurrent neural network's representation of the past is in a fixed-size state, and this fixed-size state, uh, uh, there's only so much information you can put into it.

And so as a result, there's often observed to be kind of lossy, a lossy kind of representation of its context. The solution that was derived, I think once, once people kind of really encountered this, this, um, information bottleneck in the past, was actually just keep everything around in terms of your past, uh, neural, uh, embeddings, and use

an attention operator to aggregate things on the fly. So this was the birth of attention and then shortly after, transformers. So, um, transformers then kind of led to the modern deep learning revolution as we know it, and, uh, many other progress was made.

But if we skip forward ten years, we then are in 2024. We have, uh, large language models. They're increasingly powerful general conversational agents. We have, uh, models such as Gemini, ChatGPT.

People are using them for all sorts of use cases. And there, that's where we kind of come to the bottleneck that's relevant to this talk, which is that although these models are very, very powerful, they're still trained to respond immediately to requests.

So in other words, in terms of a compute bottleneck, there is a constant amount of compute that they apply at test time to transition from your request or your question to the response or your answer.

So the bottleneck of test-time compute, this is relevant to Thinking. Uh, so we can unpack this a little bit more. So when we talk about a fixed amount of test-time compute, the test-time compute is interesting to you because that's the compute that the model is spending on your particular problem, your particular question.

And it

And, and, and the way it actually kind of mechanically works is you have some text in your request, it gets translated to tokens, and then it's going to go through a language model.

And at the transition from the request to its response, it's going to pass some computation up through a large language model, which will have some parallel computation for every layer, and it'll have some iterative computation across layers.

So that computation is really where the model can apply its intelligence to your particular problem, and it's a fixed size. One solution if you wanted a smarter model and more computation is just to make the model larger.

Then you can have more compute, and you can get a smarter response. However, it's still not really enough. Users might want to be able to think a thousand or a million times and have a very large dynamic range and a lot of compute for very hard or challenging or valuable tasks.

And also, users might want to have a very dynamic application of test-time compute. So less compute for simpler requests, more compute for harder requests, and have this process be very dynamic and, and, and instigated by the model.

And that is what motivates thinking. So thinking in Gemini, mechanically, I'm sure almost everyone in this room is familiar with this general process where we will now have a model, and we in-insert a thinking stage, uh, that the model can emit some additional text before it decides to emit a final answer.

So going back to this notion of test-time compute now, we've added an additional kind of loop, uh, of computation where the model can kind of iteratively, uh, loop and, and perform additional test-time compute, uh, during this thinking stage.

And this loop can be potentially thousands or tens of thousands of, of iterations, which gives you tens of thousands more, uh, compute before it decides to commit to what its response will be.

And also because it's a loop, it's dynamic, so the model can learn how many iterations of this loop to apply before it decides to actually commit to its answer.

We train this model, um, to think, to use this kind of thinking stage via reinforcement learning. So when we pre-train Gemini, uh, we then have after a reinforcement learning stage where we train it to do many different tasks, and we give it positive and negative rewards depending on whether or not it solves the, uh, solves the task

correctly or not. And this is essentially a very general, uh, training recipe really, and it's kind of remarkable it works. But the model is able to just get a very vague signal of what is correct and what is not correct and to back propagate this through this loop of thinking stage such that it can try and shape

how it uses its thinking computation and thinking tokens in order to be more useful. In fact, we weren't really sure this would work. Um, it wasn't clear how much structure we should put into something like a reasoning stage.

And, um, although I think probably many people here have now seen reasoning traces and played with these models, I'll just show you a historical artifact, um, from one of the times we were trying to use reinforcement learning, we start to see cool emergent behavior.

So in, in this problem, there's kind of like an integer prediction problem. This was just like a, kind of a, a particular, uh, example. Uh, in this case, kind of like, um, kind of like a math-sy example.

And what we saw was the model was using its thinking tokens to actually first pose a hypothesis and then test out the hypothesis. And then it found that basically things weren't really working, and it, and it kind of states that this formula doesn't hold.

It rejects its own idea, and then it tries an alternative approach. And I think it's easy to become desensitized to technology because it's so amazing every single day. But we were truly blown away when we saw the general recipe of reinforcement learning was creating all sorts of interesting emergent behavior, trying different ideas, self-correction.

And I think these days we see a lot of different strategies that the model learns to do. So it learns to break down, uh, the problem into various components, explore multiple solutions, draft fragments of code and, and, and build these up in a modular way, perform intermediate calculations and use tools, all under the umbrella of using more

test-time compute to give you a smarter response.

Okay. So I've talked a bit about, uh, why we are interested in thinking in terms of the path to AGI and unblocking bottlenecks of intelligence and just a little bit about mechanically what it is.

Why is it interesting to developers? Obviously, the number one reason is we think this is driving, uh, more capable models, and it also stacks on top of our current paradigms of how we accelerate model progress.

So thinking, uh, we can, uh, kind of accelerate this process by scaling the amount of test-time compute, and we find that this can stack as a paradigm on top of preexisting paradigms such as pre-training, where you can scale the amount of pre-training data and, and, and model size, and also post-training, where you can scale the quality, uh,

and diversity of human feedback for many different types of tasks. And as a result, by... Within, within Google, by investing in all of these and really accelerating all of them, uh, we get kind of a multiplicative effect.

And why is this interesting to developers? I think it results in just overall faster model improvement, which is very nice.

We also see if we kind of, uh, look back over a, uh, a lineage of, uh, recent, um, Gemini launches, um, you know, there's improved reasoning performance and, and we can actually map this to how much test-time compute these models will devote to problems.

So there's kind of like a log scale test-time compute on the x-axis and performance across like math code and some science topics. And we see that there's kind of this trend in increasing reasoning performance whilst also it tracks very well with increasing test-time compute.

And on the far left, uh, you know, you have 2.0 Flash Experimental. This was a model that, uh, was not launched with thinking back in, uh, back in December last year, so ancient history.

Uh, and now we have, uh, uh, on the left, uh, on the right-hand side, what the, the first, uh, launched version of 2.5 Pro.

So test-time scaling is working empirically. Um, but it's not just capability that matters. It's also interesting from the notion of being able to steer the model's, uh, quality, uh, over cost.

So, um, you know, before, uh, you had the option of choosing a discrete number of possible model sizes, and that was a way to gauge how much quality you wanted and also how much cost you wanted to spend, um, uh, cost you wanted to kind of incur for any given task.

But it was kind of a discrete choice.

Now with Thinking we can have a continuous, uh, budget, uh, which allows you to have a much more granular slider of how much capability you want, uh, for any given kind of class of tasks.

And we have Thinking budgets now launched in, uh, Flash and Pro, uh, in the 2.5 series. And, um, this allows you to have very granular choice of cost to performance, and also allows us to then push the frontier and, and, and allow you to kind of augment and drive cost higher and performance higher if, if your application

requires it. So, okay, I think a lot of this stuff is really covering, uh, ground that, you know, uh, up to the present day. So what, what, what's next and what are we excited about?

So we're, we're very excited about just generally improving the models and having better reasoning, of course. We're also excited about making the thinking process as efficient as possible. Really, we want Thinking to just work for you and be quite adaptive and, and be something that you don't have to actively spend a lot of energy tuning.

And a big part of that is ensuring our models, uh, are very efficient in how they use their thoughts. Uh, this is definitely an area of progress. I think we can find examples of our models overthinking on tasks, and this is just an area of research to get these things faster and faster and, and as cost effective

as possible. We're very proud of how cost effective our Gemini models are, and this is just an area, uh, for improvement as well. And there's also deeper Thinking, which is really about scaling the amount of inference compute further to drive even higher capability.

So people may be familiar with Gemini Deep Research, where you can kind of, uh, type in a query and then, and then the model will go away for a long period of time and research a topic.

We're also now, uh, have announced at I/O, and we're launching to trusted testers, a notion of Deep Think. Deep Think is a very, a very high budget, uh, mode, um, thinking budget mode built on top of 2.5 Pro, and its desired application is for things where, uh, you have a very hard problem and you're happy to essentially,

um, uh, fire off the query and then have some asynchronous process that's running for a while and you'll come back to, to arrive at a stronger solution. And its key idea is, uh, we leverage much deeper chains of thought, uh, and parallel, uh, chains of thought that can integrate with each other to produce better responses.

We find this, uh, enhances model performance on very tough multimodal code math problems. An example would be USA Math Olympiad. This is a task that basically the state-of-the-art model in January was completely negligible performance.

Uh, 2.5 Pro is now, uh, probably even better, uh, with the, the updated one today. It was about a 50th percentile of all participants that participated in Math Olympiad and, and with Deep Think it goes up to 65, 65, uh, percentile.

And the interesting thing about Deep Think is as we continue to both improve the base model and improve the algorithmic ingredients that go into Deep Think, those two will stack together as well.

Um, here is kind of like a, just like a video animation of, of one of these USA Math Olympiad algebra problems. And, and the key idea really with this video is just this notion of having multiple iterative, uh, ideas.

So maybe the model starts out with some proof by contradiction idea but then explores two different aspects, some Rolle's theorem, Newton's inequalities. It integrates them and eventually arrives at some correct proof.

There's not that much you can take away from this video, but it looks pretty cool, so I added it. [laughs] Yeah. Yeah.

One thing that's, you know, other than we talk about math a little bit in the previous slides, I'm very excited about any application where the model can spend longer and longer thinking on very open-ended coding tasks and one shot or very few interaction vibe code, things that would've taken us months, uh, in the past.

Uh, and one example that I like from a researcher is just, um, um, some of my colleagues kind of vibe coded, uh, from, from DeepMind's original DQN paper, which was a, a revolution in deep reinforcement learning, kind of vibe coded, uh, Gemini vibe coded the, the kind of training setup, the algorithm, uh, even an Atari emulator such

that it could play some of the games. And, you know, this is, uh, remarkable to me 'cause this, these kind of things would've taken me and my colleagues, uh, months in the past, and these things are starting to happen, um, uh, kind of in minutes.

One thing I'm quite excited about looking forward to the future is not really the landscape of models, but coming back to, like, what's our gold standard, which is the human mind.

I would love for our models to be able to contemplate from a very small set of knowledge and think about it incredibly deeply such that we can push the frontier.

And one example I often think about is Ramanujan, who's, uh, one of the world's greatest mathematicians, uh, from the early 20th century. And famously, he, he just had this one math textbook.

He was kind of cut away from, from the mathematical community, but he... Just from a small set of problems he spent, uh, many textbooks' worth of thinking, going through problems, inventing his own theories to further extend ideas, and he invented a, an, an incredible quantity of mathematics really just by deeply, deeply thinking from a small source subset.

And this is where I think we are going with Thinking. We want a model to be able to be incredibly data efficient and actually go to millions, uh, or, or beyond of, of, of inference tokens where the model is really building up knowledge and artifacts such that we can eventually start to push the frontier of human understanding.

So with that said, thank you very much. [clapping] And, uh, yeah. [upbeat music]