AI Engineer Europe 2026
Text Diffusion — Brendan O'Donoghue, Google DeepMind
Read the talk
Text diffusion: refining a whole answer instead of predicting the next token
Text diffusion turns generation into repeated revision, trading extra computation for lower latency, editable answers, and inference budgets that adapt to the task.
From a talk by Brendan O'Donoghue
Before you start: Familiarity with tokens, transformer attention, and the distinction between latency and throughput will help with the hardware and serving discussion.
From a corrupted sentence to readable text
Take a clean sentence, replace some of its tokens with random vocabulary entries, and ask a neural network to repair it. Repeat that training exercise at different noise levels. At inference time, start with an entirely random sequence and repeatedly remove the corruption until readable text emerges. This is text diffusion: the same broad denoising principle used for images and video, applied to a sequence of tokens.
Corruption can happen in a continuous space or directly over discrete tokens. Brendan O’Donoghue’s explanation uses the discrete case. A small Python example makes the training-side operation concrete: noise_probability controls how much of a clean token sequence is replaced, while the original sequence remains the target to recover.
python
import random
def corrupt_tokens(
clean: list[int],
vocabulary_size: int,
noise_probability: float,
rng: random.Random,
) -> list[int]:
if vocabulary_size <= 0:
raise ValueError("vocabulary_size must be positive")
if not 0.0 <= noise_probability <= 1.0:
raise ValueError("noise_probability must be between 0 and 1")
return [
rng.randrange(vocabulary_size)
if rng.random() < noise_probability
else token
for token in clean
]
clean = [12, 41, 7, 93]
rng = random.Random(0)
training_input = corrupt_tokens(clean, 100, 0.5, rng)
inference_start = corrupt_tokens(clean, 100, 1.0, rng)
The token IDs here are teaching values. The learned part is the reverse operation: successive predictions turn the noisy canvas into a coherent sequence, filling in information in the order the model finds useful.
That process produces a recognizable visual effect: scattered text gradually settles into a clean response. Gemini Diffusion brought it to a research preview that O’Donoghue describes as reaching about 100,000 people. The model branched from the architecture of Gemini 2.0 Flash-Lite, which also served as its historical comparator. O’Donoghue describes broadly similar overall quality, with some advantages in code and disadvantages elsewhere, at substantially lower latency. These comparisons concern the preview he describes as roughly a year old, rather than a current model ranking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A revisable canvas changes the serving trade-off
An autoregressive model generates a response one token at a time. Each new token depends on the context and the tokens already produced. Diffusion instead initializes a canvas—potentially hundreds or thousands of tokens—and processes the whole block repeatedly. It still needs multiple forward passes, but each pass can update many positions rather than append only one token.
The immediate attraction is faster generation through better use of GPUs and TPUs. But the change in generation order also creates other capabilities:
- Bidirectional attention: a position can attend to later positions within the canvas, allowing later reasoning to inform an earlier answer.
- Adaptive computation: a model can learn to spend more refinement steps on a difficult response and fewer on an easy one.
- In-place editing: some tokens can remain fixed while other positions change—for example, holding a suffix constant and generating a compatible prefix.
These capabilities follow from treating the response as something to revise, rather than a stream whose emitted tokens are already committed.
Lower latency for one request does not imply higher throughput across many requests. Autoregressive serving can combine a large batch of queries, using the accelerator efficiently even while each individual user waits for sequential decoding. Diffusion repeatedly processes the same output positions and reaches the accelerator’s compute limit sooner.
| Serving property | Autoregressive decoding | Text diffusion |
|---|---|---|
| Work per pass | Next token per request | Many positions per request |
| Single-request latency | Sequential decoding limits speed | Parallel refinement can reduce latency |
| Large-batch throughput | Batching amortizes hardware use | Repeated passes consume more compute |
O’Donoghue identifies this throughput penalty, and its corresponding serving cost, as the main obstacle to diffusion in large hosted models. His aside about Claude capacity concerns underscores why providers care about aggregate throughput even when users would prefer faster individual responses.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why extra computation can still be faster
The hardware explanation starts with the gap between arithmetic capacity and memory bandwidth. Tensor cores perform matrix multiplications efficiently, but the weights, activations, and key-value cache reside in high-bandwidth memory, or HBM. Moving that data into the compute units takes time. GPUs and TPUs have abundant arithmetic capacity relative to the bandwidth available to feed it, so doing more useful computation per transfer can improve utilization.
At batch size one, autoregressive decoding is often memory-bound: the model streams weights and relevant cache data to produce a token, then repeats for the next token. A diffusion pass also moves model data, but it works on a block of output positions. O’Donoghue’s illustrative calculation uses 24 passes to generate 256 tokens: roughly ten times fewer memory transfers, with roughly tenfold speed possible only if execution remains truly memory-bound.
The condition matters: once arithmetic becomes the bottleneck, reducing transfer count no longer translates directly into the same latency reduction.
O’Donoghue reports approximately 2,000 tokens per second delivered in the Gemini Diffusion browser demo, including prefill, with speed depending on query and output length. Longer outputs amortize the initial processing of the prompt; a one-token answer can be dominated almost entirely by that prefill cost. The figure describes the browser experience he reports, not a hardware-independent decoding rate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An answer changes from 60 to 49 to 39
The arithmetic demonstration from Google I/O exposes what bidirectional generation makes possible. The prompt involves the square root of 81 and two-thirds squared, followed by additional arithmetic; the target answer is 39. After its first forward pass, Gemini Diffusion has already written an opening answer of 60 and begun a solution underneath it. That answer is still part of a mutable canvas.
On the second pass, the opening answer becomes 49 and the reasoning advances. On the third, the solution reaches 36 + 3 = 39, and the model changes the earlier answer to match.
| Forward pass | Opening answer | Reasoning state |
|---|---|---|
| First | 60 | Solution begins |
| Second | 49 | More intermediate steps appear |
| Third | 39 | Reaches 36 + 3 = 39 |
Additional passes still clean up remaining text. The demonstration therefore shows the core correction by the third pass, not a complete response produced in exactly three passes. Later tokens provide information that can repair an earlier position before the answer is finalized.
O’Donoghue contrasts this with historical runs of the same prompt on two models he describes as much larger. GPT-4o initially answered 40, worked through the reasoning, and then appended a correction to 39. Gemini 2.5 Flash answered 42 and retained that mistake, eventually treating 36 + 3 as 42. These examples illustrate two consequences of committing an answer before completing its justification: a correction must be appended, or subsequent reasoning may accommodate the initial error. O’Donoghue acknowledges that modern thinking models can address this problem; diffusion’s distinctive capability is to revise the existing answer directly within the canvas.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More time to revise, and a learned decision to stop
Dynamic computation means increasing the available inference budget. More denoising passes give the model additional opportunities to inspect and repair an answer, even after it appears nearly clean. Across six internal coding evaluations, O’Donoghue reports generally improving quality as the denoising-step budget increases, though not strictly monotonically.
Adaptive computation is a separate choice: train the model to determine when it has finished, so different requests consume different amounts of the available budget. The preview examples make the difference visible:
| Request | Reported denoising steps |
|---|---|
| First 100 digits of pi, described as 100 tokens | 4 |
| FizzBuzz code | 18 |
| Quantum mechanics in one paragraph | 31 |
O’Donoghue attributes the short pi run to memorization: the model can recover a familiar sequence quickly. He contrasts its four refinement passes with only four tokens from autoregressive generation in the same time. The code and explanatory paragraph use more passes because the model chooses to keep refining them.
The same pattern appears across tasks in the preview’s evaluations. GPQA Diamond was difficult for the targeted model size and took longer; Mostly Basic Python Problems, or MBPP, generally required less time. The stopping decision came from the model rather than a manually assigned step count for each benchmark. These observations concern the older preview and its evaluation setup.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Edit a region while preserving its surroundings
Image inpainting provides a useful analogy for editing text. Remove part of an image and a diffusion model can fill the gap using the surrounding pixels and an instruction. The slide illustrates localized changes to a dog’s appearance. Unlike raster-order generation, which proceeds left to right and top to bottom, diffusion can condition a missing region on context throughout the image.
The text demonstrations apply the same principle at different scales. First, a request to fix a code bug produces a local indexing correction. Next, a request for documentation inserts explanatory text into the code. Finally, a story receives a new middle paragraph that can depend on both the first and third paragraphs. The important operation is context-preserving insertion or revision: the model uses material on both sides of the edit instead of simply regenerating every token from the beginning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate the interface at each click
Low latency becomes more interesting when generation moves into an interaction loop. The first application demonstration resembles Wikipedia, but both the article text and its HTML are generated on demand. Clicking a link produces the next page quickly enough to resemble ordinary navigation. The model is generating the page itself, not merely answering a question inside a fixed interface.
A Reddit-style demonstration extends that idea to invented communities, posts, and comments, including a sharks-themed community. Gemini Diffusion generates text and HTML, while a separate image model supplies the pictures. O’Donoghue tentatively recalls that image model as Juno, from before Nano Banana; the text model itself does not generate images. The slower pictures arrive after the text, which he describes as about a minute later, while the site remains an interactive, generated simulation.
The next demonstration presents an operating-system simulation. Each click prompts generation of the next screen: opening a README produces its contents, other interactions produce web pages, and navigation can return to the desktop. The application opportunity is an interface whose content and screens are synthesized as the user explores them, provided generation stays within the latency budget of that interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A voice-driven loop for building an app
An external user’s demonstration moves the interaction loop into software creation. O’Donoghue corrects his introduction from an API to the Gemini Diffusion webpage: that is the interface used in this historical demo. The spoken requests build a to-do application incrementally:
- Create the app and add 10 random to-dos.
- Add a completed state and mark four random items complete.
- Enable sorting by name and by state.
- Test sorting, add an item, and delete several items.
- Request dark mode.
The demonstrator reports that the tested interactions work and calls the sequence “15 seconds of work.” That is the demonstrator’s description, rather than an independently measured end-to-end duration.
The prepared talk ends with this product question: what becomes possible when a model responds quickly enough to keep up with a user’s next instruction? O’Donoghue anticipates another generation of models and invites engineers to explore those experiences, without supplying a release date.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Training compatibility, larger models, and pricing
The audience questions first separate data compatibility from algorithm compatibility. Asked about pretraining corpora and reinforcement learning from human feedback, O’Donoghue says the team uses the same data, but the algorithms need to change. He also confirms that diffusion models can be distilled, without describing a technique or identifying a published implementation. Asked about general availability, he anticipates a release soon but offers neither a date nor a definite general-availability commitment.
Increasing model size introduces another compute trade-off. O’Donoghue reports that larger models tend to need fewer denoising passes for the same output. Each pass costs more FLOPs, but fewer passes can partially offset that increase; he provides no quantitative scaling curve. Commercial pricing remains a separate, unanswered question: when asked whether tokens are the right billing unit, he says pricing has not yet been determined.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fixed windows, bounded steps, and committed output
A text response has no predetermined dimensions like an image, so how large should the diffusion canvas be? The simplest approach uses a fixed-length window and generates successive windows autoregressively. Another option is a prediction head that estimates response length. A fixed window does not impose a fixed total answer length: the model can continue through as many windows as needed.
Adaptive stopping also operates inside a limit. O’Donoghue clarifies that the earlier examples all had a maximum denoising-step budget; the model could finish before exhausting it. This gives the system a cap on refinement work while preserving the ability to return easy responses early.
The next question establishes the boundary of self-correction. In the team’s implementation, completed windows are frozen before generation continues. Revisiting them is conceivable, but it is not the behavior he describes. Tokens are revisable within the active window; earlier completed windows are committed.
That makes the architecture a hybrid already: diffusion within blocks and autoregression across blocks. Context prefill remains the same; generation then proceeds in fixed-size blocks. O’Donoghue offers 512, 1,000, and 32 tokens as illustrative sizes, not a single disclosed configuration. Calling the process autoregressive across blocks does not mean that each token inside the block is generated sequentially.
Nor does the process require diffusion in a latent embedding space. A conventional prediction head can produce logits over token IDs. In the discrete process described here, every denoising step takes tokens in and returns tokens out. Latent-space approaches also exist, but O’Donoghue characterizes most contemporary text-diffusion work as discrete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where low latency matters most
For the near term, O’Donoghue expects diffusion and autoregression to serve different use cases. On a phone or robot, a model may serve one local interaction rather than share an accelerator with thousands of unrelated requests. That removes much of the large-batch throughput advantage of autoregressive serving. He reports that text diffusion is already used in some on-device applications within Alphabet, mentioning robotics.
Asked whether diffusion can reach frontier-model quality, he identifies throughput rather than quality as the hosted-serving obstacle and affirms that his team’s research is roughly at that quality level. No supporting frontier benchmark is presented in this exchange. The deployment decision he describes is therefore about whether the application values a fast individual response enough to accept the cost of repeated refinement.
The remaining training questions receive brief confirmations. Reinforcement learning for reasoning is possible with algorithm changes. Diffusion and autoregressive models can also be combined, and O’Donoghue confirms that his team has done so, without explaining the implementation.
Initializing the canvas with a smaller model’s draft sounds like a way to avoid starting from noise, but it changes the input distribution. A denoiser trained on noise expects noise. To refine a smaller model’s output reliably, it would need training on that kind of output instead. O’Donoghue says the team does not usually take this route because it adds complexity. Asked for an implementation paper, he offers none from his team and points generally to external literature.
The final question turns the objective around: what happens if the user asks the model to generate noise? O’Donoghue says he does not know and does not think the team has tried it. His guess that it would produce something remains an open experiment—a fitting boundary between a demonstrated denoising mechanism and behavior that still needs to be tested.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official explanation of Gemini Diffusion, with historical Flash-Lite comparisons and sampling-speed methodology.
The May 2025 announcement introducing the experimental text-diffusion demo.
Original research introducing the graduate-level science question benchmark underlying GPQA Diamond.
Google Research's Python programming dataset, including task descriptions, reference solutions, and tests.
Further reading
- Block DiffusionPaper
Research combining autoregression across blocks with diffusion within blocks to support flexible-length generation and KV caching.
Official Block Diffusion code, checkpoints, and instructions for training, sampling, and evaluation.
Updates since the talk
- Introducing DiffusionGemmaArticle
June 2026 announcement of Google's experimental open diffusion model, including hardware-dependent speed claims and serving trade-offs.
Architecture and deployment guidance covering 256-token denoising blocks, KV caching, adaptive stopping, and local vLLM serving.
Read the complete timestamped transcript
- 0:15
Some people are still filtering into the room. But, uh, it's mostly intro stuff for the first couple of slides, so they won't miss anything. Uh, okay. Welcome everybody. My name's Brendan.
- 0:23
I'm, I'm, uh, a research scientist at DeepMind. I'm talking today about text diffusion, which is kind of a more forward-looking research area, uh, at, at DeepMind. So you're probably familiar with image and video diffusion, which is kind of state-of-the-art for these modalities right now, where you, you know, you take ground truth, say image, you add noise to
- 0:42
it in training, and then you train a neural network to remove that noise gradually. And then at inference time, you just initialize the, the picture with pure noise, and then you iteratively refine out the noise to recover back to, you know, whatever image or video or audio or whatever you're looking for.
- 1:00
And the principle is essentially the same for text, for text diffusion, where you start with a clean, uh, sequence of tokens, like so a clean sentence or something like that, and then you gradually add noise.
- 1:12
You corrupt it somehow. There- there's lots of different ways to do that. You can do it in a continuous or discrete way, but let's just say discrete for now, which would in this case just mean adding random tokens or replacing tokens with other random tokens.
- 1:23
And you do that for a bunch of different noise levels, and you train the neural network to try to fill in, to try to correct the mistakes basically in the text.
- 1:32
And then at inference time, you initialize the sequence of tokens to just pure noise, like pure random discrete tokens from the vocabulary, and then you iteratively refine through that to fill in the information in the order that the neural network wants to do to recover back to, say, a clean sentence.
- 1:49
Uh, and then in practice, so we, you know, I saw- we sh- I showed you some GIFs here of what it looks like for images. You get very similar looking outputs for text where it kind of, uh, you know, starts off all noisy and then gradually like fills in the text and you- and you get clean, relatively
- 2:03
clean outputs at the end. Okay. So y- the, the team I'm on, we, uh, we had a, a research demo release o- one year ago now, uh, called Gemini Diffusion, which was a variant of a Gemini model which did text diffusion instead of autoregressive next token generation.
- 2:19
And that was like a research preview that was open to about 100K people. Uh, uh, and, you know, we're still, you know, keep, keep posted for new, new developments in that direction, uh, upcoming soon.
- 2:32
Um, and we did have some good numbers at the time, but again, it's a year ago, which is like, you know, prehistoric times in this field. Uh, our kind of main comparator model was Gemini 2.0 Flashlight at the time 'cause that was the architecture we were branching from, the text diffusion model, and we basically had very similar
- 2:48
quality across the board there. Um, mostly, you know, a l- a little bit of advantage in code, a little bit of disadvantage in some other areas, but kind of relatively similar performance at much better latencies.
- 3:00
Um, but again, yeah, this is a year ago, so I wouldn't fixate too much on these numbers. Okay. So what's the difference between autoregressive generation and diffusion? So in, in the standard vanilla Gemini, Gemma, you know, GPT, whatever, um, generation of text, you do this, you know, you have some context that comes in and you wanna generate
- 3:17
some response to that, and the model does it one token at a time. So it generates the first token and then condition on that generates the next token and so on.
- 3:24
Whereas in diffusion, you have the, you know, a context will come in, whatever that is, and it'll initialize like un- like I mentioned, like a long sequence of tokens, could be hundreds, could be thousands, could be shorter, depends on the model, uh, to be random noise, and then it iteratively refines that canvas to remove the noise over
- 3:41
the course of a few denoising steps. So rather than one token at a time, it does the entire block together but over a couple of iterations. So it's not just one pass, it does multiple passes, but it gets to attend to the future tokens and so on.
- 3:55
So it's a kind of a different way of generating text. Uh, um, so that obviously has some pros and cons. So the main pro that people really like and probably is the biggest advantage that text diffusion models have is that it's, it's just faster inference.
- 4:09
It just generates faster tokens per second 'cause it makes much better use of the hardware, the TPU and the GPU. And I have some slides on that to explain why.
- 4:17
Um, but some other advantages are it can do bidirectional attention within the, within this canvas of tokens. So, you know, uh, autoregressive models can only attend to the past.
- 4:26
They have causal attention within their, you know, their transformer. Whereas the, a text diffusion model is not restricted by that. It can attend to the future. And that has some i-interesting properties, like it can do self-corrected generation based on future tokens.
- 4:39
So it could like, you know, do some reasoning, see that it got the answer incorrect, and then go back and fix the reasoning and do it again. I have, I have a demo of that.
- 4:47
Um, because this, this process is, is iterative, it does a number of steps to, to, to respond. It- that means a model can actually do adaptive computation. It, it, it turns out that you can train the model to spend more time on harder problems and less time on easier problems.
- 5:02
And like, like diffusion models in general, you can do things like in-place editing where you say like, "Fix the last tokens and give me the prefix that corresponds to those tokens," and stuff.
- 5:11
Um, but the main disadvantage it has and the reason why it's not kind of used everywhere right now is lower throughput for large batches. So autoregressive models are slow, but you can have a big batch of queries together and then if you push that through the neural network to, uh, on the GPU, each individual user is slow
- 5:31
but the- you make good use of the TPU by doing that and so you can serve a lot of queries and so you keep your costs down. You can serve a lot.
- 5:38
Whereas since text diffusion does m-multiple forward passes on the same data, it, you know, hits a compute threshold basically earlier and it, it's even though it's lower latency for any one user, it tends to be, you know, lower throughput overall, so higher cost to serve.
- 5:53
And right now, you know, if you have, if you've played with Claude recently, uh, you'll know that they have some throughput concerns so, uh-
- 6:02
People really care about throughput right now, and so no one's landing text diffusion into any of these big models primarily because of that disadvantage. It's just too expensive to serve, even if it is much lower latency.
- 6:13
Okay. So just leading into why does it have lower latency, unless you-- in case you're not familiar with kind of the architecture of how GPUs and TPUs run today.
- 6:20
So in a, in a GPU, there's like a, a tensor core which does these like big matrix multiplies. It's very efficient, has a lot of flops or hops or, you know, whatever.
- 6:29
Uh, and then the memory that sits on the TPU, GPU, this, this, uh, HBM, that's where the weights and the activations and everything are stored, and it has to transfer over from the memory all the weights and the activations in the KV cache into the tensor core in order to do the computation.
- 6:46
And so it has to flow through this bandwidth channel, and that bandwidth channel is very tight. It turns out that the, that both GPUs and TPUs have a lot of flops and not that much bandwidth.
- 6:56
It's quite hard to... It's expensive to put bandwidth onto these chips, and it's easy to put flops. So because of that ratio, you can, uh, you, you know, you can...
- 7:04
If you do the more flops you do for each streaming amount of data you put through, the better, right? So,
- 7:12
so ba- uh, when we're serving, uh, an autoregressive model, we- these, these, these chips are memory bound. They're basically bottlenecked by this bandwidth. Um, so when you do autoregressive next token generation, for each token, you're doing one token at a time, let's say batch size one.
- 7:27
You have to stream over the entire neural network and all the KV cache and everything to get one token, and then you do it again for the next token, and so on.
- 7:35
Whereas for text diffusion, you're generating, say, two fifty-six tokens. You still stream over everything. But if you can do that less times than the number of tokens, this iterative refinement process, then you'll get a speed up.
- 7:49
So if you can do, say, twenty-four passes to generate two fifty-six tokens, you'll be doing ten times fewer memory transfers than an autoregressive model. And if you are truly memory bound, then you'll be ten times faster, something like that.
- 8:02
So that's, that's the real reason. That's the hardware reason why, uh, text diffusion models are much lower latency than autoregressive models. Okay. So, you know, we had this Gemini diffusion demo.
- 8:13
M- maybe some of you got access to it, uh, last year. And, you know, that was able to hit, you know, something like two thousand tokens a second, um, pretty consistently, depending on the length of the query.
- 8:22
Obviously, it depends. The longer sequence it's generating, the less it's prefill dominated, and so you can really lean into these very long sequences of very fast tokens. But if you're only generating one token, for instance, you'll be just dominated by the cost of the prefill.
- 8:35
And the tokens per second number that was reported on this webpage was incorporated prefill and everything like that. So this was two thousand tokens a second. That's genuine raw tokens that you would receive in your web browser.
- 8:46
Okay. So that was the kind of a whirlwind tour of text diffusion and its main advantage, which is latency. But I wanna dig in a little bit into some of the other advantages that text diffusion has, which are kind of a little bit less talked about in the literature, um, but I think are pretty cool, and this
- 9:00
is why I'm excited about it. So at, at, at the, at I/O last year, Google I/O last year, they showed this demo for the text diffusion model, which is, you know, it's this really easy prompt.
- 9:10
But, uh, you know, lots of models actually make mistakes on it. So the prompt is, could you go to next slide? What is the square root of eighty-one times two-thirds squared plus, you know, blah, blah, blah?
- 9:20
And I think the answer is thirty-nine to this problem.
- 9:23
And so, you know, you pass that into the model, and you ask the, you ask the Gemini diffusion to respond to that. And after one forward pass, these are the tokens it's generated.
- 9:33
So one forward pass through the model, it's starting to respond. Now, it's doing this iterative refinement process, so one forward pass is, is not all it's going to do.
- 9:41
But after one forward pass, it has this. So it has answer equals, and then it says sixty. It's not correct, but that's what it's guessing for now. And then it starts to do the reasoning.
- 9:50
So it has solution, calculate the square root of eight- of eighty-one, and so on. After two forward passes, it's changed sixty to forty-nine, and it's gotten a little bit further into the reasoning.
- 10:00
So it's gotten like, you know, five steps into the reasoning. Two squared equals four. And some of the blue tokens are kinda still gonna change. And then after three forward passes, it's actually gotten all the way through the reasoning.
- 10:12
So it gets, it gets, and it gets the answer correct at the end. Th- thirty-six plus three is equal to thirty-nine. And it's gone back and fixed the original response to say thirty-nine.
- 10:20
So it's, uh, it had a mistake twice, sixty and a forty-nine. But once it finished the reasoning, it was able to return back and fix the mistake that it made at the start.
- 10:30
Now, it's gonna do a couple more forward passes in order to fix, you know, some of these tokens that aren't quite right in the text. But overall, that's basically the structure of the output that it'll return.
- 10:38
And this is, you know, this is a, a property that text diffusion models have, this ability to do bidirectional reasoning. So to, to not only see the past but also see the future that it's going to utter, it's gonna respond, and also to use that information to do self-correction.
- 10:52
So it made a mistake, but it was able to, you know, had another forward pass. It was able to go through and fix that mistake. At the time, you know, much, much bigger models than the one we were serving made a mistake for this problem.
- 11:04
So both Cha- ChatGPT 4o, which was new at the time, and Gemini 2.5 Flash, which was brand new at the time, both made an error on this p- on this exact problem.
- 11:12
So you give them the exact same prompt, and then they would say, you know, remember the answer is thirty-nine. The GPT 4o said forty 'cause that's the best guess it can do at that one token.
- 11:21
It went through the reasoning, and then it said thirty... It did manage to figure out it was thirty-nine. It said, "I made a m- Sorry, I made a mistake.
- 11:26
It's thirty-nine, not forty." Uh, Gemini 2.5 Flash also made a mistake, said forty-two, and then it, it actually just stuck to its guns and never changed it and said, "Thirty-six plus three is forty-two."
- 11:35
So it like inc- incorporated the error into its reasoning later. Uh, and these are way bigger models than the Gemini diffusion model. So it's, it really is a property, a flaw of autoregressive models that the text diffusion models don't have.
- 11:47
Um, and you know, you can fix this with modern reasoning thinking models, uh, but you know, then you're just kind of punting the, the problem into something else. But Anyway, okay.
- 11:56
So that's, that's, that's one advantage, which is bidirectional reasoning, self-correction. Another one is what I, I hinted at before, which is dynamic computation. So you can, you can give the model more time at inference, more forward passes.
- 12:07
You give it a bigger budget, and it can just do better. It's not exactly monotonic, but it is roughly monotonic that the quality across every eval basically just continues to go up because it can...
- 12:17
It gets... E-even if the, even if the solution is almost entirely clean and correct, it gets to look at it and see that it made a mistake and then fix it.
- 12:25
So you get this, you get this nice kind of curve where you always see as the number of denoising steps, which is the forward passes go- increases, you get, you go- overall the, the, uh, quality gets higher.
- 12:35
And these are just six coding evals that we monitor internally.
- 12:39
On top of that, a slightly different concept is it, is the model can do adaptive computation, which is that you can allow the model, you train it in a way to determine itself when it is finished, and then for easy responses, it can use a little bit of compute, and for harder responses, it can take longer.
- 12:56
So here's just three examples from, from the Gemini diffusion model, which is what are the first one hundred digits of pi? This is actually a hundred tokens. It looks like a short response.
- 13:04
It's actually a hundred tokens. And it only takes four steps to do that because the model i- it's an easy promp- it's an easy response 'cause you've just memorized the hundred digits of pi.
- 13:12
You can just output it. Whereas, uh, you know, four, uh, an autoregressive model in f- the same time would have only done four tokens.
- 13:19
Um, so that's a very easy one. Slightly more challenging is to write a little bit of code. So that takes, you know, 18 forward passes, uh, to generate fizz buzz.
- 13:27
And then some- something more complicated is explain quantum mechanics in a single paragraph, and that took 31 denoising steps. It just took its time, just for whatever reason, decided to spend longer on those ones.
- 13:36
And so the model naturally gets to decide, I guess, to determine when it's going to finish and return the response. And, and typically, we see that harder evals take more time.
- 13:47
So th-this was, this is a year ago now, so the evals are kind of old school. But, um, the, you know, on the higher- on the harder end is GPQA diamond, which for th-the model size we were targeting was quite a hard eval, and that took a long time for it to respond to those ones.
- 14:03
Whereas on the other end are like MBPP, which is mostly basic Python programs. It was very, it was very easy to re- you know, took a very little time for it to respond to these things.
- 14:11
And this was entirely determined by the model itself. Just easier problems, easier prompts it could respond to quickly, and harder ones it decided itself to spend more time reasoning.
- 14:20
Okay. So that's, that's another property, which is the dynamic and adaptive computation. Lastly is the kind of i- fast in-place editing. So diffusion models in general have this very nice property where you can so take an image, like in this example, you know, cut something out of it or whatever, or give it a little prompt, and it'll
- 14:41
fill it in, and you can use the context, uh, uh, that, that you haven't cut out to fill in the, the piece you've cut out correctly. And so you, you can use that for like clever image editing, things like that.
- 14:54
Um, and the becau- the reason it can do that is because, uh, you know, it's not autoregressive. There are autoregressive image generators, right, which go left to right, top to bottom, like raster order, generating pixels.
- 15:03
You know, but diffusion doesn't work like that. It'll just see the entire image and then start to denoise it. And because of that, because it gets to see every pix- every pixel gets to see every pixel, it can fill in the missing information and, and do it in a way that's kinda consistent whatever prompt you're giving.
- 15:20
Uh, so we can do something similar. So I have a couple of demos here. Uh, can you see that? Yeah. So this is just some code, and you say, "There's a bug in this code.
- 15:28
Can you fix it?" And it just, it'll just make the edit in the correct place. Like, it won't... You, you can barely see that, but it's just did a little fix here of the indices.
- 15:35
Uh, and you can say, uh, things like, "Can you add documentation?" It'll go in. It's not re- it's not just one by one generating all the tokens. It's doing a clever editing procedure to, to actually fill in the, uh, the correct edits here.
- 15:47
You can do that with, you know, more general text. Like, you can take a story and then say, "Add a middle paragraph." And because it can see the, the first and the, and this third paragraph, it can fill in the paragraph in a way that's consistent with the, the rest of the story, for instance.
- 16:02
And this is just in-place editing, basically. Um, okay. So that's, uh... Those are some of the advantages. Uh, don't have a lot of time. Uh, m- the biggest advantage, like I mentioned, is this low latency, and, you know, we, we really lean into that.
- 16:17
And, uh, I just wanna show you a couple of demos of some of the things that people internally have built to kind of show what the advantage of low latency can give you.
- 16:25
So it's not just the same thing faster. It can really unlock some new, some really new a-applications. So in your own work, you're all AI engineers, uh, it'd be interesting to see when, when the next diffusion model comes out from our team, what the low latency could unlock, uh, and kinda what new applications can be built.
- 16:43
So here are just some demos. So this is Wikipedia. Let me just pause it, actually. This is Wikipedia where everything is generated on the fly, even the HTML. So e- this, this is actually being...
- 16:53
Oops. This is actually being generated by the, by the model on the fly. So the- it's a webpage with the HTML and the text and everything being generated on the fly.
- 17:02
So it looks like regular Wikipedia, and when you click on it, it's, it's... The, the latency is low enough that it can just fill in the page as if it was a real Wikipedia page.
- 17:10
Um, so that's kind of, uh, Wikipedia generated on the fly by a, just a very low latency model. We have a similar thing where we did it for Reddit.
- 17:18
So now all the responses to your posts will be by bots, if they weren't already. Um, [chuckles] and it's generating fake comments. Not the... So, uh, the Gemini diffusion model was not an image-generating model.
- 17:31
So this demo links in the... which was our start at state-of-the-art image generator model at the time, which I think was Juno. Was it? It was before NanoBanana. So it's the two of these models working together to fill in the webpage.
- 17:43
So it's the image generation model is a little slower, um, but you can see that it's like, you know, you can, you can invent any Reddit you want, sharks in this case, and it'll generate the page with the text.
- 17:55
The images follow a minute, a minute later, and then you can interact with this website as if it was a real website with, you know, real users and so on.
- 18:02
Just being, uh, the entire... All the comments, all the images, all the HTML, everything is being generated entirely on the fly here. This is being generated by the model.
- 18:10
Um, I love this one. This is my favorite one. This is an operating system also being entirely generated on the fly. So every click here is generating the next page of the operating system.
- 18:22
So it looks like a real operating system, but it's all being generated by the model on the fly, responding to every click. So every time you enter like the README, it generates the text, but it also generates, you know, the, the web pages you can do.
- 18:32
C- you know, go back to, to desktop and, and so on. I think this...
- 18:37
Yeah. Okay. Um, and then this is a, this is a demo from someone on Twitter who used the,
- 18:44
who used the Gemini Diffusion API, uh, or sorry, not the API, the web page, to do some v- vibe coding with his voice. So I, I really like this one.
- 18:55
Create a to-do app. Add 10 random to-dos.
- 19:05
Allow to-dos to have a completed state. Mark four random to-dos as completed.
- 19:14
Allow me to sort to-dos by name and by state.
- 19:23
All right, let's see if this works. Sort by name, sort by state. Testing. Enter. Was added to the bottom. We'll try deleting a few. Let's add one. Everything's working.
- 19:31
Please convert this to dark mode. And this was literally 15 seconds of work.
- 19:40
Okay. So that was, uh, yeah, it was someone outside of our team, so he, he could say that. Uh, yeah, the vibe coding by voice. But, but just in general, we think that, you know, low latency models can really unlock some new experiences, uh, for users and new products, and so we're excited to see what people will
- 19:54
do when the next generation comes out. Okay. And on that note, thank you very much. [audience applauding]
- 20:05
Questions? Yeah.
- 20:06
Yeah. Uh, did, did you use the same corpuses of text to pre-train, uh, Vision models as for the autoregressive models? Also, how, how does it work for, uh, reinforcement learning with human feedback?
- 20:19
So you use the same data too?
- 20:22
Yeah. Yeah. So yeah, we use all the same data. Yeah. I mean, the algorithms have to change a bit, but we use all the same data. Yeah.
- 20:28
And also maybe, uh, can you distill these models? Are there any techniques for doing that?
- 20:33
You can distill them. Yeah. Yeah. There... I'm not sure if there are any published ones.
- 20:37
Can you share more about it? [laughs]
- 20:38
I don't, I don't think so. Maybe there's some externally, but...
- 20:41
W- w- will we have in the near future like general available viability of this?
- 20:45
There's a, the, uh... Yeah. So we're gonna release something soon. Yeah.
- 20:51
Cool.
- 20:52
Yeah. Yeah.
- 20:53
Yeah. So as you scale the model like more parameters, more data, do you find like the adaptive computation increases like number of steps increases with like bigger models for even like easy prompts or?
- 21:04
Uh, yeah. So there's a... The bigger models tend to require less steps for the same output. So they kind of, you know, even if the model get, is getting bigger and the FLOPS per forward pass are getting bigger, they tend to reduce the, the forward passes they need.
- 21:22
So it's kind of a, it... You kind of have some sort of a diminishing cost of serving even the biggest models. Yeah.
- 21:29
So the next question, do you... How do you price the, uh, price? Does tokens, does it matter for the diffusion model?
- 21:38
Uh, I don't know. We haven't got to that yet. I'm a research scientist. [audience laughing] Uh,
- 21:43
yeah.
- 21:45
How do you define the size of the answer? Because for-
- 21:48
Yeah
- 21:48
... images, you expect that I have this image and I have the output and-
- 21:53
Yeah
- 21:53
... frame. How do you do that in-
- 21:55
Mm
- 21:56
... coding or text?
- 21:57
So, so there's a few different ways. The easiest way is to just fix some, some window length and then just iterate on that. So it's like autoregressive, blockwise autoregressive.
- 22:07
It's kind of the standard way to do it. But you can have like a, a head that'll predict the length of the response and stuff like that if you wanted.
- 22:13
Yeah.
- 22:13
Yeah. But, but you have to fix the outcome in order to produce a length model to, to work.
- 22:19
No, it still can generate unlimited text, uh, text, but it's just if you fix a window length, then it just does that window length autoregressively if it needs to generate, uh, many, many windows of text.
- 22:31
Yeah.
- 22:32
You might have already mentioned this, but when you gave that example of the denoising steps being different ranges for different, you know, uh, causing the problems-
- 22:39
Yeah
- 22:40
... could you ahead of time set like a, a limit on the denoising-
- 22:44
Yeah
- 22:44
... steps that you, that you require for a problem so you can-
- 22:46
Yeah
- 22:47
... almost just understand what your latency is gonna be ahead of time?
- 22:50
Mm-hmm. Yeah, yeah. These are all with a limit, but it just finishes with earlier than the limit.
- 22:55
All right. Okay.
- 22:55
Yeah.
- 22:56
If you, uh, if you had multiple windows, like you just said, going through, can, can they then go back and attempt to prove this window?
- 23:03
Yeah.
- 23:03
Or is that kind of as the window goes, it's-
- 23:05
It's, uh-
- 23:06
... set in stone
- 23:06
... yes. It's, it's... I mean, uh, you could potentially, but for, for us, we just set it in stone and-
- 23:10
Right
- 23:10
... continue.
- 23:11
And it just moves on.
- 23:11
Yeah. Yeah. Lots of questions here.
- 23:15
You mentioned it's, it can be slow on large, larger batches. Would it make sense to have like a hybrid between autoregressive diffusion where it's autoregressive of like a token at a time or like a chunk at a time?
- 23:24
Yeah, yeah. That's how it works. Yeah.
- 23:26
Oh, it sort of works.
- 23:26
Yeah.
- 23:27
Okay. Thanks.
- 23:28
So it starts to prefill with autoregr- uh, autoregressive style, and then it goes into diffusion mode or?
- 23:35
Oh, uh, no. So it's... Prefill is the, the same. It's just so you've got some context and you pre- you prefill. Uh, and then after that, the generation step is typically in blocks of some fixed size, like 512 or 1,000 or 32 or whatever you want.
- 23:49
And then that's autoregressive. Yeah.
- 23:53
I, I get that you're doing the denoising inside like a bi-directional embedding space. But how do you... What is the process of getting back the token IDs? Is it like, uh, Latent diffusion instead of-
- 24:06
So you can do that. Um, the easiest way to do it is to just have like, uh, logits at the top, just vanilla prediction head at the top.
- 24:16
Well, so th- this is like, this is all like discrete diffusion, right? So it's always tokens in, tokens out. Like if I show you the-
- 24:24
For every step.
- 24:24
For every step, yeah. So if I go back here. It's always like a discrete corruption process, and then you fill in a discrete token back in. So it's always...
- 24:33
So you're always in a discrete space. But you can do it in latent spaces, but-- and people have done that. But most of the, most of the text diffusion models and literature is discrete diffusion today.
- 24:45
Yeah.
- 24:46
Is there only a Gemini Diffusion?
- 24:48
There might be one day. [laughs] Yeah.
- 24:52
Yeah, maybe what, what's your outlook like? Do you think one of the architectures will win in the future or will they have different use cases or?
- 25:02
I think for now they have different use cases. So l- you know, if you think about what a low latency model provides that's like, that's worse throughput. So what's that trade-off?
- 25:12
Is on-device applications. So we, we are in a couple of on-device applications already, um, in within like the Alphabet ecosystem. So robotics, things like that, where you wanna run a, a model on the device itself.
- 25:25
So your phone or a robot or whatever. And then you... But you want it to be low latency and you're not batching with thousands of other queries like Gemini being served in the server side.
- 25:33
So you want the lowest latency model. Quality isn't really any-- They're the same quality basically. So then you may as well pick the low latency one 'cause you don't have the throughput concerns.
- 25:43
What do you mean like in the, in the future like can we get quality up to the par with, with the current like protein models?
- 25:51
I-- Quality isn't the concern. It's the throughput, uh, for serving in a big batch setting.
- 25:55
Okay. Got it.
- 25:57
Yeah. Yeah. Yeah.
- 25:58
So currently you are getting more or less the same quality of front-end models in your research?
- 26:02
Yeah. Yeah.
- 26:03
Okay. So what about reasoning tasks? Is it possible to differently verify what we want in some way?
- 26:09
Sorry?
- 26:10
What, what about reasoning tasks? Is it possible to differently verify-
- 26:14
Yeah. Yeah. Yeah. You can do RL. Yeah. You just need to change the algorithm. Yeah, but you can still do it. Yeah.
- 26:22
Yeah.
- 26:23
Is there much scope for combining diffusion and also aggressive text models?
- 26:28
Yeah, you can do that. Yeah.
- 26:31
Has your team done that? [laughs] [laughs]
- 26:34
Yeah. [laughs] [laughs] Yeah.
- 26:38
Can you like initialize with a smaller model so you don't have to start from noise, so you can have to do less forward passes?
- 26:43
Um, the, the problem with that is if you train it with noise, it expects noise. You'd have to train it with the small model as you like the output and that just adds complexity.
- 26:55
So we don't usually do that. Yeah.
- 26:59
Is there a paper like several versions back that you can actually go and implement and learn more about it?
- 27:05
Um, well, not from, not from our team, but yeah, there are-- there's a bunch of literature out there, yeah.
- 27:09
Okay.
- 27:10
Yeah.
- 27:11
Which are respectively composed.
- 27:13
Yeah. You get the idea, I think.
- 27:16
Yeah.
- 27:17
Yeah. Any other questions? There was a lot of questions there. Gotten through them all. That's good. Oh, one more. Okay.
- 27:25
All right. This is a random one. What happens if you ask it to generate noise?
- 27:29
Um, I don't know. [laughs] I don't think we've tried to do that. [laughs] [laughs]
- 27:37
Probably would work. It would probably do something.
- 27:39
Yeah.
- 27:41
Yeah.
- 27:43
Probably.
- 27:43
Cool. Okay. Thanks, everybody. [clapping] [outro jingle]