AI Engineer World's Fair 2025
Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU
Read the talk
Teaching Gemini to Speak YouTube
YouTube’s Large Recommender Model turns videos into semantic tokens, learns from watch sequences, and generates recommendations under demanding freshness and serving constraints.
From a talk by Devansh Tandon
Before you start: Familiarity with LLM tokens, embeddings and the distinction between model training and inference will help; no prior knowledge of YouTube’s recommendation system is required.
The recommendation you never asked for
Google Search, ChatGPT and Perplexity make the application of LLMs to search visible: you ask a question and receive an answer. But what happens when there is no explicit question—when a service must decide what you might want to watch next? Devansh Tandon opens with the proposition that recommendations could become an even larger consumer application of LLMs, precisely because their influence is easy to overlook.
At YouTube, that problem spans Home, Watch Next, Shorts and even personalized search results. Tandon says recommendations drive a large majority of YouTube watch time. Improving them can change the experience of an enormous audience without introducing a chat box or asking users to learn a new interface. The engineering path runs through adapting Gemini, building a compact language for videos, and turning that language into recommendation outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From user context to a shared checkpoint
The recommendation problem is a function from a user and their context to a set of suggested videos. Inputs can include demographics, age, gender and location; the last hundred videos watched; engagement depth; comments; and subscriptions. The current viewing situation matters alongside the longer history.
YouTube had already explored multi-headed rankers, embedding models, sequence-to-sequence models and transformers. The next question was how to build on Gemini rather than develop another recommendation model in isolation. The team started with a base Gemini checkpoint and adapted it into a unified YouTube-specific Large Recommender Model, or LRM.
That shared checkpoint becomes the starting point for smaller models aligned to particular tasks, such as retrieval and ranking, and particular recommendation surfaces. The distinction matters: retrieval supplies candidate videos, while ranking orders candidates for presentation. At the time of the talk, LRM retrieval had launched in production; ranking remained an area of experimentation.
Suggest correction
Making videos into a language
A text model consumes tokens and predicts tokens. The corresponding recommendation interface would accept video tokens and emit video tokens representing good recommendations. That requires compression: even a million-token context window cannot economically hold rich representations of all the videos a recommender needs to reason about. Before settling on video tokens, the team tried generating search queries to retrieve videos and recommending videos directly; Tandon says those approaches were not good enough.
Semantic IDs, abbreviated SIDs, provide that compact representation. The pipeline collects a video's title, description, transcript, audio and frame-level information, combines those features into a multidimensional embedding, and quantizes the embedding with a residual-quantized variational autoencoder, or RQ-VAE. The result is a discrete representation that can participate in a token sequence. The companion paper, Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations, appeared as a preprint in 2023 and at RecSys in 2024. It studies ranking, so its encoder and evaluation details should not be assumed to describe the later Gemini retrieval system.
The useful property is shared semantic structure. Imagine the first part of an ID separating music, gaming and sports. Within sports, later parts distinguish individual sports. Two volleyball videos can share a prefix while retaining distinct complete identifiers. These topic names explain the organization; they are not literal text labels that must appear in each ID.
| Representation | Relationship between related videos |
|---|---|
| Arbitrary hashed video identity | Similar content need not have related IDs |
| Semantic ID | Similar content can share part of its representation |
The change is from arbitrary identity to identity that carries useful similarity. Tandon describes semantic tokenization as already in production. This does not imply that every downstream use of hashing disappears; it changes what the video representation encodes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connecting English, video IDs and watch sequences
Having a video vocabulary does not mean Gemini understands it. Continued pre-training teaches the existing checkpoint to work across English and the new video language. The training has two complementary parts: connect text to SIDs, then learn relationships among videos from sequences of watches.
The first part uses metadata tasks. Given the SID of a tennis-highlights video and a prompt asking for its title, the model learns to produce the title. Other prompts ask for its creator or topics. Repeated across videos, these tasks connect discrete video identifiers to the natural-language concepts already available in the base model.
The second part uses engagement history. A watch sequence such as A, B, C, D becomes a prediction task by masking a video and asking the model to recover it. The following Python constructs one such training example, using letters as readable stand-ins for video SIDs:
python
watch_sequence = ["A", "B", "C", "D"]
masked_index = 2
model_input = watch_sequence.copy()
target = model_input[masked_index]
model_input[masked_index] = "<MASK>"
training_example = {
"input": model_input,
"target": target,
}
print(training_example)
# {'input': ['A', 'B', '<MASK>', 'D'], 'target': 'C'}
The target is a video from an observed watch sequence. Learning to recover it teaches relationships grounded in what people watch together, extending the content relationships encoded by semantic IDs.
After these tasks, the checkpoint can connect video IDs to explanations in English. In the demonstration, a watch history contains examples associated with Wimbledon, the Spanish Grand Prix and pi. Prompts connect them to tennis fans, F1 fans and math fans. When asked to complete the explanation for a fourth video, the model connects it to technology fans because it concerns AI. Tandon says this works from the semantic-ID representation with little other video information supplied in the prompt. The model has learned enough of the video language to use it alongside English.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generating candidates for an Olympics viewer
The first recommendation task built on this checkpoint was generative retrieval. Instead of asking the model to explain a video, construct a prompt describing a user and ask it to decode candidate videos as SIDs. The example is a 24-year-old woman in the United States using Android, currently watching Olympics highlights. Her prompt also includes fifty previous watches and information about how she engaged with them. Demographics, the context video and watch history all contribute to the prediction.
| System in the example | Retrieved candidates |
|---|---|
| Production system before LRM | Other men's track races |
| LRM | Related women's races, using demographic and watch-history connections |
The earlier system followed the immediate similarity between track-race videos. LRM found an additional connection between the viewing context and the user's history and demographics. Tandon reports particularly useful, differentiated recommendations on difficult tasks and for users about whom the system knows less. The example illustrates a different candidate set; it does not supply a measured quality uplift.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paying for inference—or moving it offline
The experiments exposed a mismatch between learning efficiency and serving efficiency. Tandon describes LRM as quick to learn, efficient with training data and effective on difficult recommendation tasks. But generating recommendations at YouTube's scale was too expensive. Tandon reports more than 95% TPU serving-cost savings after the initial experiments, enabling production launch. He does not disclose the optimization techniques, measurement baseline or quality and latency constraints behind that reduction.
A second strategy changed when inference happened. The team kept the model and prompt structure but removed the personalized fields, turning the request into: given video A, which videos would make good next watches? The output could be stored in an offline recommendation table.
Unpersonalized recommenders normally lag personalized ones, but Tandon says the large starting checkpoint still produced differentiated candidates. The team could run inference ahead of time for the head of the video corpus—the videos accounting for a large share of watch time—and serve those candidates with a lookup.
| Serving path | Inference input | Request-time work |
|---|---|---|
| Personalized generation | User, history and context video | Generate candidate SIDs |
| Offline candidate table | Context video without personal fields | Look up precomputed candidates |
The offline path gives up request-specific personalization in that candidate source in exchange for removing model inference from the serving path. It reuses the model's learned relationships while changing the cost of delivering its outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A vocabulary that changes every day
A video recommender also faces a different kind of vocabulary growth. Tandon uses a dictionary analogy: roughly 100,000 English words, with about 1,000 additions per year, versus a YouTube library on the order of 20 billion videos with millions added daily. These are a scale analogy, not specifications of Gemini's tokenizer. The practical difference is a continually expanding set of items that the recommender must learn to represent and retrieve.
Freshness makes that expansion urgent. Tandon contrasts a model missing the 2023 word “rizz” with a recommender missing a newly released Taylor Swift music video. His illustrative claim that the former could still answer 99% of questions emphasizes tolerance for missing a word; the latter needs to make an important release recommendable within minutes or hours.
That requirement changes the training schedule. Tandon describes LRM continued pre-training on days-and-hours timescales, compared with a three-to-six-month cadence for conventional Gemini pre-training in his account. New content must become useful to the right viewers while it is still new.
Model size imposes another constraint. Tandon describes Gemini Pro as too expensive for the latency and scale requirements of serving this recommender to the audience he frames as billions of daily users. The team instead focused on Gemini Flash and even smaller checkpoints. The strongest available model is not automatically the model that fits a continuously operating recommendation surface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building the domain language first
The reusable procedure begins before personalization:
- Tokenize the content. Extract rich features, build an embedding, and tokenize or quantize that representation. The result is a domain-specific language with compact units the model can consume and generate.
- Adapt the LLM to that language. Build training tasks that connect English to the new tokens and teach relationships within the domain. The intended result is a bilingual model that can move between natural language and domain representations.
- Specialize for the recommendation task. Construct prompts containing user demographics, activity and actions, then train models for particular tasks or surfaces.
Tandon calls this a compact account of roughly two years of work. Its ordering is consequential: the personalized prompt becomes useful after the content representation and the model's understanding of it are established.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Letting users steer the recommender
The deployed experience described so far is mostly invisible augmentation: a feed improves, but the user cannot tell whether Gemini inference happened. A model that understands both natural language and recommendation tokens could expose a different interface. Tandon describes experiments in which users express goals in natural language, steer recommendations toward those goals, and receive explanations of why a candidate was recommended.
Such an interface begins to blur search and recommendations: an explicit request can guide a system that already understands viewing context and candidate content. Tandon then looks further ahead to recommending personalized versions of content and eventually generating content for one particular user. This is a future direction, beyond the recommendation augmentation and interface experiments described earlier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping the model bilingual
The first audience question identifies a tension in that future: can a model learn the semantic-ID language without damaging its English? Early applications optimized primarily for recommendation quality, so the team heavily emphasized SID examples. Tandon says that with enough of this training, the model forgets how to speak English. He suggests it might still perform some reasoning in intermediate layers before producing semantic IDs, but offers that as a possibility rather than an established explanation.
The team was experimenting with mixture-of-experts approaches in which some experts retain text capability while others specialize in semantic IDs. Interactive recommendation experiences increase the importance of English inputs and outputs, motivating a shift back toward text in the training balance. A checkpoint that is excellent at producing recommendation IDs does not necessarily retain the language skills needed to discuss those recommendations with a user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What semantic compression preserves
Asked about cold-starting domain-token embeddings, Tandon clarifies that semantic-ID training is entirely unsupervised. The quantizer discovers its own organization of the corpus. When the team inspects it, groupings such as sports, movies and entertainment appear even though those categories were not explicitly taught.
That organization gives a new video a place in a semantically meaningful space before extensive engagement history exists. Tandon reports better performance for videos uploaded in the last day or week, attributing it to improved understanding of fresh and tail content. No numerical improvement is disclosed in this answer.
The next question probes how visual information enters the representation. An audience member proposes sampling at 3–30 frames per second, making frame grids, and using SigLIP or SigLIP2. Tandon does not confirm that pipeline. He describes experiments selecting frames from key moments, including peaks in engagement of the kind highlighted in the YouTube player. Scale prevents sampling many frames, so the selection must be economical.
A follow-up asks whether this can preserve an important small object, such as a person in the distance. Tandon says all the video information is compressed into eight tokens. He cannot say exactly which details survive that compression, or whether the particular small object would be retained. Semantic usefulness at the recommendation level does not establish fidelity to every visual detail.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Video tokens today, possible user tokens later
Another audience question asks whether pre-training includes search queries paired with subsequent watches, and whether users receive semantic IDs too. In the described system, only videos are tokenized, and training focuses on watch sequences rather than query-to-watch pairs. A user remains context supplied to the model, not another implemented SID type.
Tandon suggests that a user token could represent something like the last 500 watches. The team had experimented in that direction, but the work was less advanced. He also confirms that the recommendation checkpoint starts from an existing pre-trained Gemini model. An audience suggestion to connect video SIDs to Veo 3 receives a brief acknowledgment, not an account of an implemented integration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How much better is it?
The closing question asks both how much LRM improves on traditional recommendation systems and when to choose one approach over the other. Tandon says he cannot share code or metrics. He describes LRM as the biggest improvement in recommendation quality the team has seen in the last few years, but gives neither a numerical uplift nor an explicit model-selection rule. The disclosed result is a production retrieval system with a new representation and training approach; the magnitude of its advantage remains a qualitative assessment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
YouTube research on replacing random video-ID representations with semantic codes to improve ranking generalization for new and long-tail videos.
Further reading
YouTube’s explanation of recommendation signals, satisfaction and responsible recommendations, published before the Gemini adaptation described here.
April 2025 figures for cumulative video uploads and average daily uploads, alongside anniversary announcements.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] There's a lot of attention in terms of how LLMs are gonna transform search.
- 0:19
Uh, Google Search is having a revolution. ChatGPT has a big chat interface. Perplexity is a product that a lot of people use. Um, but I think recommendations is, uh, probably a bigger problem that is under-hyped because it's kind of transparent to the user.
- 0:37
Um, and I think the application of LLMs to recommendations is gonna be a bigger consumer application than search.
- 0:44
Um, so in terms of my talk, I just wanna introduce the problem of YouTube recommendations and then talk about how we've built large recommender models. We're adapting Gemini for YouTube, how we build semantic ID, and how we're using that, and then end with this recipe of how you might use an LLM to make a recommendation system.
- 1:03
Um, to start, why this is important, um, who here watches YouTube every day?
- 1:10
It's one of the biggest consumer apps in the world, um, and a large majority of the watch time on YouTube is driven by the recommendation system. Uh, and we serve recommendations across Home, Watch Next, we have a big Shorts product, and even a lot of our search results are personalized in some way.
- 1:29
Uh, and so if you think about consumer applications of LLMs, I think in terms of consumer engagement and impact, recommendations is gonna be a much bigger, uh, application than, than search is, and this is true of any consumer app with a billion DAU.
- 1:48
Um, the way I think about the recommendation problem is you're trying to learn this function of you get a user and their context as input, and you're trying to give them a bunch of recommendations.
- 2:01
Um, at YouTube, we have a bunch of user information like their demographics, their age, their gender, where they're located. We have a lot of context about them, what are the last hundred videos they watched?
- 2:13
How deeply did they engage with them? What did they comment on? Who are they subscribed to? And we use all of that to make, uh, video recommendations.
- 2:22
We've tried a lot of different modeling techniques here, multi-headed rankers, embedding models, sequence-to-sequence, transformers. There's a, there's a long history. Um, and about two years ago, we started thinking, how can we rethink this recommendation system on top of Gemini, which has been making incredible progress in modeling?
- 2:43
How can we adapt that for YouTube? And so we've built this system, which we call LRM, Large Recommender Model, uh, where we adapt Gemini for recommendations. So we start with this base Gemini checkpoint, um, and then we are adapting it for YouTube recommendations, teaching it a lot of information about YouTube to get this kind of
- 3:08
unified YouTube-specific checkpoint of Gemini, which we call LRM.
- 3:14
Then we can align it for different recommendation-related tasks like retrieval and ranking, um, and basically make a small custom version of this model for all of the major recommendation surfaces.
- 3:27
Um, and so this is a model that we have launched in production at YouTube for a while in terms of the retrieval system, and we're experimenting a lot on the ranking side.
- 3:36
So I wanna start with just kind of explaining how we built this YouTube and Gemini model, and then we'll talk about how we use it for retrieval.
- 3:44
The first step of this kind of a model is you have to develop a way to tokenize videos. So when you, uh...
- 3:53
In, in terms of an LLM, when you give it an input, it to- it tokenizes that text and then is predicting the next text token. Uh, the ideal product we wanted to make was we wanna give this model an input of a number of video tokens and then just get video tokens out that would be good recommendations.
- 4:12
Um, we had to build this because even with a million tokens of context, when you wanna reason over many videos, you have to compress that video representation in some way.
- 4:22
Um, and before we kind of settled on this approach, we tried a bunch of other things like, uh, predicting search queries and retrieving videos through that or trying to just, uh, recommend videos directly, and those solutions were just not good enough.
- 4:34
And so we built Semantic ID, which we actually wrote a paper about last year, uh, and it was presented at REXES. The way that this Semantic ID works is you take a video, um, you extract a number of features out of it, like the title, description, transcript, even the audio and video frame level data.
- 4:53
You put all of that into a multidimensional embedding, um, and then you quantize it using RQ-VAE to give every video, uh, a token. Um, we've written a pretty detailed paper about this if people are interested, but at a high level, the way I think about this is we're making the atomic units for a new language of YouTube
- 5:14
videos. Um, once we have these tokens, you can imagine the whole corpus of billions of videos on YouTube gets organized around these semantically meaningful tokens. And so you could imagine the first token representing topics like music, gaming, sports.
- 5:30
Within sports, you would have different, different sports, and then you can, uh, get to volleyball. And so these two volleyball videos would share some tokens in the prefix but also then have a unique identifier.
- 5:42
Um, and this, this I think in itself is, is an interesting milestone to move away from hash-based tokenization into a semantically meaningful one, um, and we use this in production at YouTube.
- 5:54
Uh, what we then tried to do is this process of what we call continued pre-training, where we're trying to take this model and have it understand both English and this new YouTube language.
- 6:06
And we do this in, in two big steps. One is around linking text and SID. Um, and then the second step is around having it understand sequences of watches and be able to reason across this video space.
- 6:19
And so some of the example training tasks that we're teaching this model, you have this video, it, it's a tennis highlights video which has some semantic ID, and you can prompt it and say, "Hey, this video has title XYZ," and the model starts to learn to output the title.
- 6:35
Um, you could imagine a very similar thing where you could say, Has creator or has topics, and so on. And so you're basically trying to connect text and this video token.
- 6:47
Then what we can try to do is we have a corpus of all the YouTube engagement data, all the paths that users took through YouTube when they watch videos together.
- 6:57
Um, and you can prompt the model with things like, a user has watched the following videos A, B, C, D, and you mask some of those videos. And the model starts to learn to predict those masks, and now it's starting to understand what are videos that are watched together and make relationships between videos on the basis of
- 7:14
user engagement, right? Um, after a, a bunch of pre-training tasks like this, we get this really interesting model that can reason across English and YouTube videos. And so this is an example from a user's watch history, um, and we find that this model can now reason across these videos.
- 7:35
So you could prompt it with things like, "Hey, video one is interesting to tennis fans because it's about Wimbledon. Video two is interesting for F1 because it's about the Spanish Grand Prix.
- 7:45
Video three is interesting to math fans because it's about pi." And then you prompt video four is gonna be interesting too, and the model starts to be able to understand that it's interesting technology fans because it's about AI.
- 7:56
And this is just based on the semantic ID, uh, definition of a video. It doesn't really have a lot of other information, uh, to go off of. So I think this in itself is a very interesting checkpoint that is starting to reason across English and YouTube.
- 8:16
Once we have this model, we think about how we can use this for different video recommendation tasks at YouTube. And the first one that we focused on is generative retrieval.
- 8:27
And so here you could just construct a prompt for every user and see what this model recommends. And so in this example, you have a user, they would be a [REDACTED:age] [REDACTED:gender] in the US on Android.
- 8:40
They're watching this highlight video from the Olympics, um, and they have some watch history of, you know, fifty videos they've watched in the past, how they engaged with it.
- 8:51
And you can just construct a prompt like we have on the right with this user demographic information, the context video, and have the model decode some video recommendations as SIDs.
- 9:04
We find that this gives really interesting, unique recommendations, especially for our hardest recommendation tasks. So in this example, when you're watching, uh, this highlight from the Olympics, the production system before LRM would give you other men's track races.
- 9:24
Um, now with this new model, it's able to find this unique connection between the user demographic and their past watch history and find related women's, uh, rec- races that we weren't able to recommend in the past.
- 9:39
And so we find that especially for users where we don't know as much about them, we get very interesting and unique recommendations out of this strategy.
- 9:49
Um, and so we've experimented with this and launched it in a few places at YouTube. The big findings from this is that LRM is a very powerful model, but it's really expensive to serve.
- 10:02
It, it learns very quickly, it's very training data efficient, um, and it handles our toughest recs tasks. But the biggest limitation was that the serving costs are too high, especially for the scale that YouTube operates at with billions of users.
- 10:16
And so after we got our first experiments working, we spent a lot of time just reducing the TPU serving cost and, you know, we got ninety-five percent plus cost savings to be able to actually launch this in production.
- 10:29
Um, one other strategy that we used, which I think is kind of interesting, is we tried to turn this into an offline problem, where it's the same prompt and the same model, we just removed the personal-- personalized aspects of this prompt.
- 10:45
And we wanted to build just an offline recommendations table where if you're watching video A, what are the candid videos that would be good to watch next? Um, and normally these unpersonalized recommendation models just don't hold a candle to a personalized recommender.
- 11:03
But because this LRM is trained from a really big checkpoint, it actually gives us some differentiated recommendations. Um, and so in the YouTube context, like we can take our corpus of billions of videos, look at the head, which represent a lot of the watch time, do offline inference, um, make this offline recs table, and then we can
- 11:24
just do a simple lookup to serve some recommendations. Um, and so this was kind of a, a complete way around our serving problems.
- 11:33
Um, I wanna talk a bit about the challenges for YouTube, and I think in some ways, making an LLM-based recommendation system is harder than training an LLM. Um, one of the big differences is the vocabulary and size of the corpus, right?
- 11:49
So for Gemini, if you're training an English LLM, your vocabulary is about a hundred thousand words in the Oxford Dictionary, and they add about a thousand words every year.
- 11:59
Um, at YouTube, if you imagine the library of YouTube, it has billions of videos. We have twenty billion videos on YouTube, um, with millions added every day. Um,
- 12:12
and the freshness of videos is really important, much more so than LLMs. So if you think about a new word that's added to the English dictionary, word of 2023 was rizz.
- 12:22
If your model Gemini doesn't know about rizz, it can still answer 99% of questions that people would have. Maybe it misses some jokes, maybe it misses some pop culture references.
- 12:33
But in the world of YouTube, if Taylor Swift drops a new music video, you have to be able to recommend it within the next minutes or hours, otherwise a lot of users are gonna be upset.
- 12:44
So even within this large corpus, you have to very quickly understand what are the videos that are important and start recommending them to the right user. Um, and so what we do with this LRM recommender is we have to continuously pre-train it on the order of days and hours, which is very different than classical LLM pre-training like
- 13:06
Gemini, which happens maybe, like, once in three to six months. Um, and so in that way, it's a much harder problem. Uh, and then the last part is scale.
- 13:16
Um, we have great models in Gemini. Gemini Pro is incredible, but there's no way that you can serve that to billions of daily active users. Um, and so for YouTube, we had to focus on the smaller, more efficient models like Flash and, and even smaller checkpoints than that, um, just to be able to hit the latency and
- 13:35
scale requirements that we have. Um, so I wa- kinda wanna summarize the journey that we've been on YouTube in this, what I think of as a LLM and REXUS recipe that you can maybe adapt to your own application, and there's three major steps to this, right?
- 13:50
The first is you wanna find a way to tokenize your content. Um, just like LLMs tokenize text, you wanna find-- you wanna make some essence of your content into an atomic token.
- 14:02
Uh, one way to do that, which we've done, is you find some rich representation, a bunch of features, build an embedding, and then find a way to tokenize or quantize it.
- 14:11
And the outcome of this is, like, you're making your own domain-specific language.
- 14:16
The second step is you then wanna adapt the LLM and basically make links between English and your domain language, uh, and find training tasks that help you reason across English and these new tokens you've built.
- 14:31
And so the outcome after this step, in my mind, is it's a bilingual LLM that can speak English and natural language, but it can also speak your domain-specific language.
- 14:40
Um, and then once you have this, you can do the third step of prompting it with user information, where you can just construct personalized prompts with user demographic, user activity, different actions, um, and then train task-specific or surface-specific models, and you have a generative recommendation system on top of an LLM.
- 15:01
And there-- this is, like, a tweet-sized summary of maybe two years of work. Um,
- 15:08
maybe the last thing that I wanna talk about is kind of where I see this going, um, and some possible future directions for LLM and REXUS. I think the stage that we're at right now is that LLMs are just augmenting recommendations.
- 15:22
They bring these magical recommendation experiences. They enhance the quality, but they're largely invisible to users. Like, your YouTube feed just got better, but you don't really know whether a Gemini inference happened or not.
- 15:36
Um, this is why I think the LLM application of REXUS is very under-hyped because users don't directly know what's happening. Um, I think we're close to a world, and we're experimenting with this, if you have, like we talked about, a bilingual LLM across English and recommendations, users can then talk to it in natural language.
- 15:56
And I think you're gonna start to see experiences where users can steer recommendations to their own goals. The recommender can explain why a candidate was recommended to a user, um, and users can start to align it towards their own goals expressed in natural language.
- 16:13
Um, and I think also the lines between search and recommendations start to blur in this world. Um, and then maybe a hint of the future is I think you're gonna see recommendation and generative content start to come together in the future, where we're gonna be recommending a personalized version of a piece of content.
- 16:34
And in the future, instead of recommending content, we may even start creating it, and you can get to really interesting N of one content that's generated for the user.
- 16:43
Um, I think we're a bit away from this, but it's gonna come sooner than you expect with all the advances happening in AI.
- 16:53
Um, so yeah. Thank you. [audience applauding] I'll take any questions.
- 17:01
Thank you, Devansh. Um, we have time for a few questions.
- 17:05
Hi, great talk. Um, uh, one question on generally how you balance the learning of the semantic ID embeddings within the model versus keeping the general language capability not damaged by learning through, for example, a tokenized user history, which is a very second language, very different from English.
- 17:27
Any, uh, high-level takeaway that you can share?
- 17:31
That's a super interesting question. Um, we've struggled with this a lot. Um, in terms of some of our early applications, we mostly cared just about recommendation quality, in which case we over-indexed on speaking the semantic ID language.
- 17:46
Mm-hmm.
- 17:46
And as you over-train on more and more of those examples, actually the model forgets to speak English. Maybe it's reasoning in some intermediate layers-
- 17:54
Mm-hmm
- 17:55
... which finally end up in semantic ID language. Um, we are trying a bunch of things like, you know, with mixture of experts. Maybe we can have a few experts that retain the text capability-
- 18:06
Mm-hmm
- 18:07
... while other experts focus on the semantic ID capability. And so-
- 18:14
It, it's, it's a balance, and I think we're gonna shift more towards text as we try to build these interactive experiences where, uh, text input from user is gonna become more important.
- 18:23
Thank you.
- 18:26
So during this process, did you learn any, uh, any good suggestions for cold-starting embeddings on these domain-specific, uh, tokens?
- 18:37
Yeah. So the semantic I- one thing is semantic ID training process is entirely unsupervised.
- 18:42
Okay.
- 18:42
We're not telling... Like, it's making its own quantization of the video corpus.
- 18:48
Mm.
- 18:48
When you sample to see what the model is doing, we find that it's learning concepts like sports versus movies and entertainment. But we didn't actually try to teach that explicitly-
- 18:59
Mm
- 18:59
... which I think is very interesting. I think the sidekin aspect is because of semantic ID, we can warm start into a semantically meaningful space.
- 19:07
Yeah.
- 19:07
And what we find is performance for videos that were uploaded in the last day-
- 19:12
Mm-hmm
- 19:12
... or the last week, uh, gets much better because we're better understanding this fresh and tail content.
- 19:19
Got it. Thank you.
- 19:22
Hey. Quick question. So when you said you extract frames as part of making the semantic ID, are you just running a video at, let's say, three to 30 FPS, uh, making a grid of them, running SigLIP or SigLIP2 and inserting that?
- 19:38
We're just trying to sample video frames. Um,
- 19:43
we- we've tried a few different approaches where, like, maybe we try to sample from, like, key moments in the video. We actually have the engagement data. If you've seen in the YouTube player, uh, it can highlight what are the places where people had the most engagement, so we try to sample from there.
- 19:59
Um, you know, given the scale, we can't sample a lot of video frames, so we try to intelligently select it, but we do have video frames, and over time, I think we'll get more.
- 20:09
In this way of selecting it, are you able to highlight important things that are based on small objects in a video pretty well?
- 20:19
Let's say there's a person in the distance that's of attention of this video.
- 20:24
Hard to say 'cause, like, at the end, all of this video information gets compressed into eight tokens. So it's probably learning something, but it's hard to know exactly, you know, what it picked up from that video frame.
- 20:40
Uh, yeah, so it, it's unclear.
- 20:44
Thank you.
- 20:45
Yeah.
- 20:46
So, yeah, it was a pretty good talk. Uh, I have a question regarding pre-training. Okay. So, uh, did you also feed in a user query and what they watched also as a pre-training data?
- 21:00
If yes, then did you also use semantic ID for user as well in pre-training or, or just semantic ID is only for, for the videos? Yeah.
- 21:11
Yeah, so in this case, we have only tokenized videos. Um, and we focused more on sequences of watches rather than search query to what watch, uh, originated from that search query.
- 21:27
You could imagine some parallel work where you try to tokenize users and build some kind of user token that represents, like, the last 500 watches that they have had and so on.
- 21:38
Um, we've experimented with some stuff there. I think it's less far along. Um, but yeah, I think it's a very interesting, like, research direction to do.
- 21:46
So, so the pre-training was done on top of existing Gemini pre-trained model, right?
- 21:53
Yeah, we basically take a Gemini checkpoint and then adapt it for this YouTube purpose and get this, like, YouTube and Gemini LRM checkpoint.
- 22:03
Okay. Yeah, so last, it would be cool to see semantic ID of videos to VO3, you know? [laughs] That'll be...
- 22:10
Yeah.
- 22:11
Hey, uh, I'm kind of curious, how much, uh, improvement do we see compared to the non-LLM or more traditional recommendation system? And when should we use a more traditional one, and when should we use LLM-based recommendation system?
- 22:25
Yeah. I, I can't really share metrics. Like I, I was... I can share everything except code and metrics-
- 22:31
Uh-huh
- 22:31
... you know? [laughs] Um, and so we've given you as much conceptual steps of what we did. Maybe what I'll say is, I think it's been the biggest improvement to recommendation quality we've seen-
- 22:41
Mm
- 22:41
... in the last few years, so I do think it's quite significant. [upbeat music]