Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

Google Search, ChatGPT and Perplexity make the application of LLMs to search visible: you ask a question and receive an answer. But what happens when there is no explicit question—when a service must decide what you might want to watch next? Devansh Tandon opens with the proposition that recommendations could become an even larger consumer application of LLMs, precisely because their influence is easy to overlook.

At YouTube, that problem spans Home, Watch Next, Shorts and even personalized search results. Tandon says recommendations drive a large majority of YouTube watch time. Improving them can change the experience of an enormous audience without introducing a chat box or asking users to learn a new interface. The engineering path runs through adapting Gemini, building a compact language for videos, and turning that language into recommendation outputs.

The recommendation problem is a function from a user and their context to a set of suggested videos. Inputs can include demographics, age, gender and location; the last hundred videos watched; engagement depth; comments; and subscriptions. The current viewing situation matters alongside the longer history.

YouTube had already explored multi-headed rankers, embedding models, sequence-to-sequence models and transformers. The next question was how to build on Gemini rather than develop another recommendation model in isolation. The team started with a base Gemini checkpoint and adapted it into a unified YouTube-specific Large Recommender Model, or LRM.

That shared checkpoint becomes the starting point for smaller models aligned to particular tasks, such as retrieval and ranking, and particular recommendation surfaces. The distinction matters: retrieval supplies candidate videos, while ranking orders candidates for presentation. At the time of the talk, LRM retrieval had launched in production; ranking remained an area of experimentation.

A text model consumes tokens and predicts tokens. The corresponding recommendation interface would accept video tokens and emit video tokens representing good recommendations. That requires compression: even a million-token context window cannot economically hold rich representations of all the videos a recommender needs to reason about. Before settling on video tokens, the team tried generating search queries to retrieve videos and recommending videos directly; Tandon says those approaches were not good enough.

Semantic IDs, abbreviated SIDs, provide that compact representation. The pipeline collects a video's title, description, transcript, audio and frame-level information, combines those features into a multidimensional embedding, and quantizes the embedding with a residual-quantized variational autoencoder, or RQ-VAE. The result is a discrete representation that can participate in a token sequence. The companion paper, Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations, appeared as a preprint in 2023 and at RecSys in 2024. It studies ranking, so its encoder and evaluation details should not be assumed to describe the later Gemini retrieval system.

The useful property is shared semantic structure. Imagine the first part of an ID separating music, gaming and sports. Within sports, later parts distinguish individual sports. Two volleyball videos can share a prefix while retaining distinct complete identifiers. These topic names explain the organization; they are not literal text labels that must appear in each ID.

The change is from arbitrary identity to identity that carries useful similarity. Tandon describes semantic tokenization as already in production. This does not imply that every downstream use of hashing disappears; it changes what the video representation encodes.

Having a video vocabulary does not mean Gemini understands it. Continued pre-training teaches the existing checkpoint to work across English and the new video language. The training has two complementary parts: connect text to SIDs, then learn relationships among videos from sequences of watches.

The first part uses metadata tasks. Given the SID of a tennis-highlights video and a prompt asking for its title, the model learns to produce the title. Other prompts ask for its creator or topics. Repeated across videos, these tasks connect discrete video identifiers to the natural-language concepts already available in the base model.

The second part uses engagement history. A watch sequence such as A, B, C, D becomes a prediction task by masking a video and asking the model to recover it. The following Python constructs one such training example, using letters as readable stand-ins for video SIDs:

watch_sequence = ["A", "B", "C", "D"]
masked_index = 2

model_input = watch_sequence.copy()
target = model_input[masked_index]
model_input[masked_index] = "<MASK>"

training_example = {
    "input": model_input,
    "target": target,
}

print(training_example)
# {'input': ['A', 'B', '<MASK>', 'D'], 'target': 'C'}

The target is a video from an observed watch sequence. Learning to recover it teaches relationships grounded in what people watch together, extending the content relationships encoded by semantic IDs.

After these tasks, the checkpoint can connect video IDs to explanations in English. In the demonstration, a watch history contains examples associated with Wimbledon, the Spanish Grand Prix and pi. Prompts connect them to tennis fans, F1 fans and math fans. When asked to complete the explanation for a fourth video, the model connects it to technology fans because it concerns AI. Tandon says this works from the semantic-ID representation with little other video information supplied in the prompt. The model has learned enough of the video language to use it alongside English.

The first recommendation task built on this checkpoint was generative retrieval. Instead of asking the model to explain a video, construct a prompt describing a user and ask it to decode candidate videos as SIDs. The example is a user in the United States using Android, currently watching Olympics highlights. The prompt also includes fifty previous watches and information about how the user engaged with them. Demographics, the context video and watch history all contribute to the prediction.

The earlier system followed the immediate similarity between track-race videos. LRM found an additional connection between the viewing context and the user's history and demographics. Tandon reports particularly useful, differentiated recommendations on difficult tasks and for users about whom the system knows less. The example illustrates a different candidate set; it does not supply a measured quality uplift.

The experiments exposed a mismatch between learning efficiency and serving efficiency. Tandon describes LRM as quick to learn, efficient with training data and effective on difficult recommendation tasks. But generating recommendations at YouTube's scale was too expensive. Tandon reports more than 95% TPU serving-cost savings after the initial experiments, enabling production launch. He does not disclose the optimization techniques, measurement baseline or quality and latency constraints behind that reduction.

A second strategy changed when inference happened. The team kept the model and prompt structure but removed the personalized fields, turning the request into: given video A, which videos would make good next watches? The output could be stored in an offline recommendation table.

Unpersonalized recommenders normally lag personalized ones, but Tandon says the large starting checkpoint still produced differentiated candidates. The team could run inference ahead of time for the head of the video corpus—the videos accounting for a large share of watch time—and serve those candidates with a lookup.

The offline path gives up request-specific personalization in that candidate source in exchange for removing model inference from the serving path. It reuses the model's learned relationships while changing the cost of delivering its outputs.

A video recommender also faces a different kind of vocabulary growth. Tandon uses a dictionary analogy: roughly 100,000 English words, with about 1,000 additions per year, versus a YouTube library on the order of 20 billion videos with millions added daily. These are a scale analogy, not specifications of Gemini's tokenizer. The practical difference is a continually expanding set of items that the recommender must learn to represent and retrieve.

Freshness makes that expansion urgent. Tandon contrasts a model missing the 2023 word “rizz” with a recommender missing a newly released Taylor Swift music video. His illustrative claim that the former could still answer 99% of questions emphasizes tolerance for missing a word; the latter needs to make an important release recommendable within minutes or hours.

That requirement changes the training schedule. Tandon describes LRM continued pre-training on days-and-hours timescales, compared with a three-to-six-month cadence for conventional Gemini pre-training in his account. New content must become useful to the right viewers while it is still new.

Model size imposes another constraint. Tandon describes Gemini Pro as too expensive for the latency and scale requirements of serving this recommender to the audience he frames as billions of daily users. The team instead focused on Gemini Flash and even smaller checkpoints. The strongest available model is not automatically the model that fits a continuously operating recommendation surface.

The reusable procedure begins before personalization:

1. Tokenize the content. Extract rich features, build an embedding, and tokenize or quantize that representation. The result is a domain-specific language with compact units the model can consume and generate.
2. Adapt the LLM to that language. Build training tasks that connect English to the new tokens and teach relationships within the domain. The intended result is a bilingual model that can move between natural language and domain representations.
3. Specialize for the recommendation task. Construct prompts containing user demographics, activity and actions, then train models for particular tasks or surfaces.

Tandon calls this a compact account of roughly two years of work. Its ordering is consequential: the personalized prompt becomes useful after the content representation and the model's understanding of it are established.

The deployed experience described so far is mostly invisible augmentation: a feed improves, but the user cannot tell whether Gemini inference happened. A model that understands both natural language and recommendation tokens could expose a different interface. Tandon describes experiments in which users express goals in natural language, steer recommendations toward those goals, and receive explanations of why a candidate was recommended.

Such an interface begins to blur search and recommendations: an explicit request can guide a system that already understands viewing context and candidate content. Tandon then looks further ahead to recommending personalized versions of content and eventually generating content for one particular user. This is a future direction, beyond the recommendation augmentation and interface experiments described earlier.

The first audience question identifies a tension in that future: can a model learn the semantic-ID language without damaging its English? Early applications optimized primarily for recommendation quality, so the team heavily emphasized SID examples. Tandon says that with enough of this training, the model forgets how to speak English. He suggests it might still perform some reasoning in intermediate layers before producing semantic IDs, but offers that as a possibility rather than an established explanation.

The team was experimenting with mixture-of-experts approaches in which some experts retain text capability while others specialize in semantic IDs. Interactive recommendation experiences increase the importance of English inputs and outputs, motivating a shift back toward text in the training balance. A checkpoint that is excellent at producing recommendation IDs does not necessarily retain the language skills needed to discuss those recommendations with a user.

Asked about cold-starting domain-token embeddings, Tandon clarifies that semantic-ID training is entirely unsupervised. The quantizer discovers its own organization of the corpus. When the team inspects it, groupings such as sports, movies and entertainment appear even though those categories were not explicitly taught.

That organization gives a new video a place in a semantically meaningful space before extensive engagement history exists. Tandon reports better performance for videos uploaded in the last day or week, attributing it to improved understanding of fresh and tail content. No numerical improvement is disclosed in this answer.

The next question probes how visual information enters the representation. An audience member proposes sampling at 3–30 frames per second, making frame grids, and using SigLIP or SigLIP2. Tandon does not confirm that pipeline. He describes experiments selecting frames from key moments, including peaks in engagement of the kind highlighted in the YouTube player. Scale prevents sampling many frames, so the selection must be economical.

A follow-up asks whether this can preserve an important small object, such as a person in the distance. Tandon says all the video information is compressed into eight tokens. He cannot say exactly which details survive that compression, or whether the particular small object would be retained. Semantic usefulness at the recommendation level does not establish fidelity to every visual detail.

Another audience question asks whether pre-training includes search queries paired with subsequent watches, and whether users receive semantic IDs too. In the described system, only videos are tokenized, and training focuses on watch sequences rather than query-to-watch pairs. A user remains context supplied to the model, not another implemented SID type.

Tandon suggests that a user token could represent something like the last 500 watches. The team had experimented in that direction, but the work was less advanced. He also confirms that the recommendation checkpoint starts from an existing pre-trained Gemini model. An audience suggestion to connect video SIDs to Veo 3 receives a brief acknowledgment, not an account of an implemented integration.

The closing question asks both how much LRM improves on traditional recommendation systems and when to choose one approach over the other. Tandon says he cannot share code or metrics. He describes LRM as the biggest improvement in recommendation quality the team has seen in the last few years, but gives neither a numerical uplift nor an explicit model-selection rule. The disclosed result is a production retrieval system with a new representation and training approach; the magnitude of its advantage remains a qualitative assessment.