Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify

Matching a listener to something worth hearing becomes difficult when the choices span more than 100 million music tracks, plus videos, podcasts and audiobooks. Yves Raimond opens with the scale Spotify reported at the time of the recording: about 760 million monthly active users across 184 markets. The challenge is both the size of the catalog and the variety of intentions a listening session can contain.

Spotify’s first personalization mechanism was human curation: people assembled playlists around particular tastes. At the time of the talk, the service contained about 10 billion playlists. Recommendation systems then turned curation and other signals into suggestions that could reach individual listeners at scale, with Discover Weekly serving as the familiar example.

The next step, which Spotify calls generative personalization, adds an interactive layer to that matching process. A ranked list predicts what someone might want. A language-capable system can also consider whether the recommendation fits the current context, accept a correction, explain a choice, and generate parts of the experience around it. Raimond describes this as a move toward reasoning and steerability: the listener can tell the system what it missed rather than communicate only through subsequent plays and skips.

The product examples introduce progressively different kinds of control:

• Spotify DJ: A personalized music session can change direction while it is running. The listener taps a control and tells the algorithm what they want next.
• Prompted playlists: A request can be broad or detailed. One example asks for bands playing in San Francisco that night; another asks for music to accompany a run. The running example shapes the presentation as well as the selections, including an introduction and different phases of the run.

An editable taste profile addresses a more persistent problem: listening history can describe the device’s household rather than one person’s preferences. Raimond’s example is Disney music played by his children on shared devices. Those plays enter the history, but their presence does not mean he wants Disney recommendations for himself. The profile exposes what the algorithm has understood in natural language, giving him a place to explain who those plays belong to.

The correction changes the meaning of an existing signal. The listening events still happened; the user supplies the missing context that they reflect the children’s taste. Raimond describes the intended result as the algorithm taking that edit into account so it avoids recommending the music in the wrong context. The same editing channel can express a future goal—exploring a genre, learning about a topic or getting into a language—before listening history contains much evidence of that interest.

Personal podcasts extend generation to the content itself. The announced example is a daily brief about what is happening in the listener’s community, generated on a daily cadence. These examples occupy different product stages in the recording: the taste profile had launched in New Zealand, with more markets planned, while personal podcasts were described as coming soon.

A shared system called the Large Taste Model sits behind the applications Raimond has described. It combines historical interaction and content knowledge with prediction, reasoning and user prompts. At the time of the talk, Spotify reported daily interaction with the system by about one in four US Premium subscribers, along with gains on autoplay, podcast discovery and interaction with DJ messages. Those gains are described qualitatively here, so they do not establish a numerical comparison between the products.

Jacqueline Wood, a staff machine learning engineer at Spotify, then explains a core component: semantic IDs. Spotify starts with existing content embeddings, such as embeddings of podcast episodes. A quantization algorithm converts those representations into discrete tokens. The tokens become additional entries in an open-weight LLM’s vocabulary, allowing catalog entities and ordinary language to appear within the same model.

Consider the request for a podcast about morality. The prompt includes the natural-language request and the listener’s history represented as semantic IDs. The trained model responds with a semantic ID identifying a relevant podcast episode and a natural-language explanation of why it suits that listener. The catalog representation makes the recommendation addressable; the language representation lets the user express intent and receive an explanation.

How does a language request connect to a specific episode? The flow below shows the two kinds of input meeting inside one model and the two kinds of output leaving it. The listening history supplies personal context alongside the topic request; the response carries both a catalog identity and an explanation.

Adding tokens does not by itself teach the model what they mean or how to recommend them. NEO, Spotify’s four-stage training recipe, separates those jobs. The semantic foundation stage constructs meaningful semantic IDs and adds them to the LLM vocabulary. Domain grounding then learns mappings from semantic IDs to text, from text to semantic IDs, and combinations of the two, aligning the new token embeddings with the model’s existing language embedding space.

The important restriction is what can change during grounding. Spotify freezes the LLM backbone, including its original weights and embeddings, and trains only the new semantic-ID embeddings. The catalog vocabulary must learn to fit into an existing language system. This protects the pretrained model’s core language abilities from catastrophic forgetting while the new tokens acquire meaning.

Capability induction follows with multitask instruction tuning on Spotify tasks, including next-item recommendation and retrieval. At this point the recipe unfreezes the model and supports full-parameter or adapter-based fine-tuning. An optional fourth stage adds further post-training, such as reinforcement-learning fine-tuning. The sequence lets the model learn the catalog’s language before adapting its behavior across recommendation tasks.

When does the backbone become trainable? The diagram makes the ordering visible: catalog construction precedes a restricted embedding-learning stage, followed by task adaptation with the model unfrozen. The freeze applies to domain grounding, rather than to the entire training recipe.

Multitask training raises an obvious concern: could learning several jobs make the model worse at each one? Spotify compared the shared model with single-task variants and found that it matched or exceeded their performance across the tested tasks. Audiobooks were a particularly useful case. As a newer content type at Spotify, they benefited from learning shared with other catalog items and tasks, including podcast recommendations. Wood presents this as evidence that multitask learning can help cold-start entities.

The recipe ablations test different reasons for keeping the stages:

• Remove domain grounding: Skipping the frozen-backbone grounding stage degraded performance.
• Combine grounding and task learning: Folding grounding into multitask instruction tuning also degraded performance.
• Remove pretraining: Starting from a randomly initialized backbone produced the largest performance drop. The pretrained LLM contributes useful knowledge beyond the added catalog tokens.

Continued pretraining exposes a subtler failure. It caused only minimal degradation on the task-specific evaluation, yet the natural-language and world-knowledge capability results fell to essentially zero in the presented evaluation. Frozen-backbone grounding retained those capabilities while still learning semantic IDs. The practical consequence is that recommendation accuracy alone could conceal damage to the abilities needed to understand requests and explain results. The talk does not identify the language evaluation or its scoring scale, so the near-zero result should remain scoped to that experiment.

Wood reports that the findings held with both Qwen and Llama backbones. That comparison supports the staged training approach beyond one model family, while keeping the claim within the backbones Spotify tested.

At inference time, generating catalog tokens creates two different questions: does the output identify valid content, and is that content a good recommendation? Spotify tested beam search with and without constrained decoding. Without constraints, 98% of generated semantic IDs were valid. That measures identifier validity, not recommendation quality.

Constrained decoding adds some latency but can also enforce a desired content subset, such as recommending only new content. Separately, top-p sampling significantly hurt accuracy compared with beam search in Spotify’s tests. The team accepted beam search’s additional latency because its accuracy tradeoff worked better for the recommendation task.

NEO’s combination brings grounded catalog items, natural-language steering, search, recommendation and explanation into one system, with tool-free inference for these capabilities. Wood reports production use in podcast discovery, where the model helped listeners move beyond habitual patterns toward unfamiliar content. This gives the shared model a practical purpose beyond consolidating tasks: it can connect what a person already listens to with something they would not ordinarily find.

The ending returns to a problem that grows with the product’s capabilities: how do you judge the result? Traditional offline recommendation metrics can indicate whether a user interacted with content. They cannot fully answer whether a recommendation makes sense for that user, whether its explanation is accurate, or whether it follows the user’s intent. A system that produces explanations needs evaluation of those explanations as well as the selected items.

LLM judges can assess those qualities, but Spotify found that they need meaningful grounding to align with human preferences. Two examples supply different kinds of context:

• Listening-history profiles: For podcast recommendations, Spotify summarizes a user’s listening history into a textual profile and passes it to the judge. The reported alignment with human preferences was 75%.
• Behavior on similar queries: For search, the judge receives similar queries and the user’s past interactions with them. Spotify reported a 5% increase in alignment overall and a 91% increase for ambiguous queries. Past behavior is especially useful when the query alone leaves the intended meaning unclear.

The final example uses grounded judges to expand Cranfield-style evaluation collections. Candidates from multiple sources enter a shared pool, which a human would ordinarily rank. Human ranking is expensive, so Spotify invested in grounding an LLM judge and reported agreement of 0.87 with human system rankings. These evaluation figures describe separate tests: the talk does not specify their datasets or sample sizes, whether the 5% and 91% increases are relative changes or percentage-point gains, or which agreement metric produced 0.87. They should therefore retain their individual meanings rather than be compared as interchangeable scores.

The training and evaluation problems meet here. A recommender needs grounded catalog knowledge to turn a request into an item; a judge needs grounded listener context to assess whether that item fits. Teaching an LLM to speak Spotify consequently involves preserving its language abilities, giving it a catalog vocabulary, adapting it to recommendation tasks, and supplying enough context to evaluate the experience it produces.