# Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify Yves Raimond, Spotify · Jacqueline Wood, Spotify AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/2LRIAfng7eA-teaching-llms-speak-spotify) · [Video](https://www.youtube.com/watch?v=2LRIAfng7eA) ## Teaching LLMs to Speak Spotify Yves Raimond and Jacqueline Wood explain how Spotify connects language models to its catalog, preserves their language abilities during training, and evaluates recommendations that users can steer and question. ### From assembling playlists to shaping an experience [0:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=12s) Matching a listener to something worth hearing becomes difficult when the choices span more than 100 million music tracks, plus videos, podcasts and audiobooks. Yves Raimond opens with the scale Spotify reported at the time of the recording: about 760 million monthly active users across 184 markets. The challenge is both the size of the catalog and the variety of intentions a listening session can contain. ![Source frame: From assembling playlists to shaping an experience](https://ai.engineer/api/talk-essay-media/cf8a3f91b42060f3a91e8abf960c37784c9f90d137346417f2b8a30910eeca39) Source frame: From assembling playlists to shaping an experience Spotify’s first personalization mechanism was human curation: people assembled playlists around particular tastes. At the time of the talk, the service contained about 10 billion playlists. Recommendation systems then turned curation and other signals into suggestions that could reach individual listeners at scale, with Discover Weekly serving as the familiar example. The next step, which Spotify calls generative personalization, adds an interactive layer to that matching process. A ranked list predicts what someone might want. A language-capable system can also consider whether the recommendation fits the current context, accept a correction, explain a choice, and generate parts of the experience around it. Raimond describes this as a move toward reasoning and steerability: the listener can tell the system what it missed rather than communicate only through subsequent plays and skips. ### Give the listener a way to correct the model [4:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=245s) The product examples introduce progressively different kinds of control: - **Spotify DJ:** A personalized music session can change direction while it is running. The listener taps a control and tells the algorithm what they want next. - **Prompted playlists:** A request can be broad or detailed. One example asks for bands playing in San Francisco that night; another asks for music to accompany a run. The running example shapes the presentation as well as the selections, including an introduction and different phases of the run. ![Source frame: Give the listener a way to correct the model](https://ai.engineer/api/talk-essay-media/ffb55b59238e8a33a0240231bea20bb9915f7c509c5ed099cf98d6b035ca5a87) Source frame: Give the listener a way to correct the model An editable taste profile addresses a more persistent problem: listening history can describe the device’s household rather than one person’s preferences. Raimond’s example is Disney music played by his children on shared devices. Those plays enter the history, but their presence does not mean he wants Disney recommendations for himself. The profile exposes what the algorithm has understood in natural language, giving him a place to explain who those plays belong to. The correction changes the meaning of an existing signal. The listening events still happened; the user supplies the missing context that they reflect the children’s taste. Raimond describes the intended result as the algorithm taking that edit into account so it avoids recommending the music in the wrong context. The same editing channel can express a future goal—exploring a genre, learning about a topic or getting into a language—before listening history contains much evidence of that interest. Personal podcasts extend generation to the content itself. The announced example is a daily brief about what is happening in the listener’s community, generated on a daily cadence. These examples occupy different product stages in the recording: the taste profile had launched in New Zealand, with more markets planned, while personal podcasts were described as coming soon. ### Teach the model a vocabulary for Spotify’s catalog [6:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=411s) A shared system called the **Large Taste Model** sits behind the applications Raimond has described. It combines historical interaction and content knowledge with prediction, reasoning and user prompts. At the time of the talk, Spotify reported daily interaction with the system by about one in four US Premium subscribers, along with gains on autoplay, podcast discovery and interaction with DJ messages. Those gains are described qualitatively here, so they do not establish a numerical comparison between the products. ![Source frame: Teach the model a vocabulary for Spotify’s catalog](https://ai.engineer/api/talk-essay-media/049256dcc6ee16f21613577a6eaad963c33c703730af9ba5d1b3a8f3a417c29d) Source frame: Teach the model a vocabulary for Spotify’s catalog Jacqueline Wood, a staff machine learning engineer at Spotify, then explains a core component: semantic IDs. Spotify starts with existing content embeddings, such as embeddings of podcast episodes. A quantization algorithm converts those representations into discrete tokens. The tokens become additional entries in an open-weight LLM’s vocabulary, allowing catalog entities and ordinary language to appear within the same model. Consider the request for a podcast about morality. The prompt includes the natural-language request and the listener’s history represented as semantic IDs. The trained model responds with a semantic ID identifying a relevant podcast episode and a natural-language explanation of why it suits that listener. The catalog representation makes the recommendation addressable; the language representation lets the user express intent and receive an explanation. How does a language request connect to a specific episode? The flow below shows the two kinds of input meeting inside one model and the two kinds of output leaving it. The listening history supplies personal context alongside the topic request; the response carries both a catalog identity and an explanation. A podcast request in two vocabularies Natural-language intent and semantic-ID history jointly produce an episode identifier and a personalized explanation. - Request — A podcast about morality - Listening history — Previous listening represented as semantic IDs - Catalog-aware LLM — Understands natural language and catalog tokens - Episode semantic ID — Identifies the recommended podcast episode - Explanation — Why the recommendation fits this listener - Request → Catalog-aware LLM (Natural-language prompt) - Listening history → Catalog-aware LLM (Personal context) - Catalog-aware LLM → Episode semantic ID (Catalog output) - Catalog-aware LLM → Explanation (Language output) ### NEO separates learning the catalog from learning the tasks [9:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=573s) Adding tokens does not by itself teach the model what they mean or how to recommend them. **NEO**, Spotify’s four-stage training recipe, separates those jobs. The semantic foundation stage constructs meaningful semantic IDs and adds them to the LLM vocabulary. Domain grounding then learns mappings from semantic IDs to text, from text to semantic IDs, and combinations of the two, aligning the new token embeddings with the model’s existing language embedding space. ![Source frame: NEO separates learning the catalog from learning the tasks](https://ai.engineer/api/talk-essay-media/7e64d2f54da0346859f391b2c1267230f3a922eb0e766d995a1aad4f336ee10c) Source frame: NEO separates learning the catalog from learning the tasks The important restriction is what can change during grounding. Spotify freezes the LLM backbone, including its original weights and embeddings, and trains only the new semantic-ID embeddings. The catalog vocabulary must learn to fit into an existing language system. This protects the pretrained model’s core language abilities from catastrophic forgetting while the new tokens acquire meaning. Capability induction follows with multitask instruction tuning on Spotify tasks, including next-item recommendation and retrieval. At this point the recipe unfreezes the model and supports full-parameter or adapter-based fine-tuning. An optional fourth stage adds further post-training, such as reinforcement-learning fine-tuning. The sequence lets the model learn the catalog’s language before adapting its behavior across recommendation tasks. When does the backbone become trainable? The diagram makes the ordering visible: catalog construction precedes a restricted embedding-learning stage, followed by task adaptation with the model unfrozen. The freeze applies to domain grounding, rather than to the entire training recipe. NEO’s training sequence The new catalog embeddings learn against a frozen language model before the model adapts to Spotify’s tasks. - 1. Semantic foundation — Construct semantic IDs and extend the vocabulary - 2. Domain grounding — Learn ID ↔ text mappings; train new embeddings while original weights and embeddings stay frozen - 3. Capability induction — Unfreeze the model for multitask instruction tuning - 4. Optional post-training — Further adaptation, such as reinforcement-learning fine-tuning - 1. Semantic foundation → 2. Domain grounding (Ground the added tokens) - 2. Domain grounding → 3. Capability induction (Then learn Spotify tasks) - 3. Capability induction → 4. Optional post-training (Optional) ### The ablations explain why the stages stay separate [11:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=691s) Multitask training raises an obvious concern: could learning several jobs make the model worse at each one? Spotify compared the shared model with single-task variants and found that it matched or exceeded their performance across the tested tasks. Audiobooks were a particularly useful case. As a newer content type at Spotify, they benefited from learning shared with other catalog items and tasks, including podcast recommendations. Wood presents this as evidence that multitask learning can help cold-start entities. ![Source frame: The ablations explain why the stages stay separate](https://ai.engineer/api/talk-essay-media/350884ded72647e8df9552559bec280b6b0ef0654056e9bdc43e72d65f0298e3) Source frame: The ablations explain why the stages stay separate The recipe ablations test different reasons for keeping the stages: - **Remove domain grounding:** Skipping the frozen-backbone grounding stage degraded performance. - **Combine grounding and task learning:** Folding grounding into multitask instruction tuning also degraded performance. - **Remove pretraining:** Starting from a randomly initialized backbone produced the largest performance drop. The pretrained LLM contributes useful knowledge beyond the added catalog tokens. Continued pretraining exposes a subtler failure. It caused only minimal degradation on the task-specific evaluation, yet the natural-language and world-knowledge capability results fell to essentially zero in the presented evaluation. Frozen-backbone grounding retained those capabilities while still learning semantic IDs. The practical consequence is that recommendation accuracy alone could conceal damage to the abilities needed to understand requests and explain results. The talk does not identify the language evaluation or its scoring scale, so the near-zero result should remain scoped to that experiment. Wood reports that the findings held with both **Qwen** and **Llama** backbones. That comparison supports the staged training approach beyond one model family, while keeping the claim within the backbones Spotify tested. ### Choose decoding for useful recommendations [14:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=843s) At inference time, generating catalog tokens creates two different questions: does the output identify valid content, and is that content a good recommendation? Spotify tested beam search with and without constrained decoding. Without constraints, 98% of generated semantic IDs were valid. That measures identifier validity, not recommendation quality. ![Source frame: Choose decoding for useful recommendations](https://ai.engineer/api/talk-essay-media/ebdde19fd8dbef8c577d1974f6304cdfd510fe44eafea0e152db95155c3e0aa1) Source frame: Choose decoding for useful recommendations Constrained decoding adds some latency but can also enforce a desired content subset, such as recommending only new content. Separately, top-p sampling significantly hurt accuracy compared with beam search in Spotify’s tests. The team accepted beam search’s additional latency because its accuracy tradeoff worked better for the recommendation task. NEO’s combination brings grounded catalog items, natural-language steering, search, recommendation and explanation into one system, with tool-free inference for these capabilities. Wood reports production use in podcast discovery, where the model helped listeners move beyond habitual patterns toward unfamiliar content. This gives the shared model a practical purpose beyond consolidating tasks: it can connect what a person already listens to with something they would not ordinarily find. ### A judge needs context about the listener too [16:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=970s) The ending returns to a problem that grows with the product’s capabilities: how do you judge the result? Traditional offline recommendation metrics can indicate whether a user interacted with content. They cannot fully answer whether a recommendation makes sense for that user, whether its explanation is accurate, or whether it follows the user’s intent. A system that produces explanations needs evaluation of those explanations as well as the selected items. ![Source frame: A judge needs context about the listener too](https://ai.engineer/api/talk-essay-media/0e82f4d2f1e953e88b1cea27ad6c5b4f39ade70f8e811b9b1ff03288b1c3cb4e) Source frame: A judge needs context about the listener too LLM judges can assess those qualities, but Spotify found that they need meaningful grounding to align with human preferences. Two examples supply different kinds of context: - **Listening-history profiles:** For podcast recommendations, Spotify summarizes a user’s listening history into a textual profile and passes it to the judge. The reported alignment with human preferences was 75%. - **Behavior on similar queries:** For search, the judge receives similar queries and the user’s past interactions with them. Spotify reported a 5% increase in alignment overall and a 91% increase for ambiguous queries. Past behavior is especially useful when the query alone leaves the intended meaning unclear. The final example uses grounded judges to expand Cranfield-style evaluation collections. Candidates from multiple sources enter a shared pool, which a human would ordinarily rank. Human ranking is expensive, so Spotify invested in grounding an LLM judge and reported agreement of 0.87 with human system rankings. These evaluation figures describe separate tests: the talk does not specify their datasets or sample sizes, whether the 5% and 91% increases are relative changes or percentage-point gains, or which agreement metric produced 0.87. They should therefore retain their individual meanings rather than be compared as interchangeable scores. The training and evaluation problems meet here. A recommender needs grounded catalog knowledge to turn a request into an item; a judge needs grounded listener context to assess whether that item fits. Teaching an LLM to speak Spotify consequently involves preserving its language abilities, giving it a catalog vocabulary, adapting it to recommendation tasks, and supplying enough context to evaluate the experience it produces. ## Key takeaways - Natural-language controls let listeners correct the interpretation of listening history and express interests that their past behavior does not yet capture. - Semantic IDs connect catalog entities to the LLM’s vocabulary, allowing one response to contain both a recommendation identifier and a natural-language explanation. - NEO grounds new catalog embeddings against a frozen backbone before multitask tuning. Spotify’s ablations show why task performance and retained language capabilities need separate attention. - Decoding choices serve different goals: constraints can restrict eligible content, while Spotify favored beam search over top-p sampling for recommendation accuracy despite added latency. - Evaluation must cover intent, fit and explanation accuracy. Grounding LLM judges in listener profiles and past behavior makes those judgments better informed, especially for ambiguous requests. ## Slides ### Personalization at Spotify Spotify’s matching problem spans 761M+ monthly active users and a catalog with 100M+ music tracks, alongside audiobooks, podcasts and video podcasts. [0:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=49s) [Spotify scale figures for users, markets, music tracks, audiobooks, podcasts, and video podcasts.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b7f4a4d320e0a9695464ce6ed646a749af26845bbaa984690e34276b6256469.webp) Personalization at Spotify — connecting people to content at scale 761M+ Monthly active users 184 Markets 100M+ Music tracks 700K+ Audiobooks 7M+ Podcast titles 530K+ Video podcasts ### Three Eras of Personalization Spotify moved from user-curated playlists to recommendations built from those signals. Generative personalization aims to shape experiences interactively around each user. [1:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=95s) [Three columns describe curation, recommendation, and generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/e64d132247efcea3bca7fdd22317e15a7141b6e175b3f9c82001a42badbb0a47.webp) Three Eras of Personalization Curation Users organizing tastes into over 10B playlists. Recommendation Turning those signals into personalized surfaces at scale. Generation Experiences that are interactively and dynamically shaped around each user. ### Generative Personalization Generative personalization adds reasoning and user control to recommendations. The slide illustrates an explanation for a mix of 32 saved but rarely replayed songs, moving from hip hop to spring 2023 house. [2:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=176s) [Three shifts toward reasoning, transparent and steerable personalization, and generating and explaining content.](https://assets.ai.engineer/sites/aiecode2025/sha256/9dc771e3b3433b194a5571c7505876d720b7585a586ebe9f1b2df3dcb23f76c1.webp) Generative Personalization • From personalization as guessing, to personalization as reasoning • From black-box algorithms to transparent and steerable personalization, where the user is fully in control • From personalization that can only recommend, to personalization that can generate and explain Clustering tracks... Analyzing genres This week’s mix surfaces 32 songs you saved but rarely replayed. We’re starting off your musical time journey in your hip hop era and ends in the House infused spring of 2023. ### Prompted Playlists Two prompted playlists show different levels of detail: one selects recent bands playing in San Francisco on June 30; the other organizes a running workout into warm-up and steady-run sections. [4:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=276s) [Two Spotify phone interfaces show a concert playlist being assembled and a running workout playlist.](https://assets.ai.engineer/sites/aiecode2025/sha256/7883ca55a04c045d4d892e8817a5475d6e42dbc5daa908e7f32c161cc1e591bc.webp) Prompted Playlists SF June 30 Concert Prompted by Yves A playlist of bands playing in San Francisco on June 30, tailored to your preferences. Selecting recent bands Performing in last 2 years Running Workout Warm up (5m) – Find the rhythm Folded — Kehlani — 130 bpm — 5A — 3:37 Ai preto — L7NNON — 130 bpm — 1A — 3:37 Steady Run (45m) • Steady 8 min/mile pace Sticky — Ravyn Lenae — 130 bpm — 9A — 3:37 Greedy — Tate McRae — 130 bpm — [illegible] — 3:37 Kirari — 130 bpm Interface text: 10:24 Notes Edit prompt Add Name Only A Broken Heart — Tom Petty Home Search Your Library Create 9:41 Notes Run Mix Edit prompt Auto From The Start — Laufey Home Search Your Library Create ### Prompted Playlists: Workout Detail A concert playlist gathers San Francisco show schedules, while a running playlist organizes songs into a 5-minute warm-up and a 45-minute steady run at an 8 min/mile pace. [4:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=296s) [A close-up of a running playlist shows warm-up and steady-run sections alongside the concert playlist interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/16d679f18278de0454e8b284929c3888266b5f2cba68850c424b8e81d9646975.webp) Prompted Playlists SF June 30 Concert Prompted by Yves A playlist of bands playing in San Francisco on June 30, tailored to your preferences. Gathering concert schedules Bands’ event dates and venues Warm up (5m) – Find the rhythm Folded — Kehlani — 130 bpm — 5A — 3:37 Ai preto — L7NNON — 130 bpm — 1A — 3:37 Steady Run (45m) • Steady 8 min/mile pace Sticky — Ravyn Lenae — 130 bpm — 9A — 3:37 Interface text: 10:24 Notes Edit prompt Add Name Only A Broken Heart — Tom Petty Home Search Your Library Create Notes Run Mix Edit prompt Auto ### Personal Podcasts (Coming Soon!) The upcoming personal podcast example turns a prompt into a daily brief about Hudson Valley events, concerts, weather, and tech headlines. [6:24](https://www.youtube.com/watch?v=2LRIAfng7eA&t=384s) [A phone interface shows a prompt for a personalized daily audio brief.](https://assets.ai.engineer/sites/aiecode2025/sha256/3627a4171c4731cb90cd459efe398f966acdeeb7ea761e587b1ba26f2bf37b94.webp) Personal Podcasts (Coming Soon!) Generate a personal podcast I want a short daily brief to start my day. Consider that I live in the Hudson Valley, and recommend relevant local events, concerts that match my taste, and the daily weather. Also highlight key tech headlines so I can prepare for the day ahead. Interface text: Cancel Upload PDF Ideas Next ### Semantic IDs Make the Model Catalog-Aware Spotify quantizes content embeddings into semantic ID tokens and fine-tunes an LLM to use them alongside language. In the example, listening-history IDs and a question produce a recommended ID and description. [8:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=497s) [Content embeddings are quantized into discrete tokens used by a fine-tuned language model.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d7cdce004ef4c92b9f8a9c6e5a641af15a7ab2f8d1f0f5a210dd1ca70008356.webp) Semantic IDs Make the Model Catalog-Aware Content → Embedding → Vector Quantizer → Discrete Tokens Given my listening history Semantic ID 1, Semantic ID 2, Semantic ID 3, could you recommend a podcast on latest development in moral theories? Fine-Tuned LLM Here is Semantic ID 4, a light hearted take on morality. ### How Do We Train the Model? NEO adds semantic ID tokens, aligns them with language while the LLM is frozen, then unfreezes it for multitask instruction tuning. A fourth, task-specific post-training stage was not studied in the paper. [9:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=570s) [NEO training stages, example tasks, and four model capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/292b231111bebd6fda7ad0478e221bcc77ecf38d47203f6b6c589f83180201ae.webp) How Do We Train the Model? Stage 1: Semantic Foundation Text Embedding STAGE 1: Broad Mystery Children STAGE K: Fine category Harry Potter Sherlock Holmes Final Semantic ID: 12 – 257 – 678 – 1021 Stage 2: Domain Grounding LLM Text Tokens SemID Tokens Stage 3: Capability induction LLM Text Tokens SemID Tokens Stage 4: Task-specific post-training LLM Text Tokens SemID Tokens (not studied in this paper) Describe a topic discussed in the episode → Galapagos Islands unique species and adaptations Recommend an episode for the following topic: Galapagos Islands unique species and adaptations → What is the item type of → artist Next-Item Recommendation Predicts the next interacted item User Understanding Generate a natural language description of the user Text-Based Retrieval Natural language query to SID Grounded Mixed Generation Generate an item and justification NEO: framework to adapt an open-weight LLM into a single, end-to-end catalog-grounded generator A Unified Language Model for Large Scale Search, Recommendation, and Reasoning - KDD 2026 ### Breaking Down Model Performance Multitask NEO generally matches or beats single-task models, especially on audiobooks. Random initialization causes the largest ablation drop; top-p sampling trades more accuracy for lower latency than unconstrained beam search. [11:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=688s) [Tables compare multitask performance, training ablations, and accuracy versus latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/2dca591a38eabad3a313a215bc579162ebc395495a628ac45dd15c0f49edba6f.webp) Breaking Down Model Performance Multi-Task Performance Method | HR@K↑ k = 10 | HR@K↑ k = 30 | NDCG@K↑ k = 10 | NDCG@K↑ k = 30 Episode Recommendation Baseline | – | – | – | – NEO - mono | +57% | +41% | +80% | +60% NEO - multi | +58% | +40% | +80% | +59% Show Recommendations Baseline | – | – | – | – NEO - mono | +20% | +2% | +46% | +30% NEO - multi | +24% | +2% | +58% | +39% Audiobook Recommendation Baseline | – | – | – | – NEO - mono | +36% | +6% | +73% | +52% NEO - multi | +46% | +14% | +97% | +66% Text-based retrieval Baseline | 0.45 | 0.51 | 0.37 | 0.38 NEO - mono | +45% | +36% | +62% | +58% NEO - multi | +47% | +36% | +62% | +58% Positive transfer from joint training Ablation on Training Recipe Method | HR@K↑ k = 10 | HR@K↑ k = 30 | NDCG@K↑ k = 10 | NDCG@K↑ k = 30 NEO (Baseline) | - | - | - | - A - w/o Domain Grounding | -6% | -7% | -8% | -8% B - Everything in two-stages | -8% | -7% | -10% | -10% C - Random init | -44% | -44% | -44% | -43% D - Continuos pretraining | -3% | -3% | -2% | -3% Recipe and backbone matters Accuracy vs Latency Trade-Off Inference strategy | Δ HR@K k = 10 | Δ HR@K k = 30 | Δ latency k = 10 NEO (Baseline) | - | - | - A - Beam Search w/o constrains | -1.80% | -1.93% | -2.92% B - Top-p sampling | -21.97% | -32.41% | -7.91% Beam search is more impactful than constrained decoding for this task ### NEO Capability Comparison In this comparison, NEO alone combines catalog grounding, language control, search, recommendation and explanation with low-latency, tool-free inference at industrial scale. [14:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=899s) [Table compares system capabilities and highlights NEO's combined capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/38b678de4b3f99f945c30c98314a89ade44957ca3ef006a7632f4021d41b6c23.webp) Capability ↓ / System → Columns: Generative SID Retrieval [43, 44] | Tool-based LLMs [6, 8, 32, 57] | PLUM [13] | TalkPlay [5] | NEO (this work) Discrete item pointers: ✓ | ✗ | ✓ | ✓ | ✓ Mixed text + item output: ✗ | ▲ | ✗ | ✓ | ✓ Mixed text + item input: ▲ | ✓ | ✓ | ✓ | ✓ User behaviour in input: ✓ | ▲ | ✓ | ▲ | ✓ Language-steerable behavior: ✗ | ✓ | ▲ | ✓ | ✓ Multiple item types: ▲ | ▲ | ✗ | ✗ | ✓ Search + Rec + Explanation: ✗ | ▲ | ✗ | ✗ | ✓ Tool-calling-free inference: ✓ | ✗ | ✓ | ▲ | ✓ Low-latency: ✓ | ✗ | ✓ | ✓ | ✓ Industrial-scale (>10M items): ✓ | ✗ | ✓ | ✗ | ✓ Legend: ✓ explicitly supported and demonstrated; ▲ partially supported / requires extra components; ✗ not supported / not demonstrated / not studied. NEO is the first single model to combine grounded catalog items, natural-language control, search + recommendation + explanation, and low-latency tool-free inference at industrial scale. ### Powering Podcast Discovery Today GLIDE’s production prompt combines user context and recent episode IDs. Spotify reports +5.4% non-habitual podcast streaming and +14.3% new show discovery. [15:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=939s) [GLIDE prompt structure and reported increases of 5.4% in non-habitual podcast streaming and 14.3% in new show discovery.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb9e92b6526200cf3937d74aa972c6aa35b7ce4be4285ea1791089fe7322c57c.webp) Powering Podcast Discovery Today [System Instruction] Define the role (Recommender) and output format (Semantic IDs). [User Context] ⊕ [Interaction History] [Task Instruction] "Recommend a {{ familiarity_mode }} episode aligned with user interests." +5.4% non-habitual podcast streaming +14.3% new show discovery GLIDE: a production generative recommender for podcast discovery Deploying Semantic ID-based Generative Retrieval for Large-Scale Podcast Discovery at Spotify - KDD 2026 ### Grounding is What Makes LLM-as-Judge Eval Work Past behavior improves alignment by 5% overall and 91% relative on ambiguous cases. Listening profiles yield 75% episode-level alignment with human judgments; grounded judges reach 0.87 agreement with human system rankings. [16:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=968s) [Textual profiles, behavioral signals and Cranfield-style collections support grounded evaluation, with results of 5%, 91%, 75% and 0.87 shown.](https://assets.ai.engineer/sites/aiecode2025/sha256/db799c9d3813f2228794b6abfa5dea2f4cbc79f18ec894eac65f7b6f4ee72a8a.webp) Grounding is What Makes LLM-as-Judge Eval Work Textual User Profiles Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge — RecSys 2025 Behavioral Signals As It Was: Aligning LLM Search Evaluation with Historical User Preferences — SIGIR 2026 Cranfield-Style Collections Cranfield-style Evaluation of Recommender Systems with LLM-based Judges — Under Review +5% behavior-grounded aligns better with user preferences +91% relative improvement on ambiguous cases 75% episode-level alignment with human judgments 0.87 agreement with human system rankings ### Thank You [18:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1132s) [Spotify R&D closing slide reading Thank You.](https://assets.ai.engineer/sites/aiecode2025/sha256/92365dff4fce4bfdfa30b11c4ac8451f0cf1e6614871d20977c27f65ab1da6e2.webp) Thank You! R&D ## Full transcript [0:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1s) [music] [0:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=12s) Welcome everyone. Um really glad and [0:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=15s) thank you Deven for inviting us as well [0:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=17s) to talk today. Uh so um uh Jackie and I [0:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=21s) are going to present today about making [0:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=23s) LLMs speak Spotify and how we turned our [0:27](https://www.youtube.com/watch?v=2LRIAfng7eA&t=27s) recommendation system on its head to be [0:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=29s) LLM native. Uh just a few quick words [0:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=32s) about myself. Uh I joined Spotify a year [0:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=35s) ago. Before that I was at Google working [0:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=37s) on personalization for Google search and [0:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=39s) prior to that I was uh working on [0:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=41s) personalization at Netflix. [0:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=44s) Um all right so let's dive in. Um [0:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=50s) first a couple of numbers to describe [0:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=51s) the scale of the problem that we have to [0:53](https://www.youtube.com/watch?v=2LRIAfng7eA&t=53s) solve. Um Spotify has about I was a [0:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=57s) little bit sad not to see it on the DAU [0:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=58s) by chart because it should be up there [1:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=60s) but it has about 760 million uh active [1:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=64s) uh users monthly uh in about 184 [1:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=68s) markets. But one thing that makes uh the [1:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=70s) Spotify personalization problem [1:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=71s) particularly challenging is the size of [1:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=74s) its catalog. Uh so of course you know [1:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=76s) Spotify has basically all music ever [1:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=78s) published a bit more than 100 million uh [1:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=81s) music tracks but it also has a range of [1:24](https://www.youtube.com/watch?v=2LRIAfng7eA&t=84s) videos, podcasts and audio books as [1:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=86s) well. So the matching problem uh is uh [1:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=89s) actually surprisingly uh complicated. [1:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=92s) So today what we're going to talk about [1:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=94s) Jackie and I is basically the extent of [1:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=96s) this matching problem and how we uh how [1:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=98s) we are solving it. I'm going to talk [1:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=100s) about a little bit of history, the new [1:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=102s) phase that we're entering and then [1:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=103s) Jackie is going to help us uh go into [1:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=106s) the guts of the modes as well to [1:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=107s) understand how those things are trained [1:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=109s) as well. Uh so a little bit on history. [1:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=112s) Uh initially Spotify personalization was [1:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=115s) really built around curation. So uh [1:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=117s) people basically manually assembling [1:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=119s) playlists that target specific taste. [2:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=122s) And that's still a very common uh use [2:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=124s) case on Spotify. Right now we have about [2:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=126s) 10 billion playlists with many many uh [2:09](https://www.youtube.com/watch?v=2LRIAfng7eA&t=129s) created every every hour. [2:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=132s) Uh then you know Spotify moved into [2:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=134s) taking these creation signals and order [2:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=136s) signals and moving into recommendations. [2:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=139s) So basically being able to turn these uh [2:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=142s) creation signals to something that can [2:24](https://www.youtube.com/watch?v=2LRIAfng7eA&t=144s) be applied at scale. Uh and a great [2:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=146s) example of that would be discover weekly [2:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=148s) for example that was launched in 2014 as [2:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=150s) one of the one of the early like [2:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=153s) recommendation use case on on Spotify. [2:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=156s) But the phase that we entering now which [2:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=158s) uh uh which we're going to talk about in [2:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=160s) more details is something that we call [2:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=161s) generative personalization where we are [2:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=164s) not only solving a matching problem from [2:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=166s) the user to the content. We're also [2:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=168s) solving the ability to generate an [2:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=170s) experience that is interactively and [2:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=172s) dynamically shaped around each user. [2:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=178s) And that transition from recommendations [3:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=181s) to generative personalization involves a [3:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=183s) couple of different uh a couple of [3:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=185s) different big shifts. One is move from [3:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=188s) moving moving from personalization as [3:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=191s) guessing uh where basically you have a [3:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=193s) ranking algorithms that spits out a [3:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=195s) brand ranked list of entities in your [3:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=197s) catalog to personalization as reasoning [3:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=199s) that can introspect these results and [3:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=202s) really try to understand whether that's [3:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=203s) indeed the right match for this user in [3:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=205s) this particular context. [3:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=208s) The other aspect as well is moving from [3:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=211s) blackbox algorithms. uh it's very [3:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=213s) difficult to fully introspect a [3:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=215s) multi-stage ranking system for example [3:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=218s) to transparent and steerable [3:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=220s) personalization where the user is always [3:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=221s) fully in control. So uh basically giving [3:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=225s) the ability to these mods to speak and [3:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=228s) understand English. [3:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=230s) The other thing that these these systems [3:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=232s) can do is uh uh not stopping just at [3:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=235s) recommending but also generation of [3:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=238s) experiences and explaining as well. uh [4:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=240s) and we're going to show a couple of [4:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=242s) examples of that. [4:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=245s) So one example that uh we launched a [4:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=247s) couple of years ago uh so pretty early [4:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=250s) in that journey was the Spotify DJ. Uh [4:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=252s) so the Spotify DJ is something that you [4:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=254s) can spin up that will start playing [4:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=256s) music for you of course personalized [4:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=258s) uh and but one interesting thing about [4:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=261s) it is that since last year you can tap [4:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=263s) that button on the bottom uh right and [4:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=266s) steer it in whatever direction you see [4:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=268s) fit. So at any point you can chime in [4:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=270s) and let the algorithm know what you want [4:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=272s) and it's going to steer the session in [4:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=274s) the in in that in that direction. [4:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=277s) Uh another example of what we call [4:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=279s) generative personalization is uh [4:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=281s) showcased in a prompted playlist here [4:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=283s) that Deanch showed a little bit earlier [4:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=286s) as well. Uh and here you can you [4:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=289s) basically have full unfettered access to [4:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=291s) the recommendation algorithm that [4:53](https://www.youtube.com/watch?v=2LRIAfng7eA&t=293s) Spotify has. uh and you can uh prompt it [4:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=296s) with very high level prompts or very [4:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=299s) detailed prompts. On the left hand side [5:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=301s) here, I have a prompt that tells me uh [5:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=303s) create me a playlist of bands that are [5:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=304s) playing in San Francisco tonight. Uh so [5:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=307s) turns out there's a bunch of good shows [5:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=308s) if you're excited to check them out. On [5:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=311s) the right hand side, you see a prompt [5:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=312s) that is asking for a playlist to [5:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=314s) accompany me on my run. And what's [5:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=316s) interesting with the right hand side as [5:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=317s) well is that you will see that the [5:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=319s) experience itself gets dynamically [5:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=321s) shaped as a function of the request on [5:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=322s) the user to be able to introduce itself [5:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=325s) and different phases in my run as well. [5:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=329s) Uh, another one that I'm really excited [5:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=331s) about, we launched it in New Zealand a [5:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=333s) couple of months ago and uh, it's coming [5:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=335s) soon in more markets is something called [5:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=337s) the taste profile. And that basically [5:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=339s) gives you the ability to introspect in [5:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=342s) natural language what the Spotify [5:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=344s) algorithm has understood about you in a [5:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=347s) way that you can edit and refine. Uh, so [5:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=349s) if you see something that's missing or [5:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=351s) something that's wrong. Uh so for [5:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=352s) example one of my edit is that all [5:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=355s) Disney music are my kids because we have [5:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=357s) a bunch of shared devices at home but [5:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=359s) please don't recommend that to me that's [6:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=360s) not my taste please. Uh and the [6:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=363s) algorithm would then take that into [6:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=364s) account uh making sure that we never [6:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=366s) recommend this this in the wrong [6:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=368s) context. Uh and similarly you can also [6:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=371s) use the taste profile to share some more [6:12](https://www.youtube.com/watch?v=2LRIAfng7eA&t=372s) aspiration uh goals as well. Uh so [6:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=375s) getting into a new genre, getting into a [6:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=377s) new topic, uh learning about a new [6:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=379s) language for example, all of these [6:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=380s) things can be can be done. [6:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=383s) Uh another one that's coming uh soon uh [6:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=385s) which we uh announced very recently is [6:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=388s) something called personal podcast where [6:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=390s) the the the generative personalization [6:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=392s) system doesn't stop at just recommending [6:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=393s) and ascending experiences but also [6:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=395s) generating content as well. In this [6:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=397s) particular example, I'm generating a [6:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=399s) daily brief that's uh that's generated [6:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=401s) on a cadence daily. Uh and I'm going to [6:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=405s) uh let it uh tell me about what's [6:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=408s) happening in my community. [6:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=411s) All right. So now to go into the guts of [6:53](https://www.youtube.com/watch?v=2LRIAfng7eA&t=413s) it. So there's one big system that [6:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=415s) controls like all of these different [6:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=417s) applications I mentioned and we call [6:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=418s) that internally some the large taste [7:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=420s) model. We're not great at naming these [7:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=422s) internal things. Um so it has a couple [7:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=425s) of properties. One is that it [7:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=426s) understands every historical interaction [7:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=428s) piece of content on Spotify. Uh it [7:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=431s) combines prediction and reasoning to the [7:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=433s) point that I mentioned earlier. So not [7:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=435s) only guessing but also reasoning layered [7:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=437s) on top and it gives users the ability to [7:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=439s) shape and generate experiences in real [7:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=441s) time. So it's fully steable and [7:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=442s) promptable by uh by users. Um and uh as [7:27](https://www.youtube.com/watch?v=2LRIAfng7eA&t=447s) of today about one in four US premium [7:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=449s) subscribers interact with that system on [7:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=451s) a daily basis as well. So that's pretty [7:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=452s) exciting. Um, one thing that's exciting [7:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=455s) as well to uh is that deploying this [7:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=457s) system across existing recommendation [7:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=460s) surfaces as well also led to some gains. [7:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=462s) Uh, we saw gains on autoplay, we saw [7:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=465s) gains on podcast discoveries. Uh, we saw [7:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=467s) gains on users interacting with DJ [7:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=469s) messages as well. So, we saw pretty [7:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=471s) pretty sizable gains across the board as [7:53](https://www.youtube.com/watch?v=2LRIAfng7eA&t=473s) well by deploying the system. On that [7:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=476s) note, I'm going to hand it over to [7:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=477s) Jackie to talk to us about what's one of [8:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=480s) the core component that underpins this [8:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=481s) whole system. [8:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=483s) Hi everyone, I'm Jackie or Jacqueline, a [8:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=486s) staff machine learning engineer at [8:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=488s) Spotify. So let's dive a little bit [8:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=491s) deeper and talk about how these models [8:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=493s) are actually trained at least at [8:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=494s) Spotify. So semantic IDs were presented [8:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=498s) in the previous talk, but that is how we [8:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=501s) are embedding these openweight LLMs with [8:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=505s) knowledge of Spotify's catalog. They are [8:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=508s) created by taking existing content [8:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=510s) embeddings such as podcast episode [8:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=512s) embeddings and applying a quantization [8:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=516s) algorithm to convert them to a set of [8:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=520s) discrete tokens. We then take a [8:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=523s) openweight LLM such as Quen and we [8:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=527s) modify its vocabulary to add these new [8:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=530s) special tokens and then we fine-tune the [8:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=532s) model to be able to understand both [8:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=535s) natural language as well as these new [8:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=538s) special tokens semantic IDs that [9:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=540s) represent Spotify catalog entities. So [9:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=543s) we can power experiences such as this [9:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=546s) where the user can ask in natural [9:08](https://www.youtube.com/watch?v=2LRIAfng7eA&t=548s) language for a podcast on morality. And [9:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=551s) that is passed to the prompt along with [9:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=554s) their listening history represented as [9:17](https://www.youtube.com/watch?v=2LRIAfng7eA&t=557s) semantic IDs. And the model responds [9:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=560s) both with a relevant semantic ID podcast [9:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=563s) episode [9:24](https://www.youtube.com/watch?v=2LRIAfng7eA&t=564s) as well as a natural language [9:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=566s) description of why they recommended that [9:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=568s) to this user. [9:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=573s) So how is this model actually trained? [9:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=575s) Um we published a paper linked here [9:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=579s) uh describing our training paradigm [9:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=582s) called NEO which consists of four [9:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=585s) distinct stages. The first I already [9:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=588s) covered which is the semantic foundation [9:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=592s) stage where we construct meaningful [9:55](https://www.youtube.com/watch?v=2LRIAfng7eA&t=595s) semantic ID tokens and then add them to [9:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=598s) an openway LLM's vocabulary. [10:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=601s) The second stage we call domain [10:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=604s) grounding in which we align these new [10:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=607s) semantic ID token embeddings in the [10:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=610s) original language embedding space. [10:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=613s) We do this by learning a birectional [10:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=616s) mapping between semantic ids to text, [10:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=619s) text to semantic ids and any [10:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=622s) combination. [10:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=626s) And we actually freeze the LLM backbone [10:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=629s) at this stage and only train the new [10:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=633s) semantic ID embeddings. So the original [10:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=636s) model weights and embeddings are frozen [10:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=640s) and we just learn those new semantic ID [10:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=643s) tokens and this helps us to mitigate [10:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=645s) catastrophic forgetting of the [10:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=647s) pre-trained LLM's core language [10:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=650s) abilities. [10:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=652s) The third stage we call capability [10:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=654s) induction which is multitask instruction [10:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=657s) tuning on tasks that Spotify cares about [11:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=660s) such as the ones shown here. next item [11:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=663s) recommendation retrieval etc. [11:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=667s) This is done by unfreezing the whole [11:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=671s) model all of its weights and embeddings [11:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=674s) and running either full parameter [11:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=676s) fine-tuning or Laura fine-tuning on the [11:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=679s) multiple Spotify tasks and then there is [11:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=682s) an optional fourth stage to do post- [11:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=685s) training such as RL fine-tuning etc. [11:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=691s) So, how much of a difference does this [11:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=693s) four-stage training paradigm actually [11:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=695s) make? I'll dive into a few of the [11:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=697s) abilations we've done to investigate [11:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=700s) this. First, we assessed whether [11:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=704s) multitask training is actually hurting [11:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=706s) performance by comparing the multitask [11:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=708s) model against single task variance. And [11:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=712s) we consistently saw that across our [11:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=714s) tasks, the multitask model can match or [11:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=717s) actually beat the single task [12:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=720s) performance indicating that there is [12:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=722s) some positive cross-learning happening [12:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=724s) across the tasks. This is particularly [12:09](https://www.youtube.com/watch?v=2LRIAfng7eA&t=729s) noticeable for audiobook [12:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=730s) recommendations. If you look here, which [12:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=733s) is a new newer content type at Spotify, [12:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=738s) demonstrating that these multitask [12:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=740s) models can help with cold start entities [12:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=742s) by learning from other items in the [12:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=746s) catalog such as podcast recommendations, [12:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=748s) how to make meaningful audiobook [12:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=750s) recommendations. [12:33](https://www.youtube.com/watch?v=2LRIAfng7eA&t=753s) Then we did some abilations on the [12:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=754s) actual training recipe. We evaluated [12:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=757s) both dropping the frozen backbone domain [12:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=761s) grounding stage altogether [12:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=763s) as well as combining the domain [12:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=766s) grounding stage with the capability [12:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=768s) induction stage in a multitask [12:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=771s) instruction tuning stage that those are [12:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=774s) rows A and B here in the middle and we [12:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=777s) saw for both of those that it degraded [12:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=779s) performance but actually the biggest [13:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=781s) drop in performance was from using a [13:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=784s) randomly initialized backbone instead of [13:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=787s) the pre-trained openweight LLM that we [13:09](https://www.youtube.com/watch?v=2LRIAfng7eA&t=789s) are using. We also evaluated using [13:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=793s) continuous pre-training for the domain [13:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=795s) grounding stage. And as you can see, [13:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=798s) it's it's minimal actual degradation on [13:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=801s) the task specific performance. [13:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=803s) But where continuous pre-training really [13:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=806s) hits us is on the natural language and [13:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=808s) world knowledge capabilities of the [13:31](https://www.youtube.com/watch?v=2LRIAfng7eA&t=811s) pre-trained backbone LLM we are using. [13:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=814s) after we do continuous pre-training, it [13:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=816s) goes to essentially zero versus if we do [13:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=818s) the frozen backbone domain grounding. We [13:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=821s) retain all of that core language ability [13:43](https://www.youtube.com/watch?v=2LRIAfng7eA&t=823s) and are still able to learn the semantic [13:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=825s) ids. [13:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=827s) I want to call out that these abilations [13:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=829s) were done with Quen, but we also [13:51](https://www.youtube.com/watch?v=2LRIAfng7eA&t=831s) validated that these findings hold with [13:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=834s) Llama. So, it is not specific to the [13:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=838s) model backbone, but actually the [14:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=840s) training paradigm itself. [14:03](https://www.youtube.com/watch?v=2LRIAfng7eA&t=843s) And then lastly, we did some [14:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=845s) investigation on different inference [14:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=847s) strategies and their effect on accuracy [14:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=851s) versus latency. We tested beam search [14:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=854s) with both constrained decoding and not. [14:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=858s) And we saw that even without constrained [14:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=860s) decoding, we can generate valid semantic [14:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=863s) IDs 98% of the time. [14:27](https://www.youtube.com/watch?v=2LRIAfng7eA&t=867s) Constrained decoding does add a little [14:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=869s) latency overhead, but it's also helpful [14:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=872s) for specific cases where you want to [14:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=874s) target specific types of content, such [14:36](https://www.youtube.com/watch?v=2LRIAfng7eA&t=876s) as only make new content [14:39](https://www.youtube.com/watch?v=2LRIAfng7eA&t=879s) recommendations, for example. [14:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=882s) We also compared top P sampling to beam [14:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=885s) search and saw that top P sampling [14:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=887s) pretty significantly hurts our accuracy. [14:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=890s) So although beam search is a little more [14:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=894s) latency intensive, we decided that [14:57](https://www.youtube.com/watch?v=2LRIAfng7eA&t=897s) trade-off worked for us. [15:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=902s) So what is meaningful about this? There [15:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=905s) have been lots of work in the industry [15:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=907s) in the space on generative semantic ID [15:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=910s) retrieval, toolbased LLM recommenders, [15:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=914s) the plum paper, etc. But NEO is actually [15:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=916s) the first example of combining all these [15:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=919s) capabilities into one system that [15:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=922s) understands grounded catalog items, is [15:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=925s) naturally language steerable, [15:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=928s) can do search, recommendation, [15:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=930s) explanation use cases, as well as [15:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=934s) low latency tool-free inference at an [15:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=937s) industrial scale. [15:41](https://www.youtube.com/watch?v=2LRIAfng7eA&t=941s) And we are using this in production [15:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=944s) today. Um there's a paper linked as well [15:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=947s) here for how we're using this to power [15:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=950s) podcast discovery. [15:53](https://www.youtube.com/watch?v=2LRIAfng7eA&t=953s) What we saw is that using a model train [15:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=956s) like this, we can break users out of [15:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=958s) their habitual patterns and get them to [16:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=961s) listen to more unfamiliar content. And [16:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=964s) we saw huge wins online with this. [16:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=970s) So none of this works without meaningful [16:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=973s) evaluations. So I want to talk about [16:15](https://www.youtube.com/watch?v=2LRIAfng7eA&t=975s) that a little bit as our recommendations [16:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=978s) are becoming more generative and [16:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=981s) explanatory. [16:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=983s) We our original maybe traditional [16:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=988s) offline eval metrics are not sufficient. [16:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=992s) Yes, they can tell us whether or not the [16:34](https://www.youtube.com/watch?v=2LRIAfng7eA&t=994s) user interacted with that content, but [16:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=997s) they can't tell us whether or not the [16:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1000s) user or that recommendation makes sense [16:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1002s) for the user, whether the explanation is [16:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1005s) accurate, whether it aligns with the [16:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1007s) user's intent, etc. So, that's where LLM [16:50](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1010s) judges really shine. But in our work at [16:54](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1014s) Spotify, we really find that you need to [16:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1016s) invest in grounding your LLM judges in [17:00](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1020s) meaningful data. so that they can be [17:02](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1022s) reliable [17:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1024s) evaluators that align with human [17:05](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1025s) preferences. So, a few examples um for [17:09](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1029s) evaluating the podcast recommendations [17:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1031s) that I just mentioned, we create textual [17:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1036s) user profiles summarizing the users's [17:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1040s) listening history and that is passed to [17:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1042s) the LLM as a judge and we saw that this [17:25](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1045s) corresponded with a 75% alignment [17:28](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1048s) between the LLM judge and human [17:30](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1050s) preferences. [17:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1052s) Similarly, you can use actual behavioral [17:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1055s) signals to ground these LLM judges. So, [17:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1058s) for example, for a search task, you can [17:42](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1062s) for a given query, you can take similar [17:45](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1065s) queries and how the user has interacted [17:47](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1067s) with them in the past and pass that to [17:49](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1069s) the model. And we saw that overall it [17:52](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1072s) increased alignment by 5% but on [17:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1076s) ambiguous queries it actually increased [17:59](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1079s) alignment by 91%. [18:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1081s) Showcasing the value these this [18:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1084s) grounding of the LLM judge plays [18:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1087s) especially in ambiguous cases where LLM [18:10](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1090s) judges tend to struggle. [18:13](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1093s) Lastly, we used grounded LM judges to [18:16](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1096s) scale up our Cranfield style [18:19](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1099s) collections. [18:21](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1101s) These are evaluation sets that are [18:23](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1103s) constructed by taking candidates from [18:26](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1106s) multiple different sources, creating a [18:29](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1109s) pool, and then using a human to rank [18:32](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1112s) that pool. However, that human ranking [18:35](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1115s) stage is expensive. So, we invested in [18:38](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1118s) significant grounding for our LM as [18:40](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1120s) judge and are able to have an LLM judge [18:44](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1124s) that aligns with uh human system [18:46](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1126s) rankings [18:48](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1128s) with an agreement value of 0.87. [18:56](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1136s) In summary, um like we're going to hear [18:58](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1138s) a lot about today in all the talks, [19:01](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1141s) there is a new era of personalization [19:04](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1144s) among us. this generative [19:06](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1146s) personalization. [19:07](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1147s) And if you want to power language [19:11](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1151s) steerable personalized recommendations [19:14](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1154s) for your user, this is how we taught [19:18](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1158s) openweight LLMs to speak Spotify. Thank [19:20](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1160s) you. [19:22](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1162s) [applause] [19:37](https://www.youtube.com/watch?v=2LRIAfng7eA&t=1177s) >> [music]