[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12480,
    "text": "Welcome everyone. Um really glad and"
  },
  {
    "startMs": 15120,
    "text": "thank you Deven for inviting us as well"
  },
  {
    "startMs": 17199,
    "text": "to talk today. Uh so um uh Jackie and I"
  },
  {
    "startMs": 21600,
    "text": "are going to present today about making"
  },
  {
    "startMs": 23680,
    "text": "LLMs speak Spotify and how we turned our"
  },
  {
    "startMs": 27199,
    "text": "recommendation system on its head to be"
  },
  {
    "startMs": 29439,
    "text": "LLM native. Uh just a few quick words"
  },
  {
    "startMs": 32640,
    "text": "about myself. Uh I joined Spotify a year"
  },
  {
    "startMs": 35280,
    "text": "ago. Before that I was at Google working"
  },
  {
    "startMs": 37440,
    "text": "on personalization for Google search and"
  },
  {
    "startMs": 39600,
    "text": "prior to that I was uh working on"
  },
  {
    "startMs": 41600,
    "text": "personalization at Netflix."
  },
  {
    "startMs": 44480,
    "text": "Um all right so let's dive in. Um"
  },
  {
    "startMs": 50000,
    "text": "first a couple of numbers to describe"
  },
  {
    "startMs": 51600,
    "text": "the scale of the problem that we have to"
  },
  {
    "startMs": 53760,
    "text": "solve. Um Spotify has about I was a"
  },
  {
    "startMs": 57120,
    "text": "little bit sad not to see it on the DAU"
  },
  {
    "startMs": 58960,
    "text": "by chart because it should be up there"
  },
  {
    "startMs": 60559,
    "text": "but it has about 760 million uh active"
  },
  {
    "startMs": 64239,
    "text": "uh users monthly uh in about 184"
  },
  {
    "startMs": 68080,
    "text": "markets. But one thing that makes uh the"
  },
  {
    "startMs": 70159,
    "text": "Spotify personalization problem"
  },
  {
    "startMs": 71600,
    "text": "particularly challenging is the size of"
  },
  {
    "startMs": 74000,
    "text": "its catalog. Uh so of course you know"
  },
  {
    "startMs": 76240,
    "text": "Spotify has basically all music ever"
  },
  {
    "startMs": 78799,
    "text": "published a bit more than 100 million uh"
  },
  {
    "startMs": 81280,
    "text": "music tracks but it also has a range of"
  },
  {
    "startMs": 84320,
    "text": "videos, podcasts and audio books as"
  },
  {
    "startMs": 86320,
    "text": "well. So the matching problem uh is uh"
  },
  {
    "startMs": 89200,
    "text": "actually surprisingly uh complicated."
  },
  {
    "startMs": 92479,
    "text": "So today what we're going to talk about"
  },
  {
    "startMs": 94320,
    "text": "Jackie and I is basically the extent of"
  },
  {
    "startMs": 96240,
    "text": "this matching problem and how we uh how"
  },
  {
    "startMs": 98799,
    "text": "we are solving it. I'm going to talk"
  },
  {
    "startMs": 100799,
    "text": "about a little bit of history, the new"
  },
  {
    "startMs": 102479,
    "text": "phase that we're entering and then"
  },
  {
    "startMs": 103840,
    "text": "Jackie is going to help us uh go into"
  },
  {
    "startMs": 106240,
    "text": "the guts of the modes as well to"
  },
  {
    "startMs": 107680,
    "text": "understand how those things are trained"
  },
  {
    "startMs": 109200,
    "text": "as well. Uh so a little bit on history."
  },
  {
    "startMs": 112640,
    "text": "Uh initially Spotify personalization was"
  },
  {
    "startMs": 115040,
    "text": "really built around curation. So uh"
  },
  {
    "startMs": 117759,
    "text": "people basically manually assembling"
  },
  {
    "startMs": 119759,
    "text": "playlists that target specific taste."
  },
  {
    "startMs": 122719,
    "text": "And that's still a very common uh use"
  },
  {
    "startMs": 124799,
    "text": "case on Spotify. Right now we have about"
  },
  {
    "startMs": 126880,
    "text": "10 billion playlists with many many uh"
  },
  {
    "startMs": 129360,
    "text": "created every every hour."
  },
  {
    "startMs": 132319,
    "text": "Uh then you know Spotify moved into"
  },
  {
    "startMs": 134959,
    "text": "taking these creation signals and order"
  },
  {
    "startMs": 136879,
    "text": "signals and moving into recommendations."
  },
  {
    "startMs": 139440,
    "text": "So basically being able to turn these uh"
  },
  {
    "startMs": 142000,
    "text": "creation signals to something that can"
  },
  {
    "startMs": 144000,
    "text": "be applied at scale. Uh and a great"
  },
  {
    "startMs": 146720,
    "text": "example of that would be discover weekly"
  },
  {
    "startMs": 148560,
    "text": "for example that was launched in 2014 as"
  },
  {
    "startMs": 150959,
    "text": "one of the one of the early like"
  },
  {
    "startMs": 153200,
    "text": "recommendation use case on on Spotify."
  },
  {
    "startMs": 156160,
    "text": "But the phase that we entering now which"
  },
  {
    "startMs": 158160,
    "text": "uh uh which we're going to talk about in"
  },
  {
    "startMs": 160160,
    "text": "more details is something that we call"
  },
  {
    "startMs": 161599,
    "text": "generative personalization where we are"
  },
  {
    "startMs": 164160,
    "text": "not only solving a matching problem from"
  },
  {
    "startMs": 166080,
    "text": "the user to the content. We're also"
  },
  {
    "startMs": 168000,
    "text": "solving the ability to generate an"
  },
  {
    "startMs": 170400,
    "text": "experience that is interactively and"
  },
  {
    "startMs": 172480,
    "text": "dynamically shaped around each user."
  },
  {
    "startMs": 178480,
    "text": "And that transition from recommendations"
  },
  {
    "startMs": 181360,
    "text": "to generative personalization involves a"
  },
  {
    "startMs": 183760,
    "text": "couple of different uh a couple of"
  },
  {
    "startMs": 185519,
    "text": "different big shifts. One is move from"
  },
  {
    "startMs": 188720,
    "text": "moving moving from personalization as"
  },
  {
    "startMs": 191200,
    "text": "guessing uh where basically you have a"
  },
  {
    "startMs": 193680,
    "text": "ranking algorithms that spits out a"
  },
  {
    "startMs": 195360,
    "text": "brand ranked list of entities in your"
  },
  {
    "startMs": 197440,
    "text": "catalog to personalization as reasoning"
  },
  {
    "startMs": 199920,
    "text": "that can introspect these results and"
  },
  {
    "startMs": 202319,
    "text": "really try to understand whether that's"
  },
  {
    "startMs": 203840,
    "text": "indeed the right match for this user in"
  },
  {
    "startMs": 205760,
    "text": "this particular context."
  },
  {
    "startMs": 208239,
    "text": "The other aspect as well is moving from"
  },
  {
    "startMs": 211040,
    "text": "blackbox algorithms. uh it's very"
  },
  {
    "startMs": 213760,
    "text": "difficult to fully introspect a"
  },
  {
    "startMs": 215840,
    "text": "multi-stage ranking system for example"
  },
  {
    "startMs": 218400,
    "text": "to transparent and steerable"
  },
  {
    "startMs": 220000,
    "text": "personalization where the user is always"
  },
  {
    "startMs": 221680,
    "text": "fully in control. So uh basically giving"
  },
  {
    "startMs": 225120,
    "text": "the ability to these mods to speak and"
  },
  {
    "startMs": 228239,
    "text": "understand English."
  },
  {
    "startMs": 230720,
    "text": "The other thing that these these systems"
  },
  {
    "startMs": 232640,
    "text": "can do is uh uh not stopping just at"
  },
  {
    "startMs": 235519,
    "text": "recommending but also generation of"
  },
  {
    "startMs": 238239,
    "text": "experiences and explaining as well. uh"
  },
  {
    "startMs": 240879,
    "text": "and we're going to show a couple of"
  },
  {
    "startMs": 242000,
    "text": "examples of that."
  },
  {
    "startMs": 245040,
    "text": "So one example that uh we launched a"
  },
  {
    "startMs": 247519,
    "text": "couple of years ago uh so pretty early"
  },
  {
    "startMs": 250080,
    "text": "in that journey was the Spotify DJ. Uh"
  },
  {
    "startMs": 252640,
    "text": "so the Spotify DJ is something that you"
  },
  {
    "startMs": 254159,
    "text": "can spin up that will start playing"
  },
  {
    "startMs": 256320,
    "text": "music for you of course personalized"
  },
  {
    "startMs": 258959,
    "text": "uh and but one interesting thing about"
  },
  {
    "startMs": 261120,
    "text": "it is that since last year you can tap"
  },
  {
    "startMs": 263280,
    "text": "that button on the bottom uh right and"
  },
  {
    "startMs": 266400,
    "text": "steer it in whatever direction you see"
  },
  {
    "startMs": 268080,
    "text": "fit. So at any point you can chime in"
  },
  {
    "startMs": 270400,
    "text": "and let the algorithm know what you want"
  },
  {
    "startMs": 272560,
    "text": "and it's going to steer the session in"
  },
  {
    "startMs": 274240,
    "text": "the in in that in that direction."
  },
  {
    "startMs": 277919,
    "text": "Uh another example of what we call"
  },
  {
    "startMs": 279520,
    "text": "generative personalization is uh"
  },
  {
    "startMs": 281280,
    "text": "showcased in a prompted playlist here"
  },
  {
    "startMs": 283919,
    "text": "that Deanch showed a little bit earlier"
  },
  {
    "startMs": 286320,
    "text": "as well. Uh and here you can you"
  },
  {
    "startMs": 289199,
    "text": "basically have full unfettered access to"
  },
  {
    "startMs": 291520,
    "text": "the recommendation algorithm that"
  },
  {
    "startMs": 293520,
    "text": "Spotify has. uh and you can uh prompt it"
  },
  {
    "startMs": 296960,
    "text": "with very high level prompts or very"
  },
  {
    "startMs": 299120,
    "text": "detailed prompts. On the left hand side"
  },
  {
    "startMs": 301520,
    "text": "here, I have a prompt that tells me uh"
  },
  {
    "startMs": 303440,
    "text": "create me a playlist of bands that are"
  },
  {
    "startMs": 304960,
    "text": "playing in San Francisco tonight. Uh so"
  },
  {
    "startMs": 307199,
    "text": "turns out there's a bunch of good shows"
  },
  {
    "startMs": 308639,
    "text": "if you're excited to check them out. On"
  },
  {
    "startMs": 311280,
    "text": "the right hand side, you see a prompt"
  },
  {
    "startMs": 312639,
    "text": "that is asking for a playlist to"
  },
  {
    "startMs": 314160,
    "text": "accompany me on my run. And what's"
  },
  {
    "startMs": 316000,
    "text": "interesting with the right hand side as"
  },
  {
    "startMs": 317600,
    "text": "well is that you will see that the"
  },
  {
    "startMs": 319280,
    "text": "experience itself gets dynamically"
  },
  {
    "startMs": 321039,
    "text": "shaped as a function of the request on"
  },
  {
    "startMs": 322960,
    "text": "the user to be able to introduce itself"
  },
  {
    "startMs": 325600,
    "text": "and different phases in my run as well."
  },
  {
    "startMs": 329680,
    "text": "Uh, another one that I'm really excited"
  },
  {
    "startMs": 331520,
    "text": "about, we launched it in New Zealand a"
  },
  {
    "startMs": 333199,
    "text": "couple of months ago and uh, it's coming"
  },
  {
    "startMs": 335680,
    "text": "soon in more markets is something called"
  },
  {
    "startMs": 337520,
    "text": "the taste profile. And that basically"
  },
  {
    "startMs": 339840,
    "text": "gives you the ability to introspect in"
  },
  {
    "startMs": 342720,
    "text": "natural language what the Spotify"
  },
  {
    "startMs": 344720,
    "text": "algorithm has understood about you in a"
  },
  {
    "startMs": 347199,
    "text": "way that you can edit and refine. Uh, so"
  },
  {
    "startMs": 349759,
    "text": "if you see something that's missing or"
  },
  {
    "startMs": 351360,
    "text": "something that's wrong. Uh so for"
  },
  {
    "startMs": 352960,
    "text": "example one of my edit is that all"
  },
  {
    "startMs": 355520,
    "text": "Disney music are my kids because we have"
  },
  {
    "startMs": 357520,
    "text": "a bunch of shared devices at home but"
  },
  {
    "startMs": 359199,
    "text": "please don't recommend that to me that's"
  },
  {
    "startMs": 360639,
    "text": "not my taste please. Uh and the"
  },
  {
    "startMs": 363039,
    "text": "algorithm would then take that into"
  },
  {
    "startMs": 364319,
    "text": "account uh making sure that we never"
  },
  {
    "startMs": 366319,
    "text": "recommend this this in the wrong"
  },
  {
    "startMs": 368080,
    "text": "context. Uh and similarly you can also"
  },
  {
    "startMs": 371199,
    "text": "use the taste profile to share some more"
  },
  {
    "startMs": 372720,
    "text": "aspiration uh goals as well. Uh so"
  },
  {
    "startMs": 375360,
    "text": "getting into a new genre, getting into a"
  },
  {
    "startMs": 377039,
    "text": "new topic, uh learning about a new"
  },
  {
    "startMs": 379039,
    "text": "language for example, all of these"
  },
  {
    "startMs": 380560,
    "text": "things can be can be done."
  },
  {
    "startMs": 383360,
    "text": "Uh another one that's coming uh soon uh"
  },
  {
    "startMs": 385919,
    "text": "which we uh announced very recently is"
  },
  {
    "startMs": 388240,
    "text": "something called personal podcast where"
  },
  {
    "startMs": 390000,
    "text": "the the the generative personalization"
  },
  {
    "startMs": 392000,
    "text": "system doesn't stop at just recommending"
  },
  {
    "startMs": 393919,
    "text": "and ascending experiences but also"
  },
  {
    "startMs": 395600,
    "text": "generating content as well. In this"
  },
  {
    "startMs": 397919,
    "text": "particular example, I'm generating a"
  },
  {
    "startMs": 399360,
    "text": "daily brief that's uh that's generated"
  },
  {
    "startMs": 401840,
    "text": "on a cadence daily. Uh and I'm going to"
  },
  {
    "startMs": 405759,
    "text": "uh let it uh tell me about what's"
  },
  {
    "startMs": 408240,
    "text": "happening in my community."
  },
  {
    "startMs": 411280,
    "text": "All right. So now to go into the guts of"
  },
  {
    "startMs": 413840,
    "text": "it. So there's one big system that"
  },
  {
    "startMs": 415919,
    "text": "controls like all of these different"
  },
  {
    "startMs": 417120,
    "text": "applications I mentioned and we call"
  },
  {
    "startMs": 418560,
    "text": "that internally some the large taste"
  },
  {
    "startMs": 420400,
    "text": "model. We're not great at naming these"
  },
  {
    "startMs": 422800,
    "text": "internal things. Um so it has a couple"
  },
  {
    "startMs": 425360,
    "text": "of properties. One is that it"
  },
  {
    "startMs": 426720,
    "text": "understands every historical interaction"
  },
  {
    "startMs": 428720,
    "text": "piece of content on Spotify. Uh it"
  },
  {
    "startMs": 431039,
    "text": "combines prediction and reasoning to the"
  },
  {
    "startMs": 433280,
    "text": "point that I mentioned earlier. So not"
  },
  {
    "startMs": 435120,
    "text": "only guessing but also reasoning layered"
  },
  {
    "startMs": 437199,
    "text": "on top and it gives users the ability to"
  },
  {
    "startMs": 439680,
    "text": "shape and generate experiences in real"
  },
  {
    "startMs": 441520,
    "text": "time. So it's fully steable and"
  },
  {
    "startMs": 442880,
    "text": "promptable by uh by users. Um and uh as"
  },
  {
    "startMs": 447440,
    "text": "of today about one in four US premium"
  },
  {
    "startMs": 449680,
    "text": "subscribers interact with that system on"
  },
  {
    "startMs": 451199,
    "text": "a daily basis as well. So that's pretty"
  },
  {
    "startMs": 452880,
    "text": "exciting. Um, one thing that's exciting"
  },
  {
    "startMs": 455199,
    "text": "as well to uh is that deploying this"
  },
  {
    "startMs": 457759,
    "text": "system across existing recommendation"
  },
  {
    "startMs": 460000,
    "text": "surfaces as well also led to some gains."
  },
  {
    "startMs": 462960,
    "text": "Uh, we saw gains on autoplay, we saw"
  },
  {
    "startMs": 465599,
    "text": "gains on podcast discoveries. Uh, we saw"
  },
  {
    "startMs": 467840,
    "text": "gains on users interacting with DJ"
  },
  {
    "startMs": 469440,
    "text": "messages as well. So, we saw pretty"
  },
  {
    "startMs": 471759,
    "text": "pretty sizable gains across the board as"
  },
  {
    "startMs": 473599,
    "text": "well by deploying the system. On that"
  },
  {
    "startMs": 476000,
    "text": "note, I'm going to hand it over to"
  },
  {
    "startMs": 477120,
    "text": "Jackie to talk to us about what's one of"
  },
  {
    "startMs": 480160,
    "text": "the core component that underpins this"
  },
  {
    "startMs": 481680,
    "text": "whole system."
  },
  {
    "startMs": 483440,
    "text": "Hi everyone, I'm Jackie or Jacqueline, a"
  },
  {
    "startMs": 486319,
    "text": "staff machine learning engineer at"
  },
  {
    "startMs": 488000,
    "text": "Spotify. So let's dive a little bit"
  },
  {
    "startMs": 491520,
    "text": "deeper and talk about how these models"
  },
  {
    "startMs": 493360,
    "text": "are actually trained at least at"
  },
  {
    "startMs": 494800,
    "text": "Spotify. So semantic IDs were presented"
  },
  {
    "startMs": 498720,
    "text": "in the previous talk, but that is how we"
  },
  {
    "startMs": 501440,
    "text": "are embedding these openweight LLMs with"
  },
  {
    "startMs": 505120,
    "text": "knowledge of Spotify's catalog. They are"
  },
  {
    "startMs": 508080,
    "text": "created by taking existing content"
  },
  {
    "startMs": 510479,
    "text": "embeddings such as podcast episode"
  },
  {
    "startMs": 512959,
    "text": "embeddings and applying a quantization"
  },
  {
    "startMs": 516240,
    "text": "algorithm to convert them to a set of"
  },
  {
    "startMs": 520000,
    "text": "discrete tokens. We then take a"
  },
  {
    "startMs": 523519,
    "text": "openweight LLM such as Quen and we"
  },
  {
    "startMs": 527760,
    "text": "modify its vocabulary to add these new"
  },
  {
    "startMs": 530080,
    "text": "special tokens and then we fine-tune the"
  },
  {
    "startMs": 532800,
    "text": "model to be able to understand both"
  },
  {
    "startMs": 535360,
    "text": "natural language as well as these new"
  },
  {
    "startMs": 538399,
    "text": "special tokens semantic IDs that"
  },
  {
    "startMs": 540720,
    "text": "represent Spotify catalog entities. So"
  },
  {
    "startMs": 543440,
    "text": "we can power experiences such as this"
  },
  {
    "startMs": 546399,
    "text": "where the user can ask in natural"
  },
  {
    "startMs": 548240,
    "text": "language for a podcast on morality. And"
  },
  {
    "startMs": 551760,
    "text": "that is passed to the prompt along with"
  },
  {
    "startMs": 554080,
    "text": "their listening history represented as"
  },
  {
    "startMs": 557120,
    "text": "semantic IDs. And the model responds"
  },
  {
    "startMs": 560160,
    "text": "both with a relevant semantic ID podcast"
  },
  {
    "startMs": 563120,
    "text": "episode"
  },
  {
    "startMs": 564640,
    "text": "as well as a natural language"
  },
  {
    "startMs": 566480,
    "text": "description of why they recommended that"
  },
  {
    "startMs": 568480,
    "text": "to this user."
  },
  {
    "startMs": 573040,
    "text": "So how is this model actually trained?"
  },
  {
    "startMs": 575440,
    "text": "Um we published a paper linked here"
  },
  {
    "startMs": 579440,
    "text": "uh describing our training paradigm"
  },
  {
    "startMs": 582240,
    "text": "called NEO which consists of four"
  },
  {
    "startMs": 585279,
    "text": "distinct stages. The first I already"
  },
  {
    "startMs": 588880,
    "text": "covered which is the semantic foundation"
  },
  {
    "startMs": 592560,
    "text": "stage where we construct meaningful"
  },
  {
    "startMs": 595839,
    "text": "semantic ID tokens and then add them to"
  },
  {
    "startMs": 598959,
    "text": "an openway LLM's vocabulary."
  },
  {
    "startMs": 601680,
    "text": "The second stage we call domain"
  },
  {
    "startMs": 604160,
    "text": "grounding in which we align these new"
  },
  {
    "startMs": 607920,
    "text": "semantic ID token embeddings in the"
  },
  {
    "startMs": 610160,
    "text": "original language embedding space."
  },
  {
    "startMs": 613600,
    "text": "We do this by learning a birectional"
  },
  {
    "startMs": 616079,
    "text": "mapping between semantic ids to text,"
  },
  {
    "startMs": 619680,
    "text": "text to semantic ids and any"
  },
  {
    "startMs": 622560,
    "text": "combination."
  },
  {
    "startMs": 626000,
    "text": "And we actually freeze the LLM backbone"
  },
  {
    "startMs": 629519,
    "text": "at this stage and only train the new"
  },
  {
    "startMs": 633440,
    "text": "semantic ID embeddings. So the original"
  },
  {
    "startMs": 636800,
    "text": "model weights and embeddings are frozen"
  },
  {
    "startMs": 640640,
    "text": "and we just learn those new semantic ID"
  },
  {
    "startMs": 643200,
    "text": "tokens and this helps us to mitigate"
  },
  {
    "startMs": 645680,
    "text": "catastrophic forgetting of the"
  },
  {
    "startMs": 647839,
    "text": "pre-trained LLM's core language"
  },
  {
    "startMs": 650160,
    "text": "abilities."
  },
  {
    "startMs": 652079,
    "text": "The third stage we call capability"
  },
  {
    "startMs": 654320,
    "text": "induction which is multitask instruction"
  },
  {
    "startMs": 657279,
    "text": "tuning on tasks that Spotify cares about"
  },
  {
    "startMs": 660800,
    "text": "such as the ones shown here. next item"
  },
  {
    "startMs": 663920,
    "text": "recommendation retrieval etc."
  },
  {
    "startMs": 667839,
    "text": "This is done by unfreezing the whole"
  },
  {
    "startMs": 671600,
    "text": "model all of its weights and embeddings"
  },
  {
    "startMs": 674560,
    "text": "and running either full parameter"
  },
  {
    "startMs": 676160,
    "text": "fine-tuning or Laura fine-tuning on the"
  },
  {
    "startMs": 679440,
    "text": "multiple Spotify tasks and then there is"
  },
  {
    "startMs": 682560,
    "text": "an optional fourth stage to do post-"
  },
  {
    "startMs": 685519,
    "text": "training such as RL fine-tuning etc."
  },
  {
    "startMs": 691279,
    "text": "So, how much of a difference does this"
  },
  {
    "startMs": 693839,
    "text": "four-stage training paradigm actually"
  },
  {
    "startMs": 695839,
    "text": "make? I'll dive into a few of the"
  },
  {
    "startMs": 697839,
    "text": "abilations we've done to investigate"
  },
  {
    "startMs": 700640,
    "text": "this. First, we assessed whether"
  },
  {
    "startMs": 704000,
    "text": "multitask training is actually hurting"
  },
  {
    "startMs": 706480,
    "text": "performance by comparing the multitask"
  },
  {
    "startMs": 708800,
    "text": "model against single task variance. And"
  },
  {
    "startMs": 712320,
    "text": "we consistently saw that across our"
  },
  {
    "startMs": 714560,
    "text": "tasks, the multitask model can match or"
  },
  {
    "startMs": 717680,
    "text": "actually beat the single task"
  },
  {
    "startMs": 720000,
    "text": "performance indicating that there is"
  },
  {
    "startMs": 722560,
    "text": "some positive cross-learning happening"
  },
  {
    "startMs": 724720,
    "text": "across the tasks. This is particularly"
  },
  {
    "startMs": 729279,
    "text": "noticeable for audiobook"
  },
  {
    "startMs": 730880,
    "text": "recommendations. If you look here, which"
  },
  {
    "startMs": 733200,
    "text": "is a new newer content type at Spotify,"
  },
  {
    "startMs": 738240,
    "text": "demonstrating that these multitask"
  },
  {
    "startMs": 740480,
    "text": "models can help with cold start entities"
  },
  {
    "startMs": 742959,
    "text": "by learning from other items in the"
  },
  {
    "startMs": 746079,
    "text": "catalog such as podcast recommendations,"
  },
  {
    "startMs": 748240,
    "text": "how to make meaningful audiobook"
  },
  {
    "startMs": 750880,
    "text": "recommendations."
  },
  {
    "startMs": 753040,
    "text": "Then we did some abilations on the"
  },
  {
    "startMs": 754720,
    "text": "actual training recipe. We evaluated"
  },
  {
    "startMs": 757920,
    "text": "both dropping the frozen backbone domain"
  },
  {
    "startMs": 761120,
    "text": "grounding stage altogether"
  },
  {
    "startMs": 763760,
    "text": "as well as combining the domain"
  },
  {
    "startMs": 766560,
    "text": "grounding stage with the capability"
  },
  {
    "startMs": 768959,
    "text": "induction stage in a multitask"
  },
  {
    "startMs": 771200,
    "text": "instruction tuning stage that those are"
  },
  {
    "startMs": 774160,
    "text": "rows A and B here in the middle and we"
  },
  {
    "startMs": 777120,
    "text": "saw for both of those that it degraded"
  },
  {
    "startMs": 779040,
    "text": "performance but actually the biggest"
  },
  {
    "startMs": 781200,
    "text": "drop in performance was from using a"
  },
  {
    "startMs": 784880,
    "text": "randomly initialized backbone instead of"
  },
  {
    "startMs": 787360,
    "text": "the pre-trained openweight LLM that we"
  },
  {
    "startMs": 789920,
    "text": "are using. We also evaluated using"
  },
  {
    "startMs": 793440,
    "text": "continuous pre-training for the domain"
  },
  {
    "startMs": 795920,
    "text": "grounding stage. And as you can see,"
  },
  {
    "startMs": 798480,
    "text": "it's it's minimal actual degradation on"
  },
  {
    "startMs": 801040,
    "text": "the task specific performance."
  },
  {
    "startMs": 803839,
    "text": "But where continuous pre-training really"
  },
  {
    "startMs": 806399,
    "text": "hits us is on the natural language and"
  },
  {
    "startMs": 808639,
    "text": "world knowledge capabilities of the"
  },
  {
    "startMs": 811279,
    "text": "pre-trained backbone LLM we are using."
  },
  {
    "startMs": 814399,
    "text": "after we do continuous pre-training, it"
  },
  {
    "startMs": 816480,
    "text": "goes to essentially zero versus if we do"
  },
  {
    "startMs": 818800,
    "text": "the frozen backbone domain grounding. We"
  },
  {
    "startMs": 821519,
    "text": "retain all of that core language ability"
  },
  {
    "startMs": 823920,
    "text": "and are still able to learn the semantic"
  },
  {
    "startMs": 825920,
    "text": "ids."
  },
  {
    "startMs": 827680,
    "text": "I want to call out that these abilations"
  },
  {
    "startMs": 829920,
    "text": "were done with Quen, but we also"
  },
  {
    "startMs": 831680,
    "text": "validated that these findings hold with"
  },
  {
    "startMs": 834399,
    "text": "Llama. So, it is not specific to the"
  },
  {
    "startMs": 838880,
    "text": "model backbone, but actually the"
  },
  {
    "startMs": 840800,
    "text": "training paradigm itself."
  },
  {
    "startMs": 843120,
    "text": "And then lastly, we did some"
  },
  {
    "startMs": 845600,
    "text": "investigation on different inference"
  },
  {
    "startMs": 847519,
    "text": "strategies and their effect on accuracy"
  },
  {
    "startMs": 851600,
    "text": "versus latency. We tested beam search"
  },
  {
    "startMs": 854800,
    "text": "with both constrained decoding and not."
  },
  {
    "startMs": 858720,
    "text": "And we saw that even without constrained"
  },
  {
    "startMs": 860959,
    "text": "decoding, we can generate valid semantic"
  },
  {
    "startMs": 863519,
    "text": "IDs 98% of the time."
  },
  {
    "startMs": 867360,
    "text": "Constrained decoding does add a little"
  },
  {
    "startMs": 869360,
    "text": "latency overhead, but it's also helpful"
  },
  {
    "startMs": 872480,
    "text": "for specific cases where you want to"
  },
  {
    "startMs": 874720,
    "text": "target specific types of content, such"
  },
  {
    "startMs": 876800,
    "text": "as only make new content"
  },
  {
    "startMs": 879199,
    "text": "recommendations, for example."
  },
  {
    "startMs": 882399,
    "text": "We also compared top P sampling to beam"
  },
  {
    "startMs": 885680,
    "text": "search and saw that top P sampling"
  },
  {
    "startMs": 887839,
    "text": "pretty significantly hurts our accuracy."
  },
  {
    "startMs": 890720,
    "text": "So although beam search is a little more"
  },
  {
    "startMs": 894399,
    "text": "latency intensive, we decided that"
  },
  {
    "startMs": 897120,
    "text": "trade-off worked for us."
  },
  {
    "startMs": 902160,
    "text": "So what is meaningful about this? There"
  },
  {
    "startMs": 905040,
    "text": "have been lots of work in the industry"
  },
  {
    "startMs": 907360,
    "text": "in the space on generative semantic ID"
  },
  {
    "startMs": 910800,
    "text": "retrieval, toolbased LLM recommenders,"
  },
  {
    "startMs": 914240,
    "text": "the plum paper, etc. But NEO is actually"
  },
  {
    "startMs": 916959,
    "text": "the first example of combining all these"
  },
  {
    "startMs": 919120,
    "text": "capabilities into one system that"
  },
  {
    "startMs": 922079,
    "text": "understands grounded catalog items, is"
  },
  {
    "startMs": 925120,
    "text": "naturally language steerable,"
  },
  {
    "startMs": 928720,
    "text": "can do search, recommendation,"
  },
  {
    "startMs": 930639,
    "text": "explanation use cases, as well as"
  },
  {
    "startMs": 934480,
    "text": "low latency tool-free inference at an"
  },
  {
    "startMs": 937440,
    "text": "industrial scale."
  },
  {
    "startMs": 941360,
    "text": "And we are using this in production"
  },
  {
    "startMs": 944320,
    "text": "today. Um there's a paper linked as well"
  },
  {
    "startMs": 947519,
    "text": "here for how we're using this to power"
  },
  {
    "startMs": 950480,
    "text": "podcast discovery."
  },
  {
    "startMs": 953279,
    "text": "What we saw is that using a model train"
  },
  {
    "startMs": 956399,
    "text": "like this, we can break users out of"
  },
  {
    "startMs": 958720,
    "text": "their habitual patterns and get them to"
  },
  {
    "startMs": 961759,
    "text": "listen to more unfamiliar content. And"
  },
  {
    "startMs": 964959,
    "text": "we saw huge wins online with this."
  },
  {
    "startMs": 970880,
    "text": "So none of this works without meaningful"
  },
  {
    "startMs": 973600,
    "text": "evaluations. So I want to talk about"
  },
  {
    "startMs": 975360,
    "text": "that a little bit as our recommendations"
  },
  {
    "startMs": 978959,
    "text": "are becoming more generative and"
  },
  {
    "startMs": 981279,
    "text": "explanatory."
  },
  {
    "startMs": 983519,
    "text": "We our original maybe traditional"
  },
  {
    "startMs": 988399,
    "text": "offline eval metrics are not sufficient."
  },
  {
    "startMs": 992240,
    "text": "Yes, they can tell us whether or not the"
  },
  {
    "startMs": 994959,
    "text": "user interacted with that content, but"
  },
  {
    "startMs": 997839,
    "text": "they can't tell us whether or not the"
  },
  {
    "startMs": 1000320,
    "text": "user or that recommendation makes sense"
  },
  {
    "startMs": 1002880,
    "text": "for the user, whether the explanation is"
  },
  {
    "startMs": 1005920,
    "text": "accurate, whether it aligns with the"
  },
  {
    "startMs": 1007680,
    "text": "user's intent, etc. So, that's where LLM"
  },
  {
    "startMs": 1010959,
    "text": "judges really shine. But in our work at"
  },
  {
    "startMs": 1014320,
    "text": "Spotify, we really find that you need to"
  },
  {
    "startMs": 1016320,
    "text": "invest in grounding your LLM judges in"
  },
  {
    "startMs": 1020160,
    "text": "meaningful data. so that they can be"
  },
  {
    "startMs": 1022560,
    "text": "reliable"
  },
  {
    "startMs": 1024079,
    "text": "evaluators that align with human"
  },
  {
    "startMs": 1025919,
    "text": "preferences. So, a few examples um for"
  },
  {
    "startMs": 1029520,
    "text": "evaluating the podcast recommendations"
  },
  {
    "startMs": 1031839,
    "text": "that I just mentioned, we create textual"
  },
  {
    "startMs": 1036880,
    "text": "user profiles summarizing the users's"
  },
  {
    "startMs": 1040640,
    "text": "listening history and that is passed to"
  },
  {
    "startMs": 1042640,
    "text": "the LLM as a judge and we saw that this"
  },
  {
    "startMs": 1045360,
    "text": "corresponded with a 75% alignment"
  },
  {
    "startMs": 1048480,
    "text": "between the LLM judge and human"
  },
  {
    "startMs": 1050480,
    "text": "preferences."
  },
  {
    "startMs": 1052000,
    "text": "Similarly, you can use actual behavioral"
  },
  {
    "startMs": 1055039,
    "text": "signals to ground these LLM judges. So,"
  },
  {
    "startMs": 1058559,
    "text": "for example, for a search task, you can"
  },
  {
    "startMs": 1062559,
    "text": "for a given query, you can take similar"
  },
  {
    "startMs": 1065039,
    "text": "queries and how the user has interacted"
  },
  {
    "startMs": 1067039,
    "text": "with them in the past and pass that to"
  },
  {
    "startMs": 1069440,
    "text": "the model. And we saw that overall it"
  },
  {
    "startMs": 1072799,
    "text": "increased alignment by 5% but on"
  },
  {
    "startMs": 1076960,
    "text": "ambiguous queries it actually increased"
  },
  {
    "startMs": 1079280,
    "text": "alignment by 91%."
  },
  {
    "startMs": 1081600,
    "text": "Showcasing the value these this"
  },
  {
    "startMs": 1084720,
    "text": "grounding of the LLM judge plays"
  },
  {
    "startMs": 1087520,
    "text": "especially in ambiguous cases where LLM"
  },
  {
    "startMs": 1090880,
    "text": "judges tend to struggle."
  },
  {
    "startMs": 1093120,
    "text": "Lastly, we used grounded LM judges to"
  },
  {
    "startMs": 1096960,
    "text": "scale up our Cranfield style"
  },
  {
    "startMs": 1099440,
    "text": "collections."
  },
  {
    "startMs": 1101039,
    "text": "These are evaluation sets that are"
  },
  {
    "startMs": 1103440,
    "text": "constructed by taking candidates from"
  },
  {
    "startMs": 1106960,
    "text": "multiple different sources, creating a"
  },
  {
    "startMs": 1109360,
    "text": "pool, and then using a human to rank"
  },
  {
    "startMs": 1112080,
    "text": "that pool. However, that human ranking"
  },
  {
    "startMs": 1115280,
    "text": "stage is expensive. So, we invested in"
  },
  {
    "startMs": 1118559,
    "text": "significant grounding for our LM as"
  },
  {
    "startMs": 1120559,
    "text": "judge and are able to have an LLM judge"
  },
  {
    "startMs": 1124160,
    "text": "that aligns with uh human system"
  },
  {
    "startMs": 1126880,
    "text": "rankings"
  },
  {
    "startMs": 1128480,
    "text": "with an agreement value of 0.87."
  },
  {
    "startMs": 1136000,
    "text": "In summary, um like we're going to hear"
  },
  {
    "startMs": 1138559,
    "text": "a lot about today in all the talks,"
  },
  {
    "startMs": 1141280,
    "text": "there is a new era of personalization"
  },
  {
    "startMs": 1144320,
    "text": "among us. this generative"
  },
  {
    "startMs": 1146160,
    "text": "personalization."
  },
  {
    "startMs": 1147840,
    "text": "And if you want to power language"
  },
  {
    "startMs": 1151520,
    "text": "steerable personalized recommendations"
  },
  {
    "startMs": 1154000,
    "text": "for your user, this is how we taught"
  },
  {
    "startMs": 1158080,
    "text": "openweight LLMs to speak Spotify. Thank"
  },
  {
    "startMs": 1160880,
    "text": "you."
  },
  {
    "startMs": 1162638,
    "text": "[applause]"
  },
  {
    "startMs": 1177048,
    "text": ">> [music]"
  }
]