AI Engineer World's Fair 2025
Personalization in the Era of LLMs
Read the talk
From Listening History to Steerable Recommendations
Spotify’s personalization architecture turns listening histories into user vectors, catalog items into semantic tokens, and both into context for a language model that listeners can steer.
From a talk by Shivam Verma
Before you start: Familiarity with embeddings and next-token prediction will help; the article explains the recommendation-specific representations as they appear.
What does a recommender need to know about you?
How does Spotify turn what you listen to into recommendations for songs, tracks and podcast episodes you might like? That is a context problem inside the model: deciding how to represent the listener and the available content so that a recommendation system can use them together. Here, context engineering concerns the modeling layer rather than an agent’s tools or workflow.
Shivam Verma leads user representations in Spotify’s AI Foundation organization. Its remit includes user and content representations, plus adapting open-weight language models through continued pretraining and supervised fine-tuning. A machine learning engineer who previously worked at Twitter, Verma places this work beneath the recommendation products: these foundational models supply capabilities used across Spotify’s recommendation stack.
The architecture develops in three stages. First, understand users from their actions. Second, teach a language model about the platform’s catalog. Third, combine those representations into recommendations that users can steer. The progression is action sequences → vectors → tokens → vectors and tokens together in an LLM. User understanding and catalog understanding become complementary inputs to generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a changing playlist to explicit control
Verma describes Spotify at the time of the talk as having about 750 million monthly active users, more than 100 million tracks and a presence in 184 markets. He revises his audiobook estimate upward to roughly 400,000, alongside millions of podcasts and growing support for video episodes. The challenge is to connect this varied catalog to individual listeners, across both content types and markets.
Machine learning has supported that task for at least a decade. Discover Weekly, which Verma dates to around 2015, is the familiar example: a playlist unique to the listener that keeps changing. Its value comes from making a large catalog feel personally relevant without requiring the listener to specify every selection. More personalized products and surfaces have accumulated around that baseline.
The newer interfaces let the listener express intent directly:
- AI DJ: spoken interaction can ask the system to recommend or play content.
- Prompted Playlist: natural-language requests produce custom playlists. Verma describes podcast support as arriving that week, extending the same interaction to episode selection.
- Taste Profile: Spotify exposes inferred preferences so the listener can say which should continue to shape recommendations and which should stop doing so. Verma describes Taste Profile as available in a few markets, with expansion planned later that year. This is control over inferred taste, not a demonstrated deletion mechanism for underlying listening records.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shared foundation beneath separate products
Traditional recommendation systems narrow the catalog in stages. In Verma’s description, candidate generation reduces millions of items to a few hundred, then one or more rankers produce the final recommendation list. This pattern serves home shelf ranking, personalized playlists, search, podcasts and ads.
The organizational structure often mirrors those separate products: each team owns a model, with its own features and level of quality. Spotify’s direction is toward a unified model with an LLM backbone that users can steer.
| Aspect | Traditional pipeline | Unified direction |
|---|---|---|
| Selection | Candidate generation, then ranking | Generation through an LLM backbone |
| Model ownership | Separate product models | Shared modeling foundation |
| User control | Mediated by each product | Explicit steering of recommendations |
This is a direction of development, rather than a claim that every existing pipeline has already been replaced. User modeling is one of the foundations needed to make it work.
User embeddings represent taste as vectors learned from interactions across sessions and years. Downstream recommendation and search systems can reuse those representations. Verma reports generating embeddings every day for more than one billion users, including people who are not monthly active. That population is broader than the active audience, and maintaining its representations is an expensive daily pipeline.
The earlier representation approach used an autoencoder, described in a team paper Verma recalls from the previous year. It takes user features, compresses them into a small vector, then reconstructs the original features from that vector. Reconstruction provides the learning signal: the compressed representation must retain information useful for recovering the inputs. That makes the vector reusable as a representation of the user’s interactions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model the sequence and the request together
The next step is a sequential foundation model built with transformers. User interactions become part of the model’s context, alongside the immediate request. The input includes the listening activity, request context, query or prompt, product surface and item being considered. Transformer layers and prediction heads learn how those inputs relate. Verma describes training these models on data from millions or hundreds of millions of users.
One visualization of a newer model shows what cross-content modeling can learn. Tracks appear in blue, podcast episodes in pink and users in green, including Verma and teammates. All occupy a shared embedding space. The compressed view makes neighborhoods on the underlying hypersphere visible: which content lies near a user, which lies farther away and how those positions relate to other users.
Verma’s own neighborhood provides a concrete check on that representation. As a machine learning engineer interested in Anthropic and the technology industry, his embedding sits close to a technology podcast. A second view draws listening connections outward from his point to tracks and episodes. The demonstration makes the model’s learned associations inspectable: a listener’s music and podcast activity can inform one shared representation rather than requiring separate notions of the person for each content type.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the language model a vocabulary for the catalog
Understanding the listener leaves another problem: understanding what can be recommended. Spotify represents songs, artists, podcasts and episodes with item vectors. Together with user vectors, these encode knowledge specific to the platform. Open-weight language models such as Llama and Qwen bring broader world knowledge. Fine-tuning combines those sources to support natural-language steering, recommendations and explanations. The adaptation has a cost: catastrophic forgetting can cause the model to lose some of its existing knowledge as it learns the domain.
The bridge from catalog vectors to a language model is the semantic ID. Verma tentatively traces the idea to Google research in a YouTube context. Instead of supplying a content vector directly as ordinary text, a tokenizer converts it into a short sequence of discrete tokens. His illustrative compression takes a 1,000-dimensional vector down to four or six tokens. The model can then learn to generate those tokens autoregressively: the sequence identifies a song or episode rather than spelling out ordinary words.
Spotify uses catalog data and these identifiers in continued training or post-training. The artist examples make the structure tangible: Ariana Grande and Bruno Mars each receive six token IDs, with the first two shared. Verma explains the common prefix through their shared pop characteristics; later tokens encode more specific distinctions. The identifier is hierarchical, moving from common structure toward finer detail. Predicting the complete token sequence lets the model identify the next artist, song or episode.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow an Italian listener’s history into the prompt
The next example follows an Italian listener. User context and a tokenized listening history enter the training data, teaching the language model to operate with semantic IDs. This is the domain adaptation step: the model learns how catalog tokens relate to a listener’s context and to the next item they consume. Its output target can be an episode or a track.
The podcast prompt makes the transformation explicit:
- Identify the listening-history item. The prompt initially contains a Spotify episode URI, the platform’s identifier for the episode.
- Convert the URI to semantic-ID tokens. Those tokens are the catalog representation the model attends to.
- Predict the next episode. The model uses that history and the Italian listener’s context to generate the next-item representation.
The URI names the item in the catalog; the semantic ID represents it in the model’s learned vocabulary. The slide places the prompt representations beside the listening-history cards and an output episode.
With this conversion in place, user modeling and catalog understanding can feed the same generative recommender. The system has a representation of the listener’s behavior and a token language for the content it can select. The remaining task is to make generation responsive to the particular person receiving the recommendation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the prompt specific to the listener
Taste Profile returns here as the user-facing part of the architecture. Verma describes it as launched a few weeks earlier: a textual representation of the listener that they can revise through chat, messages or added text. A person might ask for more Justin Bieber or say that a particular recommended podcast does not interest them. That edit feeds back into the generative model’s understanding of the user and influences subsequent recommendations.
Catalog adaptation alone cannot supply this individual context. Verma says the model’s limited training data cannot cover every user in Spotify’s audience. Collaborative-filtering generalization can transfer patterns between listeners, but the resulting model still needs a way to condition its response on the person currently making the request.
The mechanism is a soft token. Take the user representation, project it into the LLM’s embedding space and insert the resulting vector into the prompt. Unlike a discrete catalog token selected by ID, this is a continuous embedding supplied directly to the model. It changes with the user receiving the response, giving the same language model different personal context for different listeners.
The tensor operation can be expressed compactly in Python with PyTorch. Here, user_vectors has shape [batch, user_dim], while the already embedded prompt has shape [batch, sequence, llm_dim]. A projection produces one additional embedding per listener; concatenation places it before the existing prompt embeddings.
python
import torch
from torch import nn
class UserConditionedPrompt(nn.Module):
def __init__(self, user_dim: int, llm_dim: int):
super().__init__()
self.project_user = nn.Linear(user_dim, llm_dim)
def forward(
self,
user_vectors: torch.Tensor,
prompt_embeddings: torch.Tensor,
) -> torch.Tensor:
soft_token = self.project_user(user_vectors).unsqueeze(1)
return torch.cat((soft_token, prompt_embeddings), dim=1)
This illustrates the projection-and-insertion mechanism with a linear layer; the talk does not specify Spotify’s projection architecture or training recipe. The output is an embedding sequence for the LLM, not a generated recommendation. Semantic IDs give the model catalog items to generate; the soft token supplies the listener context for that generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Next-episode recommendations are already using the approach
Verma reports positive early results on internal metrics, without naming the metrics or quantifying the improvement. He also says an approach like this is already in production for Spotify’s podcast next-episode recommendations. That establishes a concrete deployment surface, rather than a measured claim about every recommendation product.
The resulting sequential model brings together user embeddings, semantic IDs that compress content representations, and soft tokenization that projects the listener into the LLM’s input space. At the close, Verma describes broader incorporation across Spotify’s recommender systems as work still ahead. The deployed next-episode use case is the starting point for expanding a shared architecture in which listening history informs the model and explicit user intent can steer what it generates.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Launch details and example prompts for creating podcast playlists, with beta eligibility and refresh options.
The Taste Profile beta lets listeners review inferred preferences and steer homepage recommendations.
Further reading
Spotify's technical account of catalog tokenization, language alignment, domain training, evaluation, and serving.
A production framework that learns user vectors with an autoencoder and reuses them across downstream recommendation tasks.
The TIGER approach represents items with semantic tokens and generates the next item's identifier.
Research on adapting semantic identifiers to large-scale YouTube recommendation ranking.
Read the complete timestamped transcript
- 0:00
[upbeat music] Right on time.
- 0:15
Uh, hi, everyone. I am Shivam. I'm from Spotify. Uh, and my talk is gonna be about how Spot-- at Spotify, we do personalization, um, especially in the era of LLMs.
- 0:26
So, uh, for those of you who use Spotify, um, I guess, do we have any Spotify users in, in the room? Raise your hand. Nice. Uh, yeah, thanks for using Spotify and, uh, a bit like...
- 0:40
In this talk, so this is gonna be less about context engineering from the conventional, like, agentic sense. It's gonna be more about how we do context engineering on the modeling side.
- 0:50
So if you're interested at all on how your Spotify app works, how we recommend you songs, tracks, episodes, et cetera, this talk is gonna be really useful for you to contextualize how we use your data for, for just recommending stuff to you that you like.
- 1:04
Um, so a bit about me. Um, I am the tech lead of, uh, the user representations team in Spotify's AI Foundation org. So, uh, the AI F-AI Foundation team builds all of the frontier foundational models that, that are used across the entire stack of recommendations at Spotify.
- 1:20
Um, so we do things like user representations, content representations, as well as, like, adapting open-weight LLMs. Um, we do CPT, SFT, like, all of the stuff that a lot of the frontier labs are doing, as well as, like, other sort of our competitors are doing, and we try to kind of make sure that, that our, our...
- 1:37
We're building the best, uh, music recommender system possible, uh, for you guys. So my, my background is as a machine learning engineer. I used to work at Twitter, and I live in London, so if you're around for a coffee chat, like, I'm very happy to, uh, after this talk or in general, like, talk about this stuff, uh,
- 1:54
because it's really cool. Um, so three things I'm gonna talk about today. Uh, the first thing is gonna be about foundational user modeling. Uh, so what is, what, what is that?
- 2:04
That's essentially us trying to understand you, the users. Um, these are, like, the three key aspects that we think are, like, very key aspects of the future of personalization in the era of LLMs.
- 2:15
So there's the, the user modeling component. Uh, then there's the content side. So how can you kind of teach LLMs about the content that you have, the catalog that you have on, on Spotify or whatever your sort of, um, your, uh, platform is.
- 2:27
And then the last thing, once you have these two, like, pieces of the puzzle, are building these, both of these together into, like, something which is steerable and personalized.
- 2:36
So as, as sort of steerable as possible, and I'm, I'm gonna talk a bit more about that. So generally, like, you can think about it as going from, like, sequences of actions to, to vectors.
- 2:46
Uh, and then once you have the vectors, you can go to tokens, and then once you have the tokens, you can actually combine the vectors and the tokens with the LLM that you have to get, like, like, a pipeline where you, you have something which is as personalized as possible.
- 2:59
Um, so a bit about Spotify. Uh, for those of you who have-haven't used it, uh, we have, like, about seven hundred and fifty million users right now, uh, MAUs.
- 3:11
Uh, we have, like, a hun- ca-catalog of, like, a hundred million-plus tracks. Uh, we have, I think about three hundred and fifty, I think it's more like four hundred thousand audiobooks now.
- 3:20
Uh, millions of podcasts and, like, a lot of video episodes as well. So we're, we're kind of, uh, more and more creators are sort of switching to video as, as a modality, and we, we're definitely kind of making sure that we support that.
- 3:33
Um, and we're in about a hundred and eighty-four markets. Um, so as you can see, like, we, we have a lot of users. We have a lot of data and content.
- 3:41
How can we combine all of that to build something which is, which is as useful for our users as possible? Um, the way we do that is we obviously, we've been using, um, machine learning models for at least, like, a decade, if not more.
- 3:54
Um, you might have heard of, like, Discover Weekly, or you're probably a user of that. Um, that's been around since, I think, twenty fifteen, uh, back when I was in grad school.
- 4:01
And that was one-- At that time, that was, like, for me, like, one of the coolest products, uh, with, like, my interactions across, like, the tech sort of stack that I was using at the time.
- 4:12
Just because, like, it's something which, which is, like, unique to you. It keeps changing. Um, and mo-over the last decade or so, we've kind of added more and more personalization and products, uh, to the, to the app.
- 4:23
Um, so we have, like, you know, we have a bunch of verticals. We have a bunch of new product surfaces. Uh, we have this thing called the AI DJ, where you can kind of talk to it, and it kind of recommends you stuff or it plays stuff for you.
- 4:35
Uh, we also have, like, a prompted playlist where you can actually prompt the model, uh, like Spotify's model, and it kind of generates, like, a custom playlist for you based on your prompts.
- 4:45
And as of this week, it also supports podcasts. So if you want, you can kind of just prompt it, and it'll create, like, a, like, a playlist of episodes for you based on whatever you're looking for.
- 4:55
Um, so that is sort of the future that we're heading towards, where users have steerability. Uh, users can kind of talk to Spotify in natural language. And, uh, now we also have something called the taste profile.
- 5:06
So this is, this is only supported, like, in a few markets, but this is gonna be expanded, uh, later this year. And the idea is that we wanna expose what we know about you, um, and then we wanna kind of let you choose which part of that you want us to kind of keep, which part of that
- 5:22
you want us to forget, and, you know, just allow you to kind of have as much control as possible. Um,
- 5:30
as we kind of work on this, like, just sort of some context for those of you who've not worked in the space of, like, recommender systems and machine learning.
- 5:38
So, uh, what we call trad recs, which used to be, like, the sort of predominant, uh, sort of paradigm of building recommender systems up until a few years ago, uh, is essentially, like, this multi-step pipeline where you have, like, a massive catalog of items.
- 5:55
Um, you have this candidate generation step, which kind of reduces that item space to, from, like, millions to, like, a few hundred. And then you have, like, a ranking stage, and sometimes, like, you have multiple rankers, which essentially- Bring that further down and then give you, like, the final list of whatever, like, you know, your top songs
- 6:12
that we wanna recommend to you. Uh, and we use this, like, across different products. So we have like home shelf ranking, and we have personalized playlists and search and podcasts and, like, ads and, like, a lot of other stuff.
- 6:26
And this is generally... Every team, like, every product has its own team that has its own model, so it's kind of, like, spread across different groups of people, and some models are better than others, some have different features.
- 6:37
So we're kind of moving away from that sort of siloed model of working towards this single unified model, uh, which is-- which supports, uh, similar to how LLMs work, which supports, like, an LLM backbone, and which allows you to kind of, uh, allows you, the user, to kind of steer it towards the sort of recommendations that you
- 6:54
want. Um, and one of the key components of that, so, uh, is, is the user modeling part. So that is, like, the team that I work, uh, that I work with.
- 7:03
And what we do is we build user embeddings, uh, which are essentially representations of vectors that, that tells Spotify about the user's taste across, like, all of the sort of history that we have on y- on you, the user, like, across the different sort of, uh, sessions that you've had with us over the years.
- 7:19
Um, and that becomes, like, the foundation of all of the models that are downstream, uh, that, that actually, um, recommend stuff or allow you to search for stuff. Um, and these models are, like, very complex, so they, they-- the embedding model is we generate embeddings for, like, a billion-plus users because we have, we have a lot of
- 7:37
users in general, uh, that are also, like, MAUs or that are not MAUs. Um, so we do that every day. So it's, it's, like, a massive, uh, pipeline. It's very expensive.
- 7:46
Um, and over the years, like, we've kind of moved away from having these sort of generalized user representations, which was, like, the predominant paradigm in machine learning, where you had these, uh, these models.
- 7:57
Like this-- In this case, like, this was a paper from our team last year, uh, where we, we kind of publicly spoke about the user em-embedding model that we have, which is generally, um, like at that time, it was, like, an autoencoder model.
- 8:10
If you're familiar with that, what that does is it kind of takes all of your features, it compresses it down to, like, a small vector, and then recreates your features from that.
- 8:17
And that sort of compression/decompression process allows the model to sort of learn about you, the user, and just, um, represent, represent user interactions in the form of a vector.
- 8:26
So this is fairly standard stuff, like, that, that a lot of, like, folks do in the NLP computer vision space as well. That is also kind of aligned with the way things were in recommendations.
- 8:36
Uh, we're kind of moving away from that towards, uh, the foundation modeling side. So now we kind of have this single sequential model, which as, as you can see, like, with the, the-- So the whole industry moving towards transformers and the whole shift that's happening across, uh, not just, like, the regular tech industry, but also the, the
- 8:54
companies that, uh, they use recommender systems and build recommender systems as, as sort of their main, uh, bread and butter. Um, so we're also a part of that, sort of one of those companies, and we, we- we're also moving towards using transformers for, for the, for that kind of stuff.
- 9:08
And the idea is that you wanna have, uh, the, the user's interactions as a part of the prompt. So it's, it's kind of like the context, the c-- Uh, like, I guess when we talk about context engineering, this is the context that we're talking about.
- 9:20
Um, and then there's obviously the request con-context. There is, like, the query, there's the product surface, et cetera, like, all of that stuff. And then there's the item that we-- that you're recommending.
- 9:28
So when you add all of this stuff, and then you put, like, a transformer layer and, like, a bunch of heads and all that stuff, and you, you train it over, like, millions or hundreds of millions of users' data, what you get is something really cool.
- 9:40
Um, what you get is something like this. Um, so this is an image which is from one of our, like, newer models, which, uh, what it, what it essentially shows you, this is like a compressed version of what the model is learning.
- 9:53
Um, in this case, we have tracks in the blue, and we have, like, episodes, which are like podcast episodes in pink, and then users, so that, that-- the one in green is me, and some of the other green ones are, like, other folks in my team.
- 10:06
Um, so that kind of shows you that we're kind of doing this cross-content modeling. We're embedding users, tracks, and episodes in the same sort of content space, or, or I guess embedding space.
- 10:17
Um, and you're able to kind of visualize, like, I guess from, from this image of, like, how on the hypersphere where you live alongside, like, different pieces of content and what is close to you, what is not close to you, and how can you kind of explore that, that neighborhood region, and how you kind of, uh, where
- 10:34
you live, uh, you know, contextualized by where your friends live and things like that. So this is sort of like a visualization of, like, what the model is learning.
- 10:42
And as you can see for me, like, I am a machine learning engineer. I care a lot about keeping up with, with what Anthropic is doing and what's happening in the tech industry and all that stuff.
- 10:51
So my specific embedding is really close to this big, big tech podcast. And on the right, you can see that, like, that the, the point where you see, like, those lines spreading, that is me.
- 11:01
And then the ones in pink are, like, streams for tracks, and the, the blue ones are streams for episodes. Uh, and... Or sorry, it's, it's the reverse. Um, and you can kind of visualize, like, you can contextualize whatever you're listening, whatever you're, like, not listening to, and how, how, how does the embedding space look like for users.
- 11:20
So these models are, like, really smart. Um, and the moment you give them, uh, information about the user, uh, they can, they can kind of learn to put everything together in, in, in a, in a single space, which is really cool.
- 11:32
Um, this next part is more about catalog understanding. So now that we have the users, we, we understand the users, we have a model for them. How do we understand the catalog?
- 11:42
Right. Uh, so catalog understanding generally, uh, there's like a number of ways that you can do-- you can understand the catalog. Like, the most common way is you train, like, similar to the, to the user side.
- 11:52
Like, you, you train a vector to understand the content. So you have a vector that represents the item, uh, whether that's a song or it's an artist or it's a podcast or an episode.
- 12:01
Um, and you have that vector. Um, and then alongside that, you have the user vector. So that is like the Spotify knowledge. So that's what we know about the, the content or the sort of the different entities that we're dealing with.
- 12:12
Um, and then you have, uh, the world knowledge. So that's coming not from Spotify, but it's coming from these open-weight LLMs that we're working with, um, some models like Llama or Qwen or like other sort of open source models.
- 12:24
Um, what we do is we fine-tune those models, and then we kind of embed Spotify's knowledge into those models through, uh, something that I'm gonna talk about. And, uh, what that does is it gives you steerability, it gives you better rec-recommendations, it gives you explainability.
- 12:38
So there's a lot of stuff that you get for free when you use language models for recommendations. Um, now there, there are like trade-offs here so that the model does end up like forgetting stuff, like catastrophic forgetting is an issue.
- 12:49
Um, but generally from what we've seen, like, these models are really good at combining world knowledge with whatever sort of the knowledge that you have from your platform and building something which you can kind of holistically use for, for recommendations.
- 13:02
Um, now the stuff that I was talking about where on the previous slide, um, how do we actually teach these LLMs about the content? Um, let's start with that.
- 13:12
So the way that we do that, uh, is using something called semantic IDs. Um, semantic IDs is like a fairly new concept. It, it-- I think there's, there was a paper from, uh, Google a few years ago which sort of introduced this concept in the context of YouTube.
- 13:28
Um, and what it does is, like, when you have a vector that represents a piece of content, so that's like a track or an episode in our case, what we do is we tokenize it similar to how LLMs, like, kind of we, we tokenize words.
- 13:40
Um, and what that does is it kind of compresses that massive, like, let's say, a thousand-dimensional vector into like four or six tokens. And that allows us to, uh, really use those tokens to train the, the LLM in the way that, like, LLMs are usually trained.
- 13:54
And it allows the LLM to kind of autoregressively generate the next token, or in this case, the next token is not a word, but it's the next song or it's the next episode that you're gonna be listening to.
- 14:05
Um, so that's, that's kind of what we're doing. Uh, we're post-training or we're-- I guess we're like, uh, continually training these LLMs with Spotify's sort of data that we have about the catalog.
- 14:17
Uh, we use semantic IDs to compress the vec-- like the vectors into semantic... Oh, sorry, the vectors into semantic IDs. And at the bottom you can see that we have examples of like Ariana Grande and Bruno Mars.
- 14:28
So, uh, we represent them as six tokens. So those numbers are actually like token IDs. Um, and the first two tokens for both of them are shared because they're both like pop artists and they're both like...
- 14:38
They both share something, uh, between them. But then the other tokens are different because those tokens sort of represent like more niches. So it's kind of like a hierarchical, uh, structure where you're, you're kind of compressing the embedding into these six tokens, and there's, there's a hierarchy to it.
- 14:54
And that allows the model to autoregressively generate like the next artist or the next song that you're gonna be listening to. Um,
- 15:02
so this kind of just overall-- like this kind of emphasizes how we do this. Uh, we kind of... We use the user context. In this case, we have like a user who's [REDACTED:origin], uh, their listening history, which is tokenized.
- 15:16
Uh, we send that listening history. Uh, we use that in the training data. So we, we kind of teach the LLM how to talk with semantic IDs. And, uh, that is like the domain adaptation that I was referring to earlier in the slides.
- 15:28
And then the final output is essentially generating, uh, the next item. So whether that's gonna be an episode or it's gonna be a track, like what is this person like listening to?
- 15:38
Um, this is sort of like one e-example, like taking the example of the [REDACTED:origin] person who maybe listens to an episode, like an [REDACTED:origin] podcast. Um, this is an example of a prompt that we give that model.
- 15:51
And you can see like the prompt on the left, uh, has the Spotify URI, which is like our representation of the, of, of the, uh, the item, in this case, the episode.
- 16:00
We convert that into a semantic ID, which is like the actual tokens that the, the model is, um, attending to. And that is used to finally predict what is the next episode that the mo-- that the user is gonna be listening to.
- 16:14
Um, so this is the catalog understanding part of it. So now that we have the user modeling part, we have the catalog understanding part, the next step is to essentially assemble all, all of these components, uh, to form like a single steerable personalized generative recommender system.
- 16:31
So this is sort of moving away from the traditional rec-recommender system model to, to this generative model. Um,
- 16:38
this is the sort of the product that I was talking about that we launched, uh, just a few weeks ago. This is called the taste profile. The idea is that you have some piece of text that represents what the-- who the user is.
- 16:50
Um, we, we expose it to you, the user, and then you're allowed to kind of tell Spotify by chatting or by kind of sending messages or, I guess, adding some text, text.
- 17:00
And maybe you wanna start listening to like Justin Bieber more, or you, you don't like this specific podcast that's being recommended to you. And what this does is it allows the model to essen-- like that, that data, that edit, or is gonna come back into the generative model, and it's gonna up-upgrade like the model's kind of ability
- 17:17
to, to sort of understand you and recommend stuff that's, that's better for you. Um, so essentially like we have the content piece of the puzzle, but we don't have the user piece yet.
- 17:28
Uh, and the user piece doesn't come because ultimately these models are trained on like a limited amount of training data. You cannot train them on every like seven fifty million plus users that we have.
- 17:38
Um, so there is gonna be some sort of level of like collaborative filtering. So the, the, the model is gonna kind of generalize hopefully, but it also needs to be personalized.
- 17:47
Um, the way that we do that is again, coming back to the idea of user models. Um, so, uh You have the LLM, uh, you have the user representation.
- 17:57
What you do is you project the user representation into the space of the LLM. Uh, and what that does is it kind of allows you to create what's called a soft token within the model, and it's es-essentially a token that represents the user, which is obviously contextually changed depending on, on the user that, that we're kind of
- 18:13
generating this response for. Um, and that kind of allows the model to be personalized. So that is kind of like the final piece of the puzzle where you have this vector projection, which is like you have the regular LLM, and then you have a user vector which is projected to the space of the LLM, and that gets
- 18:29
kind of inserted into the prompt, and then finally, when the model is actually, like, generating a recommendation, um, the model is personalized because it kind of has the context on you, like whoever, uh, we're kind of generating the, the recommendation for.
- 18:43
Um, these are kind of like some early results that we've had. Like, we've seen some pretty positive results on our internal metrics. Um, if you use Spotify, like if you use the next episode, sort of if you listen to, like, podcasts on Spotify, like this is something which is actually productionized now, so you-- if you're getting a
- 18:59
recommendation, it's, it's coming from, from something like this. Um, and that's, that's kind of like the final piece. So, uh, we have the embeddings, which represent the users. We have the semantic IDs, which represent, uh, like a compressed version of the content.
- 19:15
And we have this soft tokenization approach, which allows you to project, like, users into the token space of the model. And this kind of moves away from the traditional recommender system model, and it kind of moves towards this sort of sequential modeling, uh, framework, uh, which is something that we're kind of very excited about.
- 19:31
And, uh, that's... Yeah, I think that's, that's definitely, uh, gonna be like really exciting as we go forward and we build this more, more and more into all of the recommender systems that we have.
- 19:41
Uh, and we're very excited to kind of put this out there, uh, pretty soon. And yeah, uh, that's, that's my time. I'm, I'm really happy to connect or, like, if you have any questions, feel free to reach out after the talk. [clapping] [upbeat music]