Personalization in the Era of LLMs

How does Spotify turn what you listen to into recommendations for songs, tracks and podcast episodes you might like? That is a context problem inside the model: deciding how to represent the listener and the available content so that a recommendation system can use them together. Here, context engineering concerns the modeling layer rather than an agent’s tools or workflow.

Shivam Verma leads user representations in Spotify’s AI Foundation organization. Its remit includes user and content representations, plus adapting open-weight language models through continued pretraining and supervised fine-tuning. A machine learning engineer who previously worked at Twitter, Verma places this work beneath the recommendation products: these foundational models supply capabilities used across Spotify’s recommendation stack.

The architecture develops in three stages. First, understand users from their actions. Second, teach a language model about the platform’s catalog. Third, combine those representations into recommendations that users can steer. The progression is action sequences → vectors → tokens → vectors and tokens together in an LLM. User understanding and catalog understanding become complementary inputs to generation.

Verma describes Spotify at the time of the talk as having about 750 million monthly active users, more than 100 million tracks and a presence in 184 markets. He revises his audiobook estimate upward to roughly 400,000, alongside millions of podcasts and growing support for video episodes. The challenge is to connect this varied catalog to individual listeners, across both content types and markets.

Machine learning has supported that task for at least a decade. Discover Weekly, which Verma dates to around 2015, is the familiar example: a playlist unique to the listener that keeps changing. Its value comes from making a large catalog feel personally relevant without requiring the listener to specify every selection. More personalized products and surfaces have accumulated around that baseline.

The newer interfaces let the listener express intent directly:

• AI DJ: spoken interaction can ask the system to recommend or play content.
• Prompted Playlist: natural-language requests produce custom playlists. Verma describes podcast support as arriving that week, extending the same interaction to episode selection.
• Taste Profile: Spotify exposes inferred preferences so the listener can say which should continue to shape recommendations and which should stop doing so. Verma describes Taste Profile as available in a few markets, with expansion planned later that year. This is control over inferred taste, not a demonstrated deletion mechanism for underlying listening records.

Traditional recommendation systems narrow the catalog in stages. In Verma’s description, candidate generation reduces millions of items to a few hundred, then one or more rankers produce the final recommendation list. This pattern serves home shelf ranking, personalized playlists, search, podcasts and ads.

The organizational structure often mirrors those separate products: each team owns a model, with its own features and level of quality. Spotify’s direction is toward a unified model with an LLM backbone that users can steer.

This is a direction of development, rather than a claim that every existing pipeline has already been replaced. User modeling is one of the foundations needed to make it work.

User embeddings represent taste as vectors learned from interactions across sessions and years. Downstream recommendation and search systems can reuse those representations. Verma reports generating embeddings every day for more than one billion users, including people who are not monthly active. That population is broader than the active audience, and maintaining its representations is an expensive daily pipeline.

The earlier representation approach used an autoencoder, described in a team paper Verma recalls from the previous year. It takes user features, compresses them into a small vector, then reconstructs the original features from that vector. Reconstruction provides the learning signal: the compressed representation must retain information useful for recovering the inputs. That makes the vector reusable as a representation of the user’s interactions.

The next step is a sequential foundation model built with transformers. User interactions become part of the model’s context, alongside the immediate request. The input includes the listening activity, request context, query or prompt, product surface and item being considered. Transformer layers and prediction heads learn how those inputs relate. Verma describes training these models on data from millions or hundreds of millions of users.

One visualization of a newer model shows what cross-content modeling can learn. Tracks appear in blue, podcast episodes in pink and users in green, including Verma and teammates. All occupy a shared embedding space. The compressed view makes neighborhoods on the underlying hypersphere visible: which content lies near a user, which lies farther away and how those positions relate to other users.

Verma’s own neighborhood provides a concrete check on that representation. As a machine learning engineer interested in Anthropic and the technology industry, his embedding sits close to a technology podcast. A second view draws listening connections outward from his point to tracks and episodes. The demonstration makes the model’s learned associations inspectable: a listener’s music and podcast activity can inform one shared representation rather than requiring separate notions of the person for each content type.

Understanding the listener leaves another problem: understanding what can be recommended. Spotify represents songs, artists, podcasts and episodes with item vectors. Together with user vectors, these encode knowledge specific to the platform. Open-weight language models such as Llama and Qwen bring broader world knowledge. Fine-tuning combines those sources to support natural-language steering, recommendations and explanations. The adaptation has a cost: catastrophic forgetting can cause the model to lose some of its existing knowledge as it learns the domain.

The bridge from catalog vectors to a language model is the semantic ID. Verma tentatively traces the idea to Google research in a YouTube context. Instead of supplying a content vector directly as ordinary text, a tokenizer converts it into a short sequence of discrete tokens. His illustrative compression takes a 1,000-dimensional vector down to four or six tokens. The model can then learn to generate those tokens autoregressively: the sequence identifies a song or episode rather than spelling out ordinary words.

Spotify uses catalog data and these identifiers in continued training or post-training. The artist examples make the structure tangible: Ariana Grande and Bruno Mars each receive six token IDs, with the first two shared. Verma explains the common prefix through their shared pop characteristics; later tokens encode more specific distinctions. The identifier is hierarchical, moving from common structure toward finer detail. Predicting the complete token sequence lets the model identify the next artist, song or episode.

The next example follows a listener. User context and a tokenized listening history enter the training data, teaching the language model to operate with semantic IDs. This is the domain adaptation step: the model learns how catalog tokens relate to a listener’s context and to the next item they consume. Its output target can be an episode or a track.

The podcast prompt makes the transformation explicit:

1. Identify the listening-history item. The prompt initially contains a Spotify episode URI, the platform’s identifier for the episode.
2. Convert the URI to semantic-ID tokens. Those tokens are the catalog representation the model attends to.
3. Predict the next episode. The model uses that history and the listener’s context to generate the next-item representation.

The URI names the item in the catalog; the semantic ID represents it in the model’s learned vocabulary.

With this conversion in place, user modeling and catalog understanding can feed the same generative recommender. The system has a representation of the listener’s behavior and a token language for the content it can select. The remaining task is to make generation responsive to the particular person receiving the recommendation.

Taste Profile returns here as the user-facing part of the architecture. Verma describes it as launched a few weeks earlier: a textual representation of the listener that they can revise through chat, messages or added text. A person might ask for more Justin Bieber or say that a particular recommended podcast does not interest them. That edit feeds back into the generative model’s understanding of the user and influences subsequent recommendations.

Catalog adaptation alone cannot supply this individual context. Verma says the model’s limited training data cannot cover every user in Spotify’s audience. Collaborative-filtering generalization can transfer patterns between listeners, but the resulting model still needs a way to condition its response on the person currently making the request.

The mechanism is a soft token. Take the user representation, project it into the LLM’s embedding space and insert the resulting vector into the prompt. Unlike a discrete catalog token selected by ID, this is a continuous embedding supplied directly to the model. It changes with the user receiving the response, giving the same language model different personal context for different listeners.

The tensor operation can be expressed compactly in Python with PyTorch. Here, user_vectors has shape [batch, user_dim], while the already embedded prompt has shape [batch, sequence, llm_dim]. A projection produces one additional embedding per listener; concatenation places it before the existing prompt embeddings.

import torch
from torch import nn

class UserConditionedPrompt(nn.Module):
    def __init__(self, user_dim: int, llm_dim: int):
        super().__init__()
        self.project_user = nn.Linear(user_dim, llm_dim)

    def forward(
        self,
        user_vectors: torch.Tensor,
        prompt_embeddings: torch.Tensor,
    ) -> torch.Tensor:
        soft_token = self.project_user(user_vectors).unsqueeze(1)
        return torch.cat((soft_token, prompt_embeddings), dim=1)

This illustrates the projection-and-insertion mechanism with a linear layer; the talk does not specify Spotify’s projection architecture or training recipe. The output is an embedding sequence for the LLM, not a generated recommendation. Semantic IDs give the model catalog items to generate; the soft token supplies the listener context for that generation.

Verma reports positive early results on internal metrics, without naming the metrics or quantifying the improvement. He also says an approach like this is already in production for Spotify’s podcast next-episode recommendations. That establishes a concrete deployment surface, rather than a measured claim about every recommendation product.

The resulting sequential model brings together user embeddings, semantic IDs that compress content representations, and soft tokenization that projects the listener into the LLM’s input space. At the close, Verma describes broader incorporation across Spotify’s recommender systems as work still ahead. The deployed next-episode use case is the starting point for expanding a shared architecture in which listening history informs the model and explicit user intent can steer what it generates.