Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs

How can a recommendation system learn what someone might want next from the items they have already encountered? Language modeling has supplied answers for years. Word2vec, introduced in 2013, provided a way to learn item embeddings from co-occurrences in user interaction sequences: items encountered in similar contexts acquire related representations. Eugene Yan opens the inaugural RecSys track at AI Engineer World’s Fair with this history, accompanied on the slides by Latte and Mochi.

GRU4Rec extended the sequence approach with gated recurrent units, predicting the next item from a short history. Transformers and attention then made it easier to model longer-range dependencies across hundreds or roughly 2,000 item IDs. The progression leads to three opportunities: give those IDs semantic meaning, use language models to improve training data, and share models across tasks.

A newly uploaded video already has images, audio, and a description. A hash-based item ID conveys none of that content. If the recommender learns only from interactions with that ID, it must learn the new item from scratch. The same problem persists for tail items with only one, two, or ten interactions: there is too little evidence to learn a useful representation, and popular items retain an advantage. Semantic IDs let content provide a starting point when behavioral evidence is sparse.

Kuaishou, the Chinese short-video platform also known for its Kling text-to-video model, illustrates the challenge. Yan describes an enormous stream of new uploads; each one must become recommendable before it accumulates much engagement. The design question is how to combine relatively static content embeddings with dynamic user behavior.

The starting architecture is a two-tower recommender. The user tower consumes a user ID and a sequence of interaction IDs. The item tower consumes item IDs, then adds multimodal content. Its content encoders are ResNet for visuals, BERT for video descriptions, and VGGish for audio. Updating these encoders through recommendation training would be difficult, so the system introduces a trainable layer between content and behavior.

The construction has three steps:

1. Concatenate content embeddings. Join the visual, description, and audio representations.
2. Assign cluster IDs. Apply k-means to the combined representations, grouping related videos into approximately 1,000 clusters. Yan recalls roughly 100 million videos in the talk; his companion survey describes over 10 million, so the exact population is uncertain.
3. Learn embeddings for the cluster IDs. Keep the upstream content representations fixed while training the embedding tables indexed by those IDs.

The discrete cluster assignment connects content to a trainable representation. Recommendation training can therefore map content space into behavioral space without backpropagating through every content encoder.

Yan reports improvements over hash-based IDs on clicks and likes, plus a 3.6% increase in cold-start coverage. He describes coverage as the share of recommended videos that are new. Cold-start velocity instead concerns new videos reaching a view threshold; he reports an improvement but gives neither its magnitude nor the threshold in the recording. The percentage-point versus relative-lift interpretation of coverage is not established here.

Semantic IDs also create a connection between recommendation and language. Yan points ahead to Pinterest and YouTube examples, including a YouTube approach that combines language models with semantic IDs to explain why an item might appeal to a user—and connect human-readable descriptions back to IDs. The same representation that helps a new item enter the system can also make its relationship to a user more interpretable.

Search needs more than item representations. Query expansion, synonyms, spelling correction, and metadata attached to the search index all help connect what a user types to what the catalog contains. Producing that information at scale has traditionally required human annotation or conventional automation. LLMs offer another source of synthetic data and labels, especially where existing data is thin. Indeed and Spotify show two different ways to put that capability to work.

Indeed’s problem began with emailed job recommendations that were poor fits. An unsuitable recommendation does more than waste a click: repeated bad matches erode trust until a user unsubscribes, making that relationship difficult to recover. Explicit negative feedback is sparse, while implicit feedback is ambiguous. Someone might ignore a job because the timing is wrong—or because their spouse works at the company—not because the recommendation is intrinsically bad. Indeed needed a lightweight classifier that could filter bad matches before sending them.

Start with an evaluation set, then choose the model. Experts labeled user–recommendation pairs using resume and activity data. Prompted Mistral and Llama 2 models then produced overly generic judgments, failing to use the relevant resume and job-description details despite having sufficient context length. A larger context window alone did not solve the task.

The next experiments exposed a tradeoff between accurate rejection and serving cost:

Here, precision measures how often a recommendation flagged as bad actually is bad. At 63% precision, 37% of the flagged recommendations are good—not 37% of all recommendations. Rejecting that much useful material from the flagged set was an unacceptable business tradeoff. These are the historical model evaluations described in the talk.

Indeed next fine-tuned GPT-3.5. Yan says this met the desired precision guardrail and cost roughly one-quarter as much as GPT-4, but its reported 6.7-second latency was still too slow for online filtering. The useful role for the fine-tuned model was therefore label generation: its judgments became training data for a lightweight classifier.

The distilled classifier achieved a reported ROC AUC of 0.86 against LLM-generated labels. It was suitable for real-time filtering; the recording does not give a measured latency, and Yan’s suggestion of less than 200 milliseconds is an estimate. This separates the expensive work of producing labels from the frequent work of deciding whether to send a recommendation.

The online experiment challenged the assumption that fewer recommendations would mean fewer applications. Yan summarizes the intervention as removing about 20% of bad recommendations; the companion survey more precisely describes a 20% filter threshold, rather than a measured reduction of that size in bad matches. It reports 17.68% fewer batch matches, a 4.13% increase in application rate, and a 4.97% decrease in unsubscribe rate. Those last two results are rounded to about 4% and 5% in the talk. Sending fewer, better matches improved both applications and retention of the email relationship.

Spotify faced cold start at the category level. Users knew how to search for songs and artists, but adding podcasts and audiobooks did not automatically teach them to look for those formats. Expanding from music to audio required exploratory search: helping people discover possibilities they did not already know to request. The intervention was a query recommendation system.

Candidate generation combined existing techniques with language-model augmentation:

• Catalog and playlist titles: extract useful phrases from existing metadata.
• Search logs: reuse queries people already issue.
• Simple transformations: append cover to an artist name.
• LLM generation: add natural-language queries beyond those conventional sources.

The language model fills a gap in an otherwise conventional pipeline. It does not replace every source of candidates.

The system retains immediate search hits and ranks exploratory query suggestions alongside them. In the paper’s interface shown during the talk, suggestions appear above the item results. That placement can introduce podcasts and audiobooks in the course of an ordinary search, without requiring a separate announcement banner; it is a historical interface example, not a promise about today’s app.

The reported outcome was a 9% increase in exploratory searches, also supported by the paper coauthor’s announcement. Yan uses one-tenth of users exploring and growth like 1.1ⁿ to convey the potential for category expansion, but the measured query lift does not establish either a user-adoption share or a compounding growth rate. Nor does it isolate the contribution of LLM-generated candidates from the rest of the system.

Across Indeed and Spotify, augmentation supplies information that is otherwise expensive to obtain: labels for unsuitable matches, or language that connects users to underexplored inventory. Yan emphasizes richer data for tail queries and tail items at lower annotation cost and effort, and points ahead to an Instacart search example in the track.

A company may maintain separate systems for ads, search, homepage recommendations, similar-item recommendations, cart recommendations, and the thank-you page. Each adds pipelines and maintenance work, while headcount may stay fixed. Improvements to one model do not automatically improve the others. Shared models offer a way to reduce this duplication, following precedents in vision, language, and recommendation. Yan also points to Stripe’s transformer-based payments fraud model as an example of learning from sequences outside media and commerce recommendations.

Netflix’s UniCoRn, the unified contextual ranker, brings together search, pre-query recommendations shown before text entry, and similar-video recommendations. Its architecture combines a user foundation model that consumes watch history with a context and relevance model. The aim is to share learning across tasks that previously required bespoke models and operational support.

The key interface is a unified input schema. Yan describes user ID, item ID, an optional search query, country, and task. The task identifies which ranking problem the model is solving, while the shared fields make the examples compatible with one architecture.

A common schema lets these tasks share a model without pretending that their contexts are identical.

Missing fields require deliberate treatment. For item-to-item recommendations, there may be no search query. UniCoRn can use the current source item’s display-name tokens as the query representation. The source item is distinct from the candidate being scored: the title supplies context for finding related candidates, rather than using each candidate’s own title as the request.

Yan reports that the unified model matched or exceeded specialized models across multiple tasks. Even parity matters when it removes duplicated pipelines and technical debt. A shared foundation can make future improvements cheaper to develop and distribute, instead of requiring teams to repeat the same work separately for each surface.

Etsy’s changing inventory makes retrieval difficult at both ends of the query spectrum: very specific requests need precise matches, while broad requests may have little literal overlap with product descriptions. A query such as Mother’s Day gift can express a useful shopping intent even when suitable listings never contain those words. Content matching alone also misses the shopper’s preferences.

Etsy’s unified embedding-based personalized retrieval returns to the two-tower pattern: a product encoder on one side and a query encoder on the other. The product representation combines item text with behavioral information from query–product logs—what people searched for before clicking or buying a product. More specifically, the paper describes training T5-small to generate historically purchased queries from product text, then using its encoder. The query tower uses lightweight token representations rather than an identical T5 encoder.

Shared encoders represent text tokens, product-category tokens, and location. Location helps connect a shopper’s context to the product’s geography. Query–user scalar features add personalization from prior searches, purchases, and preferences. The resulting retrieval representation can express more than semantic similarity between a query string and a listing description.

The serving system adds another requirement: retrieved products should be good-quality results. Etsy appends a quality vector, including features such as ratings, freshness, and conversion rate, to the product embedding. It extends the query embedding by the same number of dimensions with a constant vector. The paper clarifies that these constants are static feature weights tuned through Bayesian optimization, not arbitrary padding. With a dot product, the construction adds a quality contribution to semantic relevance:

Here, p is the product embedding, u the query-user embedding, q the product’s quality features, and w the static quality weights. The base vectors can be normalized for cosine relevance before the extension; normalizing the entire extended vector would change this additive scoring rule.

A small Python example makes the concatenation concrete. These teaching values use two semantic dimensions and three already-scaled quality features. The same operation works with larger embeddings:

from math import sqrt


def normalize(vector):
    norm = sqrt(sum(value * value for value in vector))
    if norm == 0:
        raise ValueError("Cannot normalize a zero vector")
    return [value / norm for value in vector]


def dot(left, right):
    return sum(a * b for a, b in zip(left, right, strict=True))


query = normalize([3.0, 4.0])
product = normalize([4.0, 3.0])
quality = [0.9, 0.4, 0.2]  # rating, freshness, conversion features
weights = [0.1, 0.05, 0.2]

extended_query = query + weights
extended_product = product + quality

semantic_score = dot(query, product)
quality_score = dot(weights, quality)
retrieval_score = dot(extended_query, extended_product)

print(f"Semantic: {semantic_score:.2f}")
print(f"Quality: {quality_score:.2f}")
print(f"Combined: {retrieval_score:.2f}")

The added dimensions let the retrieval score account for quality while preserving the semantic component. They encode a weighted preference for quality features, rather than requiring those features to resemble the query text.

Yan reports about a 2.6% increase in sitewide conversion and more than a 5% increase in organic search purchases. The companion survey gives the more precise A/B lifts as 2.63% and 5.58%, respectively; organic search purchases should not be reinterpreted as purchase probability. These outcomes belong to the retrieval system as a whole, rather than establishing an isolated effect of vector concatenation. Shared representations also mean that improvements to a common model can benefit multiple use cases.

Unification has a limit: improving one task can make another worse. Yan calls this an alignment tax. Forcing twelve use cases into one model may be less useful than grouping them into two or three shared models whose objectives cooperate. The goal is to capture shared learning and reduce maintenance without accepting harmful negative transfer.

Yan closes by pointing to LinkedIn and Netflix sessions that continue the unified-model discussion. His three directions address different bottlenecks: semantic IDs bring content into sparse interaction histories, data augmentation supplies missing supervision and discovery language, and unified models let related tasks share engineering and learning. Their value comes from fitting these techniques to the system’s actual constraints—especially the serving costs, missing inputs, and task conflicts that determine whether an improvement can be used.