AI Engineer World's Fair 2025
Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs
Read the talk
Improving Recommendation Systems and Search with LLM Techniques
Semantic IDs, synthetic labels, and shared models help recommendation systems learn from sparse interactions, improve discovery, and reduce duplicated engineering.
From a talk by Eugene Yan
Before you start: Familiarity with embeddings, recommendation ranking, and basic classification metrics will help; the article explains the two-tower and vector-scoring mechanisms as they arise.
Learning the next item from a sequence
How can a recommendation system learn what someone might want next from the items they have already encountered? Language modeling has supplied answers for years. Word2vec, introduced in 2013, provided a way to learn item embeddings from co-occurrences in user interaction sequences: items encountered in similar contexts acquire related representations. Eugene Yan opens the inaugural RecSys track at AI Engineer World’s Fair with this history, accompanied on the slides by Latte and Mochi.
GRU4Rec extended the sequence approach with gated recurrent units, predicting the next item from a short history. Transformers and attention then made it easier to model longer-range dependencies across hundreds or roughly 2,000 item IDs. The progression leads to three opportunities: give those IDs semantic meaning, use language models to improve training data, and share models across tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give new items a representation before they have interactions
A newly uploaded video already has images, audio, and a description. A hash-based item ID conveys none of that content. If the recommender learns only from interactions with that ID, it must learn the new item from scratch. The same problem persists for tail items with only one, two, or ten interactions: there is too little evidence to learn a useful representation, and popular items retain an advantage. Semantic IDs let content provide a starting point when behavioral evidence is sparse.
Kuaishou, the Chinese short-video platform also known for its Kling text-to-video model, illustrates the challenge. Yan describes an enormous stream of new uploads; each one must become recommendable before it accumulates much engagement. The design question is how to combine relatively static content embeddings with dynamic user behavior.
The starting architecture is a two-tower recommender. The user tower consumes a user ID and a sequence of interaction IDs. The item tower consumes item IDs, then adds multimodal content. Its content encoders are ResNet for visuals, BERT for video descriptions, and VGGish for audio. Updating these encoders through recommendation training would be difficult, so the system introduces a trainable layer between content and behavior.
The construction has three steps:
- Concatenate content embeddings. Join the visual, description, and audio representations.
- Assign cluster IDs. Apply k-means to the combined representations, grouping related videos into approximately 1,000 clusters. Yan recalls roughly 100 million videos in the talk; his companion survey describes over 10 million, so the exact population is uncertain.
- Learn embeddings for the cluster IDs. Keep the upstream content representations fixed while training the embedding tables indexed by those IDs.
The discrete cluster assignment connects content to a trainable representation. Recommendation training can therefore map content space into behavioral space without backpropagating through every content encoder.
Yan reports improvements over hash-based IDs on clicks and likes, plus a 3.6% increase in cold-start coverage. He describes coverage as the share of recommended videos that are new. Cold-start velocity instead concerns new videos reaching a view threshold; he reports an improvement but gives neither its magnitude nor the threshold in the recording. The percentage-point versus relative-lift interpretation of coverage is not established here.
Semantic IDs also create a connection between recommendation and language. Yan points ahead to Pinterest and YouTube examples, including a YouTube approach that combines language models with semantic IDs to explain why an item might appeal to a user—and connect human-readable descriptions back to IDs. The same representation that helps a new item enter the system can also make its relationship to a user more interpretable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use language models to supply missing data
Search needs more than item representations. Query expansion, synonyms, spelling correction, and metadata attached to the search index all help connect what a user types to what the catalog contains. Producing that information at scale has traditionally required human annotation or conventional automation. LLMs offer another source of synthetic data and labels, especially where existing data is thin. Indeed and Spotify show two different ways to put that capability to work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Indeed: distill a judgment into an online filter
Indeed’s problem began with emailed job recommendations that were poor fits. An unsuitable recommendation does more than waste a click: repeated bad matches erode trust until a user unsubscribes, making that relationship difficult to recover. Explicit negative feedback is sparse, while implicit feedback is ambiguous. Someone might ignore a job because the timing is wrong—or because their spouse works at the company—not because the recommendation is intrinsically bad. Indeed needed a lightweight classifier that could filter bad matches before sending them.
Start with an evaluation set, then choose the model. Experts labeled user–recommendation pairs using resume and activity data. Prompted Mistral and Llama 2 models then produced overly generic judgments, failing to use the relevant resume and job-description details despite having sufficient context length. A larger context window alone did not solve the task.
The next experiments exposed a tradeoff between accurate rejection and serving cost:
| Model | Reported result | Constraint |
|---|---|---|
| GPT-4 | Roughly 90% precision and recall | 32-second latency; cost considered too high |
| GPT-3.5 | 63% precision for bad recommendations | Too many good matches rejected |
Here, precision measures how often a recommendation flagged as bad actually is bad. At 63% precision, 37% of the flagged recommendations are good—not 37% of all recommendations. Rejecting that much useful material from the flagged set was an unacceptable business tradeoff. These are the historical model evaluations described in the talk.
Indeed next fine-tuned GPT-3.5. Yan says this met the desired precision guardrail and cost roughly one-quarter as much as GPT-4, but its reported 6.7-second latency was still too slow for online filtering. The useful role for the fine-tuned model was therefore label generation: its judgments became training data for a lightweight classifier.
The distilled classifier achieved a reported ROC AUC of 0.86 against LLM-generated labels. It was suitable for real-time filtering; the recording does not give a measured latency, and Yan’s suggestion of less than 200 milliseconds is an estimate. This separates the expensive work of producing labels from the frequent work of deciding whether to send a recommendation.
The online experiment challenged the assumption that fewer recommendations would mean fewer applications. Yan summarizes the intervention as removing about 20% of bad recommendations; the companion survey more precisely describes a 20% filter threshold, rather than a measured reduction of that size in bad matches. It reports 17.68% fewer batch matches, a 4.13% increase in application rate, and a 4.97% decrease in unsubscribe rate. Those last two results are rounded to about 4% and 5% in the talk. Sending fewer, better matches improved both applications and retention of the email relationship.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spotify: help users discover what they can search for
Spotify faced cold start at the category level. Users knew how to search for songs and artists, but adding podcasts and audiobooks did not automatically teach them to look for those formats. Expanding from music to audio required exploratory search: helping people discover possibilities they did not already know to request. The intervention was a query recommendation system.
Candidate generation combined existing techniques with language-model augmentation:
- Catalog and playlist titles: extract useful phrases from existing metadata.
- Search logs: reuse queries people already issue.
- Simple transformations: append
coverto an artist name. - LLM generation: add natural-language queries beyond those conventional sources.
The language model fills a gap in an otherwise conventional pipeline. It does not replace every source of candidates.
The system retains immediate search hits and ranks exploratory query suggestions alongside them. In the paper’s interface shown during the talk, suggestions appear above the item results. That placement can introduce podcasts and audiobooks in the course of an ordinary search, without requiring a separate announcement banner; it is a historical interface example, not a promise about today’s app.
The reported outcome was a 9% increase in exploratory searches, also supported by the paper coauthor’s announcement. Yan uses one-tenth of users exploring and growth like 1.1ⁿ to convey the potential for category expansion, but the measured query lift does not establish either a user-adoption share or a compounding growth rate. Nor does it isolate the contribution of LLM-generated candidates from the rest of the system.
Across Indeed and Spotify, augmentation supplies information that is otherwise expensive to obtain: labels for unsuitable matches, or language that connects users to underexplored inventory. Yan emphasizes richer data for tail queries and tail items at lower annotation cost and effort, and points ahead to an Instacart search example in the track.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Netflix: unify tasks through a shared input schema
A company may maintain separate systems for ads, search, homepage recommendations, similar-item recommendations, cart recommendations, and the thank-you page. Each adds pipelines and maintenance work, while headcount may stay fixed. Improvements to one model do not automatically improve the others. Shared models offer a way to reduce this duplication, following precedents in vision, language, and recommendation. Yan also points to Stripe’s transformer-based payments fraud model as an example of learning from sequences outside media and commerce recommendations.
Netflix’s UniCoRn, the unified contextual ranker, brings together search, pre-query recommendations shown before text entry, and similar-video recommendations. Its architecture combines a user foundation model that consumes watch history with a context and relevance model. The aim is to share learning across tasks that previously required bespoke models and operational support.
The key interface is a unified input schema. Yan describes user ID, item ID, an optional search query, country, and task. The task identifies which ranking problem the model is solving, while the shared fields make the examples compatible with one architecture.
| Task | User-facing context |
|---|---|
| Search | A user has entered a query |
| Pre-query | The search page is open before query entry |
| More Like This | A source video anchors similar-video recommendations |
A common schema lets these tasks share a model without pretending that their contexts are identical.
Missing fields require deliberate treatment. For item-to-item recommendations, there may be no search query. UniCoRn can use the current source item’s display-name tokens as the query representation. The source item is distinct from the candidate being scored: the title supplies context for finding related candidates, rather than using each candidate’s own title as the request.
Yan reports that the unified model matched or exceeded specialized models across multiple tasks. Even parity matters when it removes duplicated pipelines and technical debt. A shared foundation can make future improvements cheaper to develop and distribute, instead of requiring teams to repeat the same work separately for each surface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Etsy: retrieve for intent, preferences, and quality
Etsy’s changing inventory makes retrieval difficult at both ends of the query spectrum: very specific requests need precise matches, while broad requests may have little literal overlap with product descriptions. A query such as Mother’s Day gift can express a useful shopping intent even when suitable listings never contain those words. Content matching alone also misses the shopper’s preferences.
Etsy’s unified embedding-based personalized retrieval returns to the two-tower pattern: a product encoder on one side and a query encoder on the other. The product representation combines item text with behavioral information from query–product logs—what people searched for before clicking or buying a product. More specifically, the paper describes training T5-small to generate historically purchased queries from product text, then using its encoder. The query tower uses lightweight token representations rather than an identical T5 encoder.
Shared encoders represent text tokens, product-category tokens, and location. Location helps connect a shopper’s context to the product’s geography. Query–user scalar features add personalization from prior searches, purchases, and preferences. The resulting retrieval representation can express more than semantic similarity between a query string and a listing description.
The serving system adds another requirement: retrieved products should be good-quality results. Etsy appends a quality vector, including features such as ratings, freshness, and conversion rate, to the product embedding. It extends the query embedding by the same number of dimensions with a constant vector. The paper clarifies that these constants are static feature weights tuned through Bayesian optimization, not arbitrary padding. With a dot product, the construction adds a quality contribution to semantic relevance:
Here, p is the product embedding, u the query-user embedding, q the product’s quality features, and w the static quality weights. The base vectors can be normalized for cosine relevance before the extension; normalizing the entire extended vector would change this additive scoring rule.
A small Python example makes the concatenation concrete. These teaching values use two semantic dimensions and three already-scaled quality features. The same operation works with larger embeddings:
python
from math import sqrt
def normalize(vector):
norm = sqrt(sum(value * value for value in vector))
if norm == 0:
raise ValueError("Cannot normalize a zero vector")
return [value / norm for value in vector]
def dot(left, right):
return sum(a * b for a, b in zip(left, right, strict=True))
query = normalize([3.0, 4.0])
product = normalize([4.0, 3.0])
quality = [0.9, 0.4, 0.2] # rating, freshness, conversion features
weights = [0.1, 0.05, 0.2]
extended_query = query + weights
extended_product = product + quality
semantic_score = dot(query, product)
quality_score = dot(weights, quality)
retrieval_score = dot(extended_query, extended_product)
print(f"Semantic: {semantic_score:.2f}")
print(f"Quality: {quality_score:.2f}")
print(f"Combined: {retrieval_score:.2f}")
The added dimensions let the retrieval score account for quality while preserving the semantic component. They encode a weighted preference for quality features, rather than requiring those features to resemble the query text.
Yan reports about a 2.6% increase in sitewide conversion and more than a 5% increase in organic search purchases. The companion survey gives the more precise A/B lifts as 2.63% and 5.58%, respectively; organic search purchases should not be reinterpreted as purchase probability. These outcomes belong to the retrieval system as a whole, rather than establishing an isolated effect of vector concatenation. Shared representations also mean that improvements to a common model can benefit multiple use cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Share models until the tasks conflict
Unification has a limit: improving one task can make another worse. Yan calls this an alignment tax. Forcing twelve use cases into one model may be less useful than grouping them into two or three shared models whose objectives cooperate. The goal is to capture shared learning and reduce maintenance without accepting harmful negative transfer.
Yan closes by pointing to LinkedIn and Netflix sessions that continue the unified-model discussion. His three directions address different bottlenecks: semantic IDs bring content into sparse interaction histories, data augmentation supplies missing supervision and discovery language, and unified models let related tasks share engineering and learning. Their value comes from fitting these techniques to the system’s actual constraints—especially the serving costs, missing inputs, and task conflicts that determine whether an improvement can be used.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The speaker's page containing his opening slides and the RecSys track recording.
A shared contextual model for search, pre-query recommendations and similar-video recommendations.
Etsy's product and query-user towers, training methods, quality boosting and ANN serving architecture.
Further reading
Eugene Yan's broader survey of industrial architectures, synthetic data and unified recommendation systems.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everyone.
- 0:15
Um, thank you for joining us in today's RecSys-- the inaugural RecSys track at the AI Engineer World's Fair. So today, what I want to share about is what the future might look like when we use-- when we try to merge recommendation, recommendation systems and language models.
- 0:31
So my wife looked at my slides, and she's like, "They're so plain." So therefore, I'll be giving the talk together with Latte and Mochi. You might have seen Mochi wandering the halls around somewhere, but there'll be a lot of doggos throughout the slides.
- 0:42
I hope you enjoy. First, language modeling techniques are not new in recommendation systems. I mean, it started with Word2vec in twenty thirteen. We started learning item embeddings across, um, ac-- from co-occurrences in user interaction sequences.
- 0:58
And then after that, we started using GRU4. I don't know who here remembers recurrent neural networks, gated recurrent units. Yeah. So th-those were very short-term, and we predict the next item from a short set of sequences.
- 1:10
Then of course, um, transformers and attention came about, and we, we, we became better on attention on long-range dependencies. So that's where we started, "Hey, you know, can we just process on everything in the user sequence hundreds, two thousand item IDs long, and try to learn from that?"
- 1:26
And of course, now, today in this track, I wanted to share with you about three ideas that I think are worth thinking about: semantic IDs, data augmentation, and unified models.
- 1:36
So the first challenge we have is hash-based item IDs. Who here works on recommendation systems? So you, you, you probably know that hash-based item IDs actually don't encode the content of the item itself.
- 1:50
And then the problem is that every time you have a new item, you suffer from the cold start problem, which is that all-- you have to relearn about this item all over again.
- 1:59
And, and therefore-- And there's also sparsity, right? Whereby you have a long set of tail items that have maybe one or two interactions or even up to ten, but it's just not enough to learn.
- 2:07
So recommendation systems have this issue of being very popularity biased, and they just struggle with cold start and sparsity. So the solution is semantic IDs that may even involve multimodal content.
- 2:19
So here's an example of trainable multimodal semantic IDs from Kuaishou. So Kuaishou is kind of like TikTok or Xiaohongshu. It's a short video platform in China. I think it's the number two short video platform.
- 2:31
You might have used their text-to-video model, Kling, which they released sometime last year. So the problem they had, you know, being a short video platform, users upload hundreds of millions of short videos every day, and it's really hard to learn from these short videos.
- 2:44
So how can we combine static content embeddings with dynamic user behavior?
- 2:51
Here's how they did it, with trainable multimodal semantic IDs. So I'm gonna go through each step here. So this is the Kuaishou model. It's a standard two-tower network. Um, on the left, this is the embedding layer for the user, which is a s-standard sequence, uh, sequence of IDs and the user ID.
- 3:10
And on the right is the embedding layer for the item IDs. So these are fairly standard. What, what's new here is that they now take in content input. So all of these slides will be available online.
- 3:21
Um, don't, don't worry about it. Uh, I'll make it available right immediately after this. Um, and to encode visual, they use ResNet. To encode video descriptions, they use BERT.
- 3:31
And to encode audio, they use VGGish. Now, the thing about-- the, the trick is this: when you have these encoder models, it's very hard to back propagate and try to update these encoder model embeddings.
- 3:43
So what did they do? Well, firstly, they took all these content embeddings, and then they just concatenated them together. I know it sounds crazy, right? But just concat, concat them together.
- 3:53
Then they learn cluster IDs. So I think they shared in the paper they had, like, a hundred million short videos, and they learned just be-- via k-means clustering, a thousand cluster IDs.
- 4:04
So that's what you see over there in the model encoder, which is in the boxes at the, at the bottom, which is the cluster IDs. So be-- above the cluster IDs, you have the non-trainable embeddings.
- 4:14
Below that, you have the trainable cluster IDs, which are then all mapped to their own embedding table.
- 4:20
So the trick here is this. The model encoder, as you train the model, the model encoder learns to map the content space via the cluster IDs, which are mapped to the embedding table to the behavioral space.
- 4:33
So the output is this. Um, these semantic IDs not only outperform regular hash-based IDs on clicks and likes, right? Like, that's pretty standard. But what they were able to do was they were able to increase cold start coverage, which is the-- of a hundred videos that you share, how many of them are new, they were able to
- 4:51
increase it by three point six percent, and also increase cold start velocity, which is, okay, how many new videos were able to hit some threshold of, uh, views. And this, they did not, they did not share what the threshold was, but being able to increase cold start and cold start velocity by these numbers are pretty outstanding.
- 5:09
So the long story short, the benefits of semantic IDs, you can address cold start with the semantic ID itself, and now your recommendations understand content. So later in the talk, we're gonna see some amazing, uh, sharing from Pinterest and YouTube.
- 5:23
And in the YouTube one, you see how they actually blend language models with semantic IDs, whereby it can actually explain why you might like the semantic ID because it understands the semantic ID and is able to give human-readable explanations and vice versa.
- 5:39
Now, next question, and I'm sure all of this-- everyone here has this challenge. The lifeblood of machine learning is data, good quality data at scale, and this is very essential for search and, of course, recommendation systems.
- 5:54
But search is actually far more important. You, you need a lot of metadata. You need a lot of, uh, query expansion, synonyms. Uh, you need spell checking. You need, um, you-
- 6:04
Uh, all, all sorts of metadata to attach to your search index, but this is very costly and hard for to get. In the past, we used to do it with human annotations, or maybe you can try to do it automatically, but LLMs have been outstanding at this.
- 6:16
And I'm sure everyone here is sort of doing this to some extent, using LLMs for synthetic data and labels. But I want to share with you two examples, uh, from Spotify and Indeed.
- 6:28
Now, the Indeed paper, it's quite out- uh, I really like it a lot. So the problem that they were trying to face is that they were sending job recommendations to users via email.
- 6:38
But some of these job recommendations were bad. They, they were just not a good fit for the user, right? So they had poor user experience, and then users lost trust in the job recommendations.
- 6:47
Imagine... And how, how they would indicate that they lost trust was that, "These job recommendations are not gonna a good fit for me. I'm just gonna unsubscribe." Now, the moment a user unsubscribes from your feed or for your newsletter, it's very, very, very hard to get them back.
- 7:00
Almost impossible. So while they had explicit negative feedback, thumbs up and thumbs down, this was very sparse. How often would you actually give thumbs down feedback? Very sparse. And implicit feedback is often imprecise.
- 7:11
What do I mean? If you s- if you get some recommendations, but you actually don't act on it, is it because you didn't like it? Or is it because it's not the right time?
- 7:18
Or maybe your sib- your, your wife works there, and you don't want to [chuckles] work in the same company as your wife. So the solution they had was a lightweight classifier to filter bad recs.
- 7:28
And I'll tell you why I really like this paper from Indeed, in the sense that they didn't just share their successes, but they shared the entire process and how they get-- how they got there.
- 7:37
And it was fraught with challenges. Well, of course, the first thing that makes me really like it a lot was that they started with evals. So they had their experts label, um, job recommendations and, uh, user pairs.
- 7:50
And from the user, you have their resume data, you have their activity data, and they try to see, "Hey, you know, is this recommendation a good fit?"
- 7:58
Then they prompted open LLMs, uh, Mistral and Llama 2. Unfortunately, their performance was very poor. These, these models couldn't really pay attention to what was in the resume and what was in the job description, even though they had sufficient context length, and then the output was just very generic.
- 8:15
So to get it to work, they prompted GPT-4, and GPT-4 worked really well. Um, specifically, like GPT-4 had like ninety percent precision and recall. However, it's very costly. Um, they didn't share the actual cost, but it's too slow.
- 8:28
It's thirty-two seconds. Okay, if GPT-4 is too slow, what can we do? Let's try GPT-3.5. Unfortunately, GP- GPT-3.5 had very poor precision. What does this mean? In the sense that of the recommendations that it said were bad, only sixty-three percent of them were actually bad.
- 8:48
What this means is that they were throwing out thirty-seven percent of recommendations, which is one-third. And for a company that thrives on recommendations and people, uh, recru- recruiting through your recommendations, throwing out one-third of them that are actually good is, is, is quite a, it's quite a gut rail for them.
- 9:03
This was their key metric here. And also GPT-- So what they did then is they fine-tuned GPT-3.5. So you can see the, the entire journey, right? Open models, GPT-4, GPT-3, now fine-tuning GPT-3.5.
- 9:15
Um, GPT-3.5 got the precision they wanted, zero point three precision. And, you know, it's one quarter of GPT-4's cost and latency, right? But unfortunately, it was too, still too slow.
- 9:24
It was about six point seven seconds, and this would not work in an online filtering system. So therefore, what they did was they distilled a lightweight classifier on the fine-tuned GPT-3.5 labels.
- 9:34
And this lightweight classifier was able to achieve very high performance. Uh,
- 9:40
specifically zero point eight six AUC ROC. I mean, the numbers may not make sense to you, but suffice to say that in an industrial setting, this is pretty good.
- 9:48
And of course, they didn't mention the latency, but it was good enough for real-time filtering. I think less than two hundred milliseconds or something.
- 9:55
So the outcome of this was that they were able to reduce bad recommendations. They, they were able to cut out bad recommendations by about twenty percent. So initially, they had hypothesized that by cutting down recommendations, even though they were bad, you will get fewer subscriptions.
- 10:10
It's just like sending out links, right? You ha- might have links that are clickbait. Even though they are bad, people just click on it. And they thought that even if, if we cut down recommendations, even if they were bad, we would get lower application rate.
- 10:19
But this was not the case. In fact, because the recommendations were now better, application rate actually went up by four percent, and unsubscribe rate went down by five percent.
- 10:29
That, that's, that's quite a lot. So essentially, what this means is that in recommendations, quantity is not everything. Quality makes a big difference, and quality here moved the needle quite a bit by five percent.
- 10:40
The next example I want to share with you is Spotify. So who here knows that Spotify has podcasts and audiobooks?
- 10:47
Oh, okay. I guess you, you guys are not the target, target, uh, audience in th- in this use case. So Spotify is really known for song and artists, and a lot of their users just search for songs and artists, and they are very good at that.
- 10:58
But when they started introducing podcasts and audiobooks, how would you help your users know that, you know, these new items are available? And of course, there's a huge-ass cold start problem.
- 11:09
Now, it's not only cold start on item, it's now cold start on category. How do you start growing a new category within your service?
- 11:18
And of course, exploratory search was essential to the business, right? For going-- for expanding beyond music. Spotify doesn't want to do, just do music songs. They just now want-- now they're doing audio.
- 11:28
So the solution to that is a query recommendation system.
- 11:32
So how did they recommend-- How-- First, how did they generate new queries? Well, um, they have a bunch of, uh, ideas, which is, you know, extracted from catalog titles, playlist titles.
- 11:43
You mine it from the search logs. You just take the me-- uh, you just take the artist, and then you just add cover to it. And this is what they use from existing data.
- 11:51
Now, you might be un- wondering like, "Where is the LLM in this?" Well, the LLM is used to generate natural language queries. So this might not be sexy, but this works really well, right?
- 12:02
Take whatever you have with conventional techniques that work really well and use the LLM to augment it when you need it. Don't use the LLM for everything at the start.
- 12:11
So now they have these exploratory queries. When you search for something, you still get the im- you still get the immediate results hit, right? So you take all this, you add the immediate results, and then you rank these new queries.
- 12:27
So this is why when you do a search, this is the UX that you're probably gonna get right now. I got this from a paper. It may have changed recently.
- 12:33
So you still see the item queries at the bottom. But at the top, with the query recommendations, this is how Spotify informs users without having a banner. Now we have audiobooks, now we have podcasts, right?
- 12:44
You search for something, it actually informs you that we have these new categories. The benefit here is plus nine percent exploratory queries. Essentially, one-tenth of their users were now exploring their new products.
- 12:58
So imagine that one-tenth every day exploring their new products. How quickly would you be able to grow your, your new product category, right? It's actually one point one to the power of n.
- 13:08
You'll grow pretty fast. Long story short, I don't have to tell you about the benefits of LLM-augmented synthetic data, richer high-quality data at scale on the tail queries, right?
- 13:20
Even on the tail queries and the tail items, and it's far lower cost and effort than is even possible with human annotation. So later, we also have a talk from Instacart who will tell us about how they use, uh, LLMs to improve their search and recommenda-- uh, their search system.
- 13:37
Now, the last thing I want to share is this challenge, whereby right now,
- 13:44
in a regular company, the system for ads, for recommendations, for search, they're all separate systems. And even for recommendations, the, the model for homepage recommendations, the model for item recommendations, the model for cart, uh, add to cart recommendations, the model for the thank you page recommendations, they may all be different models, right?
- 14:03
So you can imagine this, you're gonna have many, many models, but you gonna have-- well, leadership expects you to keep the same amount of headcount. So then how do you try to get around this, right?
- 14:13
You have duplicative engineering pipelines. There's a lot of maintenance costs. And improving one model doesn't naturally transfer to the improvement in another model.
- 14:22
So the solution for this is unified models, right? I mean, it works for vision. It works for language. So why not recommendation systems? And we've been doing this for a while.
- 14:30
This is not new. And aside, maybe the text is too small, but this is a tweet from Stripe whereby they built a transformer-based payments fraud model, right? Even for payments, the sequence of payments, you can build a foundation model, which is transformer-based.
- 14:49
So I wanna share an example of the unified ranker for search and RecSys at Netflix, right? The problem I mentioned, they have teams, they are building bespoke models for search, similar item rec-- similar video recommendations, and pre-query recommendations, like on the search page before you even enter a search query.
- 15:03
High operational costs, um, you know, missed opportunities from learning throughout. So their solution is a unified ranker, and they call it a unified contextual ranker, which is Unicorn. So you can see over here, uh, at the bottom, that's the user foundation model.
- 15:18
And in it, you put in a user watch history. And then you also have the context and relevance model where, where, where you put in the context of the videos and what they've watched.
- 15:27
Now, the thing about this unified model is that it takes in unified input, right? So now if you are able to find a data schema where all your use cases and all your features can use the same input, you can adopt an approach like this, which is multi-- similar to multitask learning.
- 15:44
So the, the user in-- the input will be the user ID, the item ID, you know, the video or the drama or the series, the search query, if a search query exists, the country, and the task.
- 15:55
So of course, they have many different tasks. In this example, in the paper, they have three different tasks, uh, search, pre-query, and more like this. Now, what they did then was very smart imputation of missing items.
- 16:08
So for example, if you are doing an item-to-item recommendation, you're just done watching this video, you wanna recommend the next video, you will have no search query. How would you impute it?
- 16:17
Well, you just simply use the title of the current item and try to find similar items.
- 16:23
The outcome of this is that this unified model was able to match or exceed the metrics of their specialized models on multiple tasks.
- 16:31
Y-think about it. I mean, it, it doesn't seem very impressive, right? It may not seem very impressive. Match or exceed. It might seem we did all this work just to match.
- 16:39
But imagine unifying all of it, like removing the tech debt and building a better foundation for your future iterations. It's gonna make you iterate faster.
- 16:49
The last example I wanna share with you is unified embeddings at Etsy. So you might think that embeddings are not very sexy, but this paper from E-Etsy is, is really, uh, outstanding in what they share in terms of model architecture as well as their system.
- 17:00
So the problem they had was, how can you help users get better results from very specific queries or very broad queries? And if you know that Etsy inventory is constantly changing, uh, they, they, they don't have the, the same products all, all throughout, right?
- 17:14
It's very home-homegrown. So now you might be querying for something like Mother's Day gift that would almost match very few items. I think very few items would have Mother's Day gift in their description or in their title, right?
- 17:26
And you know, lexical embedding, the other problem is that knowledge-based embeddings, like lexical embedding retrieval, don't account for user preferences. So how do you try to address this?
- 17:37
The problem-- the-- how they address this is with unified embedding and retrieval. So if you remember, at the start of my presentation, I talked about the Kuaishou two-tower model, right?
- 17:48
There's the user tower, and then there's the item tower. We will see the same pattern again. Over here, you see the product tower, right? This is the product encoder.
- 17:56
So how they encode the product is that they use T5 models for text embeddings, right? Text item descriptions, as well as a qur-query product log for s-- query embeddings.
- 18:06
What was the query that was made, and what was the product that was eventually clicked or purchased?
- 18:11
Then over here on the, on the left, you see the query encoder, which is the search query encoder. And they both share- Encoders for the tokens, which is actually a text tokens, the product category, which is a token of, a token of itself, and the user location.
- 18:27
So what this means is that your, now your embedding is able to match user to the location of the product itself.
- 18:33
And then of course, to personalize this, they encode the user preferences via the query user scalar fac- features at the bottom. Essentially, what were the queries that the user searched for?
- 18:42
What, what did they buy previously? All their preferences. Now, this is... They also shared their system architecture. And over here, this is the product encoder from the previous slide and the query encoder from the previous slide.
- 18:53
But what's very interesting here is that they added a quality vector because they wanted to ensure that whatever was searched and retrieved was actually of good quality in terms of ratings, uh, freshness, and conversion rate.
- 19:07
And you know, what they did is they just simply concatenated this quality vector to the product embedding vector.
- 19:13
But when you do that for the query vector, you have to, you have to expand the product vector by the same dimension so that you can do a dot product or cosine similarity.
- 19:21
So essentially, they just slapped on a constant vector, uh, for the query embedding, and it just worked.
- 19:27
The result, two point six percent increase in conversion across the entire site. That's quite crazy. Um, and more than five percent increase in search purchases. If you search for something, the purchase rate increases by five percent.
- 19:39
Um, this is very, very... These are very, very, very good results for e-commerce. Um, so the benefits of, uh, unified models, you simplify the system. Uh, you-- Whatever you build to improve one side of the tower, uh, im-improve your model, your new unified model, also improves other use cases that use these unified models.
- 20:00
That said, there may also be the alignment tax. You, you, you may find that when you try to build this, try to compress all 12 use cases into a single unified model, you may need to split it up into maybe two or three separate unified models because that's just the alignment tax.
- 20:12
We are trying to get better at one task, actually makes the other task worse.
- 20:16
We have a talk from, uh, LinkedIn in the, in this afternoon's, in this afternoon block, the last talk of the block. And then we also have a talk from Netflix, uh, which will be sharing about their unified model at the start of the next block.
- 20:29
All right. The three takeaways I have for you,
- 20:32
think about it, consider it, semantic IDs, data augmentation, and unified models. Um, and of course, do stay, stay tuned for the rest of the track, uh, for the rest of the talks in this track.
- 20:44
Okay, that's it. Thank you. [audience applauding] [upbeat music]