# Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash Raghav Saboo, DoorDash AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/ACPEpji5NV4-distill-llm-dont-serve-it-search-personalization) · [Video](https://www.youtube.com/watch?v=ACPEpji5NV4) ## Distill the LLM, Don't Serve It: Search & Personalization at DoorDash DoorDash uses expensive reasoning offline to teach fast retrieval and ranking models what shoppers mean. Graded relevance labels, semantic IDs and reusable consumer memory then support everything from better search results to personalized store collections. ### A shopping mission extends beyond one search [0:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=12s) A shopper who has just adopted a puppy might begin with a search for puppy food. But the job is getting through the first week: food, a crate, a leash, training pads. Those needs can unfold through suggested searches and collections across several sessions and days. In this talk, Raghav Saboo, a staff machine learning engineer working on search and personalization at DoorDash, explains how shared semantic representations help different models follow that larger journey. ![Source frame: A shopping mission extends beyond one search](https://ai.engineer/api/talk-essay-media/a29b3979ad453f9c1dc2bd2d835daea8c5d1e857778a6b0f81c9e4c888b862f6) Source frame: A shopping mission extends beyond one search The expansion beyond restaurants into grocery, retail, pets and gifting makes this problem harder. Engagement optimization can learn which products people tend to click or buy. It does not automatically explain what an item means in a particular context, or what a shopper intends to accomplish. Saboo identifies semantic understanding as the bottleneck and develops four building blocks in order: supervision, catalog semantics, consumer memory and steerable content generation. ### Teach relevance before optimizing engagement [3:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=194s) Consider the query “gluten-free pasta.” Regular spaghetti can rank well because it sells well. Gluten-free white bread can enter the candidate set because it shares part of the query. Both miss the request in different ways: the spaghetti violates the dietary constraint, while the bread misses the product type. Graded relevance gives the system a more useful target. True gluten-free pasta is highly relevant; chickpea pasta is presented as a possible substitute; popular regular spaghetti fails the stated constraint. ![Source frame: Teach relevance before optimizing engagement](https://ai.engineer/api/talk-essay-media/1870895107ca776e4c64d63d62e47c139988d506e603dc829e14e34ec4c3d020) Source frame: Teach relevance before optimizing engagement Two existing sources of supervision each have a cost: - **Human annotation:** People can judge query–item relevance, but labeling is expensive and slow. A rapidly changing catalog also makes those judgments stale. - **Behavioral signals:** Clicks and purchases are abundant, but exposure, position, price, promotions and earlier model choices all influence them. An item selling well is therefore a reason to investigate its relevance, rather than a complete answer. DoorDash starts with a human-labeled seed set of query–item pairs on a three-level relevance scale: 0, 1 and 2. Suspicious disagreements receive extra attention. If an item labeled irrelevant nevertheless has strong add-to-cart or conversion behavior, a stronger LLM reevaluates it with more granular prompts. Query-to-category models provide another check: a query mapping to valid categories can prompt a label adjustment when the item belongs to those categories. These checks build the high-precision golden dataset used to train the scalable labeler. A lightweight LLM is then fine-tuned on that dataset and run offline across the catalog to generate graded query–item labels. Retrieval and ranking learn from the same target. This is the first concrete form of the title’s advice: spend on reasoning during preparation, then transfer what it learned into models that can serve quickly and cheaply. ### Separate related items, then balance relevance with conversion [6:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=382s) Retrieval needs more than a general sense that two things are related. At e-commerce scale, standard embeddings can place exact matches, substitutes and complementary products too close together. For the pasta query, that distinction determines whether the candidate set contains products satisfying the request or merely products associated with its words. ![Source frame: Separate related items, then balance relevance with conversion](https://ai.engineer/api/talk-essay-media/061fdf4287033f246c65239a994dd5f3bdc3bf4f1be7524067785a79f7a33b78) Source frame: Separate related items, then balance relevance with conversion DoorDash uses two-stage contrastive training on the graded labels. The first stage shapes the overall embedding geometry with two-tower encoders and a multilevel supervised contrastive loss. The resulting model then exposes its own difficult cases: negatives ranked too high and strong positives ranked too low. Those cases are relabeled and used in a second stage of curriculum training. Training becomes more focused because the first model identifies the distinctions it still cannot make. What changes between the two training stages? The flow below shows how a model’s ranking mistakes become the next training material. Saboo describes overlap between relevant and moderately relevant items after stage one, followed by substantially better separation after stage two. He reports a 2.3% improvement in relevance NDCG, a metric for the quality of an ordered result list. The talk’s percentage improvements do not specify experimental windows, sample sizes, baseline details or whether the changes are relative percentages or percentage points; they describe DoorDash’s reported outcomes rather than a portable expectation. The ranking model receives a related change. An ordinal relevance tower learns from the LLM’s graded labels alongside existing towers predicting clicks, add-to-cart events and conversions. The towers share lower layers, so their training updates teach a common representation about both semantic fit and engagement. The relevance tower predicts probabilities across relevance levels; a value function blends those predictions with engagement objectives for each product surface. The system can therefore learn the difference between gluten-free pasta and popular regular spaghetti while retaining the business signals needed to order useful results. The pasta example now has a causal path through the architecture: judgments distinguish constraint satisfaction from popularity; an offline labeler expands those judgments; retrieval learns finer separation; and ranking learns an explicit relevance prediction alongside conversion predictions. The observable result reported here is improved aggregate relevance, rather than a measured before-and-after result for that individual query. Keeping the production retrieval and ranking architecture also lets DoorDash test these improvements while retaining its serving and business objectives. Training on the distinctions the first model misses The first stage establishes broad geometry. Mining and relabeling turn its errors into a focused second-stage curriculum, improving separation between relevance levels. - Graded query–item labels — A shared relevance target for retrieval - Stage one: shape geometry — Two-tower encoders with multilevel supervised contrastive loss - Mine difficult cases — Negatives too high; strong positives too low - Stage two: curriculum training — Relabel difficult cases and train on them - Better relevance separation — Relevant and moderately relevant items become more distinct - Graded query–item labels → Stage one: shape geometry (Supervise) - Stage one: shape geometry → Mine difficult cases (Use base-model rankings) - Mine difficult cases → Stage two: curriculum training (Relabel and train) - Stage two: curriculum training → Better relevance separation (Refine distinctions) ### Give the catalog a learned, shared vocabulary [9:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=571s) At DoorDash’s scale, the catalog contains a few billion store-level items. A unique item identifier distinguishes records but says nothing about meaning. A human-curated taxonomy supplies meaning, yet can be too coarse and rigid: placing products under sauces and hot sauces does not express the finer relationships among them. Semantic IDs provide a short hierarchical code whose granularity can be controlled and whose structure can be used by downstream models. ![Source frame: Give the catalog a learned, shared vocabulary](https://ai.engineer/api/talk-essay-media/8e4bbe84c4f71f0360a13c90a71e8e5a381dc9cab1b49267fa7349cc46563d82) Source frame: Give the catalog a learned, shared vocabulary The beginning of a semantic ID identifies a broad neighborhood; later tokens express finer distinctions. In the hot-sauce example, products share their first and second prefixes, then split into specialties such as Mexican, Caribbean and Korean hot sauces. Saboo describes that structure as emerging from data with zero labels. Comparing prefixes gives models a way to recognize shared meaning at different resolutions, without requiring every distinction to be a manually maintained taxonomy branch. That shared representation supports several distinct uses: - **Cross-category relationships:** Chips, salsa and guacamole may occupy different taxonomy branches, while a semantic neighborhood can connect them as parts of one shopping mission. - **Cold starts:** New store items can enter models through encoded semantic tokens used as sparse ID features. - **Tail coverage:** Items with little interaction data can inherit signal from semantically related items, reducing the need to wait for their own exposure and purchase volume. - **Catalog audits:** Comparing learned semantic groupings with human labels can reveal disagreements worth inspecting. Semantic IDs reportedly improved ranking MRR by 4–5%, with conversion gains as well. MRR measures how early the first relevant result appears. The same representation also supports query reformulation: a search for Sriracha can lead to a suggestion for chili garlic sauce because the queries map into a catalog-grounded neighborhood. Inventory matters here. A plausible suggested query has little value if DoorDash has no products to satisfy it; connecting the query graph to catalog semantics makes the suggestion useful for shopping. ### Make shopper context reusable across models [14:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=843s) Once items have useful representations, the next question is what the system knows about the shopper. User embeddings are valuable, but they do not necessarily explain why someone has a preference or intent. An LLM also cannot readily consume an existing user embedding without additional work to learn its representation. Consumer memory supplies an explicit semantic counterpart that can be inspected and reused across recommendation models and LLM applications. ![Source frame: Make shopper context reusable across models](https://ai.engineer/api/talk-essay-media/a570d9ad3c0b4a27cd86a25a9142df53d1474a2471c7a776641dc3c1c79d74bc) Source frame: Make shopper context reusable across models The framework combines three kinds of context. Saboo calls these three timescales, although the third describes how information is supplied rather than how long it lasts: - **Long-term memory:** Preferences inferred from orders, searches, browsing and support interactions. - **Real-time context:** The current cart and active searches within a session. - **Stated preferences:** Constraints and preferences explicitly supplied through agentic interactions, such as Ask DoorDash. Long-term memory is organized into blocks that can grow as the system learns more: dietary preferences, dining preferences and substitution preferences, for example. Separating this representation from downstream applications means each application does not have to reinterpret the shopper’s history for itself. Each consumer’s memory is materialized in several forms: compact, human-readable text; embeddings of memory blocks for retrieval and ranking; and graph or hierarchical relationships connecting consumers with brands, taxonomies and revealed preferences. A context graph is useful because shopping relationships are sparse and often require several hops. Connecting a consumer to extracted memory concepts creates relationships that direct consumer–item interactions may not provide, particularly at fine taxonomy levels. Saboo reports that graph-based embeddings outperform DoorDash’s existing taxonomy-based embeddings for retrieval. The resulting composite memory objects already feed personalized collections, Ask DoorDash sessions, and features for retrieval and ranking. ### Generate collections offline, assemble the shopping experience online [18:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1102s) The fourth building block brings the earlier representations together. Semantic IDs, memory blocks and relevance supervision can feed LLMs, smaller language models and traditional models. Their outputs can take different shapes: ranked semantic ID lists, carousel titles and supporting copy. Personalized collections on store pages are a concrete application of that shared vocabulary. ![Source frame: Generate collections offline, assemble the shopping experience online](https://ai.engineer/api/talk-essay-media/772284e248f841963136277e49b480569d6993b797591351b08e591ad426e4f9) Source frame: Generate collections offline, assemble the shopping experience online Historically, store collections came from a fixed library or product attributes. The newer approach generates collections for individual consumers. An offline LLM process takes consumer memory and semantic IDs and synthesizes the title, subtitle and items. Control matters as much as generation: the system can shape collections around occasions and moments, then use its existing retrieval and ranking stack for item hydration and collection ranking when serving shoppers. Where does the expensive generation end, and the serving stack begin? The diagram separates batch collection creation from item hydration and ranking. That division lets LLMs produce a richer collection concept while established models handle the delivery of the shopping experience. The talk specifies offline generation but does not detail its refresh cadence, which limits how precisely we can explain the timing of responses to new occasions or session changes. The visible changes are specific to the shopper: - **Plant-based affinity:** A store page can show plant-based pantry rows. - **Cat-only households:** Collections can focus on cat dry food. - **Pantry restocking:** Session context can bias the experience toward pantry staples. In early pets-category tests, Saboo reports order-rate increases close to 1% and a 6% increase in active users. These are the business outcomes of the collection experience, distinct from the earlier retrieval and ranking metrics. Personalized collections across offline generation and serving Consumer memory and catalog semantics shape the generated collection. Existing retrieval and ranking systems hydrate its items and rank collections for the store page. - Consumer memory — Reusable shopper preferences - Semantic IDs — Shared catalog representation - Offline LLM generation — Batch synthesis for individual consumers - Generated collection — Title, subtitle and items - Existing retrieval and ranking — Item hydration and collection ranking - Personalized store page — For example, cat dry food rows for cat-only households - Consumer memory → Offline LLM generation (Personalize) - Semantic IDs → Offline LLM generation (Supply catalog semantics) - Offline LLM generation → Generated collection (Synthesize) - Generated collection → Existing retrieval and ranking (Hydrate and rank) - Existing retrieval and ranking → Personalized store page (Present collections) ### An online LLM call is often the wrong unit of architecture [20:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1248s) The ending returns to three decisions. Discovery needs an understanding of item meaning and shopper intent alongside engagement. Expensive reasoning can be captured offline in labels, semantic IDs and memory, then served through smaller, faster models. And those representations should be shared: the same work can support retrieval, ranking and content generation rather than being rebuilt separately for every feature. ![Source frame: An online LLM call is often the wrong unit of architecture](https://ai.engineer/api/talk-essay-media/a92964a3cf4d4ff8ff2b9ea9a19df5d88579b7af9fcd848c57cc58af5f3b0792) Source frame: An online LLM call is often the wrong unit of architecture Saboo’s closing architectural judgment is that an online LLM call is often not the product architecture you need. DoorDash’s examples give that judgment substance: a relevance label can teach both candidate retrieval and final ranking; a semantic code can improve ranking and suggest an inventory-backed search; consumer memory can personalize an agent session and a store collection. The useful output of reasoning is something the rest of the system can keep using. ## Key takeaways - Graded relevance separates satisfying a shopper’s constraints from merely being popular or related. DoorDash uses offline LLM labels to teach that distinction to both retrieval and ranking. - Two-stage retrieval training uses the first model’s mistakes to build a harder curriculum. Ranking adds an explicit relevance objective alongside clicks, add-to-cart events and conversion. - Semantic IDs and consumer memory supply reusable meaning: catalog neighborhoods help with sparse items and query reformulation, while text, vectors and graphs make shopper context useful to different model types. - Personalized collections combine offline LLM generation with existing item hydration and ranking. The broader architecture captures reasoning once and lets several fast serving systems benefit from it. ## Slides ### LLMs for marketplace discovery Raghav Saboo frames semantic understanding—what items mean in context and what shoppers intend—as a bottleneck for marketplace discovery. He introduces four LLM primitives for search and recommendations. [0:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=45s) [DoorDash title slide introducing four primitives for marketplace discovery.](https://assets.ai.engineer/sites/aiecode2025/sha256/86d817cea43ba620005722949ef28d4852f3bc186900bb4ae4e770bb39a346e0.webp) AI Engineering World's Fair · San Francisco · 2026 DOORDASH LLMs for marketplace discovery, in 4 pieces. Primitives we have built at DoorDash to integrate LLMs in Search & Recommendations. RAGHAV SABOO · AI / ML, NEW VERTICALS · DOORDASH ### From a shopping moment to a complete basket A puppy-food search can expand into collections and complementary items before checkout, potentially across multiple sessions and days. Four primitives support the journey across models. [1:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=99s) [A puppy-shopping journey moves from search through collections and recommendations to checkout, supported by four primitives.](https://assets.ai.engineer/sites/aiecode2025/sha256/96bcd9105dc624809a20ad9371db91dc83848ffc29d1da9c5afa85e0d1640ef6.webp) Capturing every shoppable moment, all the way to a complete basket. DoorDash is growing beyond Restaurants into Grocery, Convenience, Alcohol, Flowers, Retail, Gifting, and more. Every vertical brings shopping missions: one idea that starts with a search, moves through discovery and complementary recommendations, and ends in a complete basket. “I just adopted a puppy, what do I need to get started this week?” 1 Search query — 2 in cart puppy food Purina Puppy Chow FEEDING RELATED SEARCHES puppy treats small breed puppy food wet puppy food grain free puppy food 2 Collections — 5 in cart COLLECTION Puppy Starter Essentials TRAINING PADS + CLEANER · HOUSE-TRAINING COLLECTION Walks & Outdoors LEASH, COLLAR, POOP BAGS · WALKS 3 Complementary — 9 in cart Goes well with your basket Crate + dog bed SLEEP & SAFETY Safety gate SLEEP & SAFETY Chew + rope toys PLAY & TEETHING 4 Checkout — 12 · done FEEDING — added HOUSE-TRAINING — added SLEEP & SAFETY — added PLAY & TEETHING — added WALKS — added Complete basket · 12 items Supervision Intent resolution Graded relevance Catalog Semantics Query Reformulation Semantic IDs Steerable Content Generation Content generation Explains recs Semantic Personalization Consumer memory Context graph ### Supervision Supervision asks how to teach retrieval and ranking systems what good means for different tasks. [3:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=192s) [Black section divider labeled Supervision and 01.](https://assets.ai.engineer/sites/aiecode2025/sha256/f83782464c9f50daf7e790a5930a919c8f065b162e8ce79835ce4982876ca624.webp) Supervision. 01 ### Engagement is a biased proxy for relevance For a gluten-free pasta query, engagement ranks regular spaghetti and bread above the exact match. Graded relevance reverses that order. Human labels are costly and stale; behavioral signals reflect exposure and position. [3:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=203s) [A gluten-free pasta example compares engagement ranking with graded relevance and describes limits of human labels and behavioral signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/b02d31f2e40619935facba3258122734bc015e8cdfb0ea941fc1a68945f780c1.webp) Engagement is a biased proxy for relevance and intent. Engagement-only ranking rewards popularity, price, and keyword overlap. The target we need is graded relevance: how well each item fits the shopper's intent. Search: gluten-free pasta RANKED BY ENGAGEMENT · today #1 Barilla Classic Spaghetti best-seller · contains gluten #2 Gluten-Free White Bread keyword overlap · not pasta #3 Barilla Gluten-Free Penne what you wanted · buried #4 Banza Chickpea Pasta substitute RANKED BY GRADED RELEVANCE · the target Barilla Gluten-Free Penne exact intent · Rel 2 Banza Chickpea Pasta gluten-free substitute · Rel 1 Barilla Classic Spaghetti popular, but contains gluten · Rel 0 Gluten-Free White Bread keyword match, not pasta · Rel 0 TWO WAYS TO LABEL RELEVANCE · why neither scales on its own Human annotation · Reliable, won't scale Accurate label by label, but slow and expensive, and the catalog changes daily, so labels land stale. • Coverage: can't reach the long tail of the catalog. • Freshness: labels arrive stale as items turn over. • Cost: human grading is too expensive to scale. Behavioral signals · Abundant, biased Clicks and orders are everywhere, but they measure popularity and exposure, not whether an item fits the intent. • Position bias: top slots get the clicks regardless of fit. • Exposure bias: an item never shown looks irrelevant. • Feedback loops: the model trains on its own past choices. The missing signal is a scalable semantic reasoning signal to supervise against. ### Distilling graded relevance supervision at scale Audited human labels and taxonomy reconciliation produce a gold set for a lightweight labeler, which grades 100M+ query–item pairs offline for retrieval and ranking. [4:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=282s) [Human annotations, auditing and taxonomy reconciliation produce a training set for a lightweight labeler used to grade query-item pairs.](https://assets.ai.engineer/sites/aiecode2025/sha256/6fe7cb878754c47833944713a24dc52883a88af3756d6e52132789b715ea33d1.webp) Distilling graded relevance supervision at scale. Human annotations, engagement-driven auditing, and Q2T reconciliation build a high-precision gold set. We fine-tune a lightweight LLM on it, then run it offline to label 100M+ query–item pairs, the shared training target for retrieval and ranking. STAGE 1 Build the labeler from human truth one-time, offline 01 · HUMAN SEED Human annotations Query–item pairs hand-labeled on a 3-level guideline: 0 / 1 / 2. ~600K pairs AUDIT 02 · ENGAGEMENT AUDIT Re-judge the suspicious 0s Labeled irrelevant yet top-50% on ATCR / CVR? A larger LLM re-evaluates with granular prompts. GPT-4o · o3 RECONCILE 03 · Q2T RECONCILE Resolve human vs. audit Query-to-Taxonomy predicts the valid categories for the query. If the item’s category is in that set, take the higher label, else keep the lower. Query → Taxonomy GOLD SET 04 · FINE-TUNE Train a lightweight labeler Fine-tune gpt-4o-mini on the audited gold set, sampling to match production label proportions for stable calibration. gpt-4o-mini · fine-tuned Run the fine-tuned labeler offline across the full query × catalog log STAGE 2 Graded supervision at scale, training both models batch, offline 100M+ graded query–item pairs 89% three-class accuracy 98% within-1 · 105K hold-out TRAINS RETRIEVAL Semantic embedding model Mine & Refine · contrastive two-tower RANKING Multi-task ranker ordinal relevance head · P(rel≥1), P(rel≥2) ### Engagement and relevance, one value function An ordinal relevance tower shares bottom layers with click, add-to-cart and conversion towers. A value function blends their predictions; production A/B reports relevance NDCG@10 rising from 0.812 to 0.962. [7:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=477s) [Diagram combining engagement and ordinal relevance towers in a unified ranking score, with A/B results.](https://assets.ai.engineer/sites/aiecode2025/sha256/7716a4f8822a00d3b48f8d2e30c73368e3d846239d7d319da73e632f33718442.webp) Engagement and relevance, one value function. A new ordinal relevance tower joins the engagement towers on a shared bottom. Supervised offline by LLM labels, it adds semantic fit inside the same forward pass, with no online LLM call. Final Ranking Score Unified Value Function CTR Score ATCR Score CVR Score P(rel≥1) “Moderate+” P(rel≥2) “High” CTR Tower ATCR Tower CVR Tower Ordinal Relevance Tower Shared Bottom Layers Query Features Item Features Consumer Features Interaction Features S = α·CTR + β·ATCR + γ·CVR + (1−α−β−γ)·rel The ordinal tower trains on 100M+ LLM-graded labels and preserves label order with P(rel≥1), P(rel≥2). α, β, γ tune the engagement–relevance trade-off, all in one forward pass. PRODUCTION A/B Joint training distilled from LLM-reasoned labels lifts both relevance and engagement. +0.15 relevance NDCG@10 0.812 → 0.962 +1.16% add-to-cart rate +1.10% conversion rate +0.50% gross order value ### Two engines, one pipeline One offline pipeline produces 100M+ graded relevance labels for both the embedding retriever and multi-task ranker, adding semantic relevance while keeping LLMs off the online request path. [8:58](https://www.youtube.com/watch?v=ACPEpji5NV4&t=538s) [Offline supervision trains an embedding retriever and a multi-task ranker, compared with other approaches.](https://assets.ai.engineer/sites/aiecode2025/sha256/dd013795fbee9d803b4cb528aa71e3c945fe8f49a5856914249a1431d38751f7.webp) Two engines, one pipeline. One offline supervision pipeline trains both production engines, the embedding retriever and the multi-task ranker. Each serves at millisecond latency, with no online LLM on the request path. Offline · one supervision pipeline Fine-tune lightweight LLMs on audited human data into 100M+ graded relevance labels, one signal shared by both engines. Online · embedding retriever Two-tower embeddings; graded labels separate the relevance bands for ANN lookup. Online · multi-task ranker Ordinal relevance tower beside the engagement towers; one unified value function. Comparison columns: Engagement & lexical (baselines) | Online LLMs (rerank & retrieve) | Offline-supervised (ours · both engines) Business-metric performance: High | Unknown | High Semantic relevance quality: Low | High | High Online inference latency: Low | High | Low Scalability: High | Low | High Bake the semantic intelligence into the existing fast retrieval and ranking pipelines, offline. ### A compact, hierarchical code per product Residual K-Means gives products hierarchical Semantic IDs: prefixes group broad neighborhoods, while later codes distinguish finer categories. The hot sauce map shows those distinctions emerging without labels. [9:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=586s) [Comparison of SKU IDs, taxonomy and Semantic IDs, with residual clustering and product-neighborhood illustrations.](https://assets.ai.engineer/sites/aiecode2025/sha256/58ac1418eb50e9fc627410efcbf9b067addbde29857c517e2b606a7306d1f572.webp) A compact, hierarchical code per product. Every model has to understand a 55M-item catalog. Raw SKUs say nothing and the human taxonomy is too coarse, so Residual K-Means gives each item a short, hierarchical Semantic ID, learned without supervision. SKU IDENTITY SKU ID Random and unique per item. Useful for joins, uninformative for similarity. [illegible] HUMAN STRUCTURE Backend Taxonomy Curated hierarchy: Sauces › Hot Sauce. Browsable, but rigid and too coarse. Sauces › Hot Sauce LEARNED STRUCTURE Semantic ID Generated from embeddings. Granular, retrainable, comparable by prefix. [237, 483, 46] HOW IT’S BUILT · RESIDUAL K-MEANS K-means Clustering Codebook 1 Codebook 2 Codebook 3 Item semantic identifiers (3, 4, 1) [Other diagram text illegible] Nearest centroid, then residual, then residual. Each item becomes Semantic IDs like (3, 4, 1). [Product-neighborhood map text illegible] Read the topography: Huy Fong, Frank’s, Tabasco, regional Mexican, Caribbean, Korean, without a single label. The structure emerges from the data. ONE SHARED VOCABULARY The same Semantic IDs name every item for every model, from retrieval and ranking to the SLMs that generate content. One catalog language, shared across the stack. ### What Semantic IDs solve Semantic IDs link products across categories and ground query suggestions in the catalog. Online A/B tests showed +4–5% search MRR for ranking and +0.6% search conversion MRR for query reformulation. [11:49](https://www.youtube.com/watch?v=ACPEpji5NV4&t=709s) [Benefits of Semantic IDs and their use in ranking and query reformulation, with A/B metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac11ab73c87cfa47e9e7bb45a857af5f86a66138b3fe42562e0ce69c62d9a449.webp) What Semantic IDs solve. A taxonomy classifies an item; it doesn’t connect it to what a shopper would actually buy alongside it. Semantic IDs add that connective tissue, a shared vocabulary every model can read. Cross-category links Chips, salsa, and guacamole live in different branches; a shared prefix says they belong together. Cold-start coverage A new item gets its SID on day one from title + image, a neighborhood with zero purchases. Tail items surface Sparse-category items inherit signal from their neighbors instead of waiting on volume. Catalog audit When the SID and the label disagree, the label is usually wrong, a free quality signal. TWO PLACES WE USE THEM NEXT 01 · RANKING SIDs as ranker features. Feed each item’s Semantic ID into the multi-task ranker as a categorical feature, so the model reasons about substitutes and complements directly, not just per-item engagement. WHY IT HELPS Taxonomy features were already top-importance in the MTML ranker. Semantic IDs are more granular than taxonomy, so the model learns affinity to a specific type of product rather than a broad category. MEASURED IMPACT · ONLINE A/B +4–5% search MRR +3.48% net-new item conversions +0.28% avg gross order value +0.51% NV add-to-carts / Cx 02 · QUERY UNDERSTANDING SIDs for query reformulation. Map a query to its Semantic ID neighborhood to suggest the next query, grounded in the catalog instead of free-text generation that drifts off-catalog. EXAMPLE “sriracha” → “chili garlic sauce” A real next-query jump, scored by NPMI between their SID clusters. Chili Garlic Sauce · Sambal Oelek · Gochujang Pruned lookup, then ranked by tier and cosine to the query. MEASURED IMPACT · ONLINE A/B +0.6% search conversion MRR ### Embeddings capture behavior. Memory captures intent Language-based memory combines long-term preference blocks, in-session context and stated constraints. New blocks can be added as the system learns more about a shopper. [14:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=856s) [Comparison of implicit embeddings and language-based memory, with three memory timescales and example blocks.](https://assets.ai.engineer/sites/aiecode2025/sha256/272a0a96187ededd00fc2c488aa5114fde48fc4e4d5032a1ee17877345d3d6f6.webp) Embeddings capture behavior. Memory captures intent. An embedding can tell that a consumer buys organic produce, but not why. Memory is the explicit, LLM-native layer, and it operates at three timescales. IMPLICIT Learned inside a user vector, opaque, locked to one model [0.27, -0.84, …] EXPLICIT Memory, written in language, inspectable, reusable “plant-forward” Long-term memory — daily / weekly LLM-synthesized blocks from orders, search, support, browsing. Durable preferences. Real-time context — in-session Cart state, active searches, items viewed or rejected. Overrides history on recency. Stated preferences — agentic Explicit constraints from the assistant. Stable until the consumer changes them. INSIDE LONG-TERM MEMORY · A GROWING STACK OF BLOCKS consolidated each batch Dietary narrative · type · strictness Dining patterns cuisine · behavior · food types Item brand narrative · brand IDs · keywords Item taxonomy narrative · substitutes · keywords Store preferences primary stores · loyalty · reorder Cross-channel complementary · substitution · seasonal + new blocks added over time SAMPLE NARRATIVE · DIETARY BLOCK “Strong affinity for organic produce; prefers premium brands in fresh categories; price-conscious on packaged goods.” “Weekly bulk shopper; consistent weekend ordering pattern; average basket $40–50.” ### How memory is represented and stored Hierarchical memory produces text and a context graph. Both feed context assembly for agents; text and graph embeddings feed retrieval and ranking models. [16:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=985s) [Memory text and hierarchy produce context summaries, semantic embeddings and graph embeddings for assistants and ranking models.](https://assets.ai.engineer/sites/aiecode2025/sha256/45045ede154dc0b0dfebc77fd59eb680f6987cd9a3db8dfdd5cdbaec12df674e.webp) How memory is represented and stored. Memory is stored hierarchically as text with metadata. From it we derive token sequences and a context graph, used for context assembly and summaries that agents read, which in turn generate semantic and GNN embeddings for retrieval and ranking models. STORED Memory object stored as text + a hierarchy, with metadata TOKEN-LEVEL TEXT t₁: “Prefers organic produce...” t₂: “Loyal to local stores...” t₃: “Weekly bulk shopper...” HIERARCHY Maria’s memory Preference Lifestyle Organic Produce Bulk Premium type · date REPRESENTED AS Text t₁ t₂ t₃ ... tT Context graph Maria prefers Organic shops at Ace ASSEMBLED & GENERATED Context assembly & summaries Semantic embedding from text GNN embedding from context graph READ BY Agents & assistants Retrieval & ranking models ### A graph for what embeddings lose Multi-hop connections link consumers to items they have not purchased. Context GNN embeddings improve relative retrieval AUC by 11–13% across taxonomy levels and 3.0% for brands versus production. [17:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1026s) [A consumer context graph accompanies a table showing higher retrieval AUC for the GNN across taxonomy levels and brands.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a2de45c8b5b6c791ec13d3ae1c75e6a2b7d04f41c680c3f23b319bbceab435d.webp) A graph for what embeddings lose, and it lifts retrieval. Consumers, brands, taxonomies, merchants, and extracted keywords as typed nodes. Multi-hop walks reach items never purchased, and the resulting Context GNN embeddings beat the production retriever at every level. Graph nodes: Food Types; Household Keywords; Dietary Keywords; Pets; Life Stage Tags; Consumer; Brand; Cuisine; Taxonomy; Merchant; Keywords Graph relationship labels: prefers; described_by; has; shops_at; sold_by +11–13% relative retrieval AUC from the Context GNN vs. production, across taxonomy levels. LEVEL | TTE | GNN | Δ Taxonomy L1 | 0.741 | 0.825 | +11.3% Taxonomy L2 | 0.729 | 0.821 | +12.7% Taxonomy L3 | 0.725 | 0.818 | +12.8% Taxonomy L4 | 0.724 | 0.816 | +12.7% Brand | 0.701 | 0.722 | +3.0% Closes the sparsity gap Neighborhood smoothing lifts the thin, fine-grained L3/L4 levels up to the denser L1/L2. ### The same inputs power every use case Semantic IDs, memory blocks, prompts and constraints feed a family of models. Their outputs—including ranked item lists and carousel copy—serve conversations, notifications, store pages, search and cart. [18:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1108s) [A diagram connects semantic IDs, memory blocks, prompts, and constraints to a family of models and five product surfaces.](https://assets.ai.engineer/sites/aiecode2025/sha256/d652629e45a5ddfcd6f63b8f44ad66f9510983a1d454fed4916a6a643bcad603.webp) The same inputs power every use case. Three shared inputs feed every use case: Semantic IDs in the vocabulary, memory blocks for conditioning, and graded relevance as the training signal. Catalog Semantics Semantic IDs in the vocabulary, catalog items and consumers input and output as tokens alongside words. Semantic Personalization Memory blocks for conditioning, the consumer represented in language the model reads. Supervision LLM-reasoned signals as the training signal for semantic fit. Steerable Content Generation Ranked SID list, carousel copy, item rationale, grounded rerank. INPUTS Semantic IDs items · brands · categories Memory blocks household · dietary · brand Prompt theme · surface · format Constraints catalog · allergen · fresh A FAMILY OF MODELS Retrieval & ranking models Small language models LLMs & agents OUTPUT SHAPES ranked SID list · query rewrites · grounded rerank carousel title + subcopy · item rationale complement / substitute sets · multi-turn reply Conversational multi-turn reply + SIDs Notifications generated copy + item carousels Store page generative carousels Search query reformulation, understanding, retrieval Cart complements & substitutes ### Offline LLM batch-generation of carousels Batch LLM jobs create and store per-consumer carousel definitions. At request time, metadata lookup and parallel vector and taxonomy retrieval feed ranking and rendering, with no LLM on the request path. [19:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1144s) [A pipeline shows offline carousel generation and storage followed by online retrieval, ranking, and rendering.](https://assets.ai.engineer/sites/aiecode2025/sha256/a83e7dbb216e9e9194a19205fda437b6cd62bcae95806746018f69b956a31a7b.webp) Offline LLM batch-generation of carousels. One instantiation of the primitives. Batch LLM jobs generate per-consumer carousel definitions offline and persist them to a vector store and the metadata store; at request time, retrieval (vector + taxonomy) and ranking assemble the carousel, with no LLM invoked on the request path. OFFLINE WRITE PATH Batch · LLM generates & stores per-consumer carousels 1 Cohort Builder Consumer signals Eligible consumers 2 Batch LLM Generation LLM Gateway Carousel JSON 3 Context Engineering SEARCH INTENTS intent 1 · intent 2 · ... intent N Embedding Model Bulk Embed 4 Store in Vector DB Vector DB vector similarity search Online Metadata Store • Title • Subtitle • Collections • Intent • Taxonomy IDs • Filters lookup by consumer_id ONLINE READ PATH Traditional retrieval & ranking · no LLM on the request path 1 Metadata Lookup Consumer opens store Lookup 2 Parallel Retrieval Vector Search Taxonomy Retrieval 3 Fuse & Rank Ranked items 4 Assemble & Render Organic produce [illegible prices] ### Three takeaways LLMs can capture item meaning and shopper intent offline as labels, semantic IDs and memory. Smaller models can then use these shared representations for retrieval, ranking and generation. [20:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1246s) [Three takeaways address semantic understanding, distilling LLM reasoning into primitives, and reusing shared representations.](https://assets.ai.engineer/sites/aiecode2025/sha256/2431fd057955f297ceed2dcf5d5386437bacba80d6b1925b8797300af4000313.webp) Three takeaways. ONE DISCOVERY IS ALSO A SEMANTIC-UNDERSTANDING PROBLEM For years we optimized engagement signals; LLMs add a new lever, teaching the system what items mean and what a shopper actually intends. TWO DISTILL LLM REASONING INTO PRIMITIVES Capture the reasoning offline as labels, Semantic IDs, and memory, then let smaller, faster models serve it. Online LLM inference is rarely the thing that has to ship. THREE SHARED REPRESENTATIONS, MANY USE CASES Each primitive is a shared understanding and representation that multiple downstream models and use cases reuse, from retrieval and ranking to generation. ### Thank you Raghav Saboo thanks the DoorDash colleagues who built and productionized the systems discussed, and points to AI/ML hiring. [21:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1302s) [DoorDash closing slide thanking collaborators for building the systems discussed and mentioning AI and ML hiring.](https://assets.ai.engineer/sites/aiecode2025/sha256/25bd7ba68986f942a83a8ab0eaa67d50dfb666fa7820a2e1b65f269e8487d1bb.webp) Thank you. This presentation is the culmination of work by many talented colleagues, who built and productionized the systems discussed here. Luming Chen, Jiaqi Xi, Nimesh Sinha, Steven Xu, Zhucheng Zhan, Veronica Sih, Yucong Ji, Neeraj Joshi, Pradeep Muthukrishnan, Saathvik Dirisala, Kyle Hsiao, Akshad Viswanathan, Camilla Zanon di Valgiurata, Camrick Solorio, Claire Tao, Elyse Winer, Kevin Zhai, Prabhjot Saini, Tito Anammah, Vivek Paharia, Aditya Dodda, Ved Sampath, Kenny Chi, Lin Song, Danny Nightingale, and more. WE’RE HIRING · AI / ML careersatdoordash.com/career-areas/ai-ml RAGHAV SABOO · AI / ML, NEW VERTICALS · DOORDASH DOORDASH ### AI Engineer World's Fair [22:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1326s) [AI Engineer World's Fair title on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/b68cd2bc11aacb649972262f8fa408494913f9c519840c0ead6d221b243850a0.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=12s) Thank you everyone for coming to this [0:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=14s) talk and Devanch for organizing this. Uh [0:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=18s) I'm Ragav. I'm a staff uh machine [0:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=20s) learning engineer at Door Dash. I work [0:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=22s) on search and personalization [0:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=24s) and uh today I'm going to talk about uh [0:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=28s) the ways that we're integrating LLMs for [0:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=30s) you know our marketplace discovery uh [0:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=33s) specifically in four pieces and um how [0:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=36s) these primitives are kind of uh uh [0:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=38s) working towards our integration of LLMs [0:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=42s) into traditional uh Rexus as well. So [0:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=45s) I'll start off uh today uh with a simple [0:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=48s) claim. Uh the claim is that to build [0:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=51s) effective discovery in this case for a [0:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=55s) marketplace like Door Dash the real [0:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=57s) bottleneck is semantic understanding. So [1:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=62s) that is knowing what you know actually [1:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=65s) what items mean uh for users given some [1:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=70s) context and what a shopper actually [1:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=73s) intends to do on you know an app like [1:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=76s) Door Dash. And historically we have [1:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=79s) treated these uh as engagement [1:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=82s) optimization problems. Um however LLMs [1:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=85s) give us a genuinely new lever to extend [1:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=89s) this. So this talk is about uh you know [1:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=92s) the four primitives that have allowed us [1:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=95s) to start leveraging LLMs for problems at [1:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=98s) Door Dash. [1:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=101s) As you may know Door Dash is now grown [1:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=104s) uh well beyond restaurants. So we're in [1:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=107s) grocery, retail, pets, gifting and more. [1:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=110s) And our goal is to capture every [1:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=113s) shoppable moment. Uh and we know that [1:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=116s) across these verticals, the users [1:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=119s) frequently have very broad shopping [2:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=121s) missions, right? So for example, a [2:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=124s) consumer may say when they come to our [2:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=126s) app is, you know, I just adopted a [2:08](https://www.youtube.com/watch?v=ACPEpji5NV4&t=128s) puppy. [2:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=130s) What do I need to get started this week? [2:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=133s) And this consumer start may start on [2:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=135s) like searching for puppy food because [2:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=138s) they just adopted the puppy. That one [2:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=141s) query could unfold into further query [2:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=144s) reformulations. [2:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=145s) Um they land on collections for you know [2:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=148s) other shopping needs, complimentary [2:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=151s) items such as crates, leashes, uh [2:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=155s) training pads all the way to checkout. [2:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=158s) And this could happen over multiple [2:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=160s) sessions, mult multiple days, right? And [2:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=164s) capturing that whole arc is pretty hard [2:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=166s) because it spans multiple models. Um, [2:49](https://www.youtube.com/watch?v=ACPEpji5NV4&t=169s) and this is where the four primitives I [2:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=172s) will cover come in. [2:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=175s) Specifically, supervision, catalog [2:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=177s) semantics, semantic personalization, [3:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=181s) and steerable content generation. As we [3:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=183s) heard previously as well, it's a system [3:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=186s) of shared representations that help us [3:08](https://www.youtube.com/watch?v=ACPEpji5NV4&t=188s) map this customer's journey uh with the [3:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=192s) use of LLMs. [3:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=194s) Now, with supervision, the question [3:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=196s) we're asking or looking to address is [3:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=199s) how do we teach our retrieval and [3:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=201s) ranking systems what good means for [3:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=204s) different tasks? And here's a simple [3:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=206s) example, right? A shopper searches for [3:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=208s) gluten-free pasta. And if we only rank [3:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=211s) from engagement, very popular regular [3:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=213s) spaghettes might show up and because it [3:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=216s) sells well, you know, it may continue to [3:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=218s) show up. Even gluten-free white bread [3:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=221s) might be retrieved and, you know, show [3:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=224s) up because it overlaps with gluten-free. [3:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=226s) But neither of these are clearly the [3:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=228s) right answer. What we actually want is [3:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=231s) graded relevance. A true, you know, [3:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=234s) gluten-free pasta should be a high [3:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=236s) relevance item. Chickpea pasta might be [3:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=239s) a reasonable substitute and regular [4:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=242s) spaghetti while it's popular it fails [4:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=244s) the constraint that the user has. Now to [4:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=247s) solve this there are obviously two [4:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=250s) sources of truth. We could look at human [4:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=252s) annotation [4:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=254s) but that's expensive slow and becomes [4:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=257s) quickly stale because our catalog [4:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=259s) changes quite rapidly. [4:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=262s) Second is behavioral signals. They're [4:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=263s) abundant but obviously as I mentioned [4:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=265s) biased by exposure, position, price, [4:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=269s) promotions [4:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=271s) and you know previous model choices. So [4:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=273s) the missing signal is a scalable [4:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=276s) reasoning signal. This is where LLMs are [4:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=279s) very useful as they offer a way to [4:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=281s) produce that supervision for such tasks. [4:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=284s) So in our case, you know, the PIP [4:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=286s) pipeline starts with human ground [4:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=288s) truths. uh we build a high quality seed [4:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=291s) set of example query item pairs and on a [4:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=295s) three three uh level relevant scale 012. [5:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=300s) We audit the suspicious cases. For [5:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=302s) example, if a human label says you know [5:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=304s) that an item is irrelevant but that item [5:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=307s) performs very well on add to cart or [5:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=309s) conversions, we send that case to a [5:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=312s) stronger LLM uh to re-evaluate with more [5:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=315s) granular prompts. [5:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=317s) We also reconcile with other models. You [5:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=320s) know, for example, our query to taxonomy [5:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=322s) or category models. And if a query, you [5:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=325s) know, query maps to a set of valid [5:27](https://www.youtube.com/watch?v=ACPEpji5NV4&t=327s) categories and the items belong to those [5:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=329s) valid categories, we may adjust the [5:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=332s) label accordingly. And through this [5:34](https://www.youtube.com/watch?v=ACPEpji5NV4&t=334s) process, we're able to achieve a pretty [5:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=336s) high precision golden data set that we [5:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=339s) then fine-tune a lightweight LLM, let's [5:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=342s) say, for example, a GPD 40 mini on top [5:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=345s) of it. And that's when it, you know, [5:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=348s) really pays off. We are able to use that [5:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=351s) fine-tuned labeler offline across, you [5:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=354s) know, our full catalog to generate uh [5:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=357s) full graded query item pairs. And those [5:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=359s) labels become one shared target for both [6:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=362s) retrieval and ranking systems. This [6:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=365s) pattern has been pretty important for us [6:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=367s) over the past uh few years. um use you [6:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=370s) know expensive reasoning once offline [6:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=374s) then distill it into models that can you [6:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=377s) know be served cheaply and quickly. [6:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=382s) Now let's take retrieval first. The [6:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=384s) challenge is you know standard [6:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=385s) embeddings um at e-commerce scale [6:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=388s) quickly collapse the relevance uh [6:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=391s) distinction. items that are merely [6:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=393s) related sit close to items that actually [6:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=396s) match and the model uh may know two [6:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=399s) items are related but not be able to [6:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=401s) separate uh exact matches, substitutes, [6:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=404s) compliments. [6:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=406s) So our fix is a two-stage contrastive [6:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=408s) method um trained on uh the same graded [6:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=411s) labels. Stage one, which we call mining, [6:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=415s) does a global geometry shaping. We use [6:58](https://www.youtube.com/watch?v=ACPEpji5NV4&t=418s) two tower encoders uh with multi-level [7:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=420s) supervised contrastive loss. Um after [7:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=423s) that we use that base model to mine [7:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=427s) harder negatives and uh these are items [7:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=430s) that the model confuses hard negatives [7:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=432s) that are ranked too high or strong [7:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=435s) positives ranked too low. And the second [7:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=437s) stage after that is where we really put [7:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=442s) the model through curriculum training. [7:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=444s) uh use those hard negatives, relabel [7:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=446s) them and our uh you know relevance of [7:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=450s) the retrieval layer improves sharply [7:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=453s) after that. So after stage one we see [7:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=455s) overlaps between you know relevant and [7:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=458s) uh moderately relevant but after stage [7:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=460s) two there is a significant difference [7:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=463s) and this has been one of like our [7:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=464s) biggest uh levers in retrieval [7:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=467s) improvement uh across the board. um you [7:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=470s) know we've improved NDCG of relevance by [7:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=473s) 2.3%. [7:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=474s) And uh that's also been true with our [7:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=477s) downstream business metrics [7:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=479s) and we did the same with uh uh exercise [8:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=483s) with our ranking models as well [8:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=484s) distilling LM reason labels into our [8:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=487s) rankers. However, combining this with uh [8:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=491s) business and engagement objectives. [8:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=495s) Here we add a new tower uh in this case [8:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=497s) an ordinal uh relevance tower on top of [8:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=500s) the LLM graded labels and uh this [8:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=505s) prediction sits right alongside existing [8:27](https://www.youtube.com/watch?v=ACPEpji5NV4&t=507s) engagement towers uh click add to cart [8:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=510s) and conversion and since they share the [8:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=512s) same uh bottom layers the semantic fit [8:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=515s) and relevance fit gets distilled in the [8:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=518s) same backward pass and uh our model is [8:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=521s) able to predict uh you know probability [8:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=523s) of relevance across the different levels [8:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=527s) and we are able to blend uh through a [8:49](https://www.youtube.com/watch?v=ACPEpji5NV4&t=529s) value function on on top of this that [8:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=532s) allows us to uh fine-tune between [8:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=534s) engagement and relevance for different [8:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=536s) surfaces. [8:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=539s) Now these two examples are part of a [9:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=542s) larger theme of our work across similar [9:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=544s) projects uh where it seems right. We're [9:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=547s) not replacing the retrieval and ranking [9:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=549s) system with an LLM, but rather [9:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=551s) distilling the uh reasoning and [9:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=553s) understanding into some production [9:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=555s) ranking architecture. And this allows us [9:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=558s) to kind of quickly test and also [9:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=560s) scalably improve our models uh with LLMs [9:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=563s) while keeping uh you know the business [9:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=565s) and system objectives uh of these [9:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=568s) surfaces in mind. [9:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=571s) So once we have supervision, the next [9:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=573s) question is how do we represent a very [9:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=576s) large and constantly changing catalog uh [9:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=579s) in a way that every model be it language [9:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=581s) models or uh you know traditional [9:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=584s) retrieval and ranking models uh that [9:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=586s) they can understand and at you know Door [9:49](https://www.youtube.com/watch?v=ACPEpji5NV4&t=589s) Dash's scale uh you can imagine we're [9:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=591s) kind of surfacing items across stores [9:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=593s) across internationally actually uh now [9:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=597s) and you know that's easily a few billion [9:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=599s) items at the store item level and for [10:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=602s) these items you obviously have unique [10:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=604s) skew ids but it says nothing [10:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=606s) semantically. So we also have a human [10:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=609s) curated taxonomy that helps us classify [10:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=611s) and categorize these items into [10:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=614s) meaningful spaces. But often this is too [10:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=617s) coarse and too rigid. [10:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=619s) So for example something like hot sauces [10:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=622s) may just have like sauces hot sauces in [10:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=624s) the taxonomy [10:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=626s) uh but it doesn't tell you anything [10:27](https://www.youtube.com/watch?v=ACPEpji5NV4&t=627s) about the relationship between hoof or [10:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=630s) franks or tobasco. [10:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=633s) So that's where we've also adapted uh or [10:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=636s) adopted um semantic ids. Uh it's been [10:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=640s) introduced previously already. It's a [10:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=642s) backbone for a lot of our uh work uh [10:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=646s) now. And uh what it does is it gives us [10:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=650s) a you know a short hierarchical code [10:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=652s) that's analogous to our taxonomy but uh [10:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=656s) the ability to control the fine grain [10:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=659s) nature of it. And the result of this [11:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=661s) learn taxonomy is that each item gets [11:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=665s) this hierarchical code and the prefix [11:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=667s) captures broad neighborhoods and you [11:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=671s) know the later tokens capture finer [11:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=673s) distinctions. [11:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=675s) So for example in the map over on on the [11:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=678s) slide you'll see hot sauces and the [11:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=680s) structure emerging from the data with [11:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=683s) zero labels. each one share the same [11:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=686s) first and second prefix in this uh ID [11:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=689s) sequence but then split by specialtity. [11:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=692s) So now you've got Mexican, Caribbean, [11:34](https://www.youtube.com/watch?v=ACPEpji5NV4&t=694s) Korean hot sauces uh being split up. And [11:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=699s) the important property is that this code [11:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=701s) that's learned is you know comparable by [11:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=703s) as I said by prefix but also usable by [11:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=707s) all of our downstream models [11:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=710s) and that unlocks a few things for us. So [11:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=713s) the first is cross-category comparisons [11:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=715s) or relationships. You know chips, salsa, [11:58](https://www.youtube.com/watch?v=ACPEpji5NV4&t=718s) guacamole may live in different taxonomy [12:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=721s) branches but a shared semantic ID [12:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=723s) neighborhood can uh indicate that they [12:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=725s) belong together and maybe a shopping [12:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=727s) mission. Second cold start problems. uh [12:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=731s) oftentimes we get new catalog items [12:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=733s) added by stores and with techniques like [12:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=736s) engram and bite parent coding of these [12:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=739s) tokens we're able to scalably add these [12:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=741s) to our models as sparse ID features [12:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=744s) third tail coverage so sparse items uh [12:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=749s) and uh inherit a lot of signal from [12:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=752s) semantically related items and uh you [12:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=755s) know we don't have to wait for the [12:37](https://www.youtube.com/watch?v=ACPEpji5NV4&t=757s) volume or exposure uh to you um for to [12:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=760s) consumers for these items. And then [12:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=763s) lastly, nice to have is the fact that we [12:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=766s) can actually also do a reverse audit. So [12:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=768s) we can audit our catalog and how well [12:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=770s) our human labels agree with the [12:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=772s) semantically learned labels. [12:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=776s) So a couple of examples of where we're [12:58](https://www.youtube.com/watch?v=ACPEpji5NV4&t=778s) using it today uh just to get give some [13:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=780s) real metrics. Um it has been one of our [13:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=784s) biggest improvements to our ranker. Um [13:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=786s) in in this case it's actually improved [13:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=789s) our uh ranker by uh improving MRR [13:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=792s) between four to 5% and that's translated [13:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=795s) to pretty big conversion uh wins as [13:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=797s) well. The second one that I'm [13:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=800s) particularly [13:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=801s) fond of is query reformulation. So again [13:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=804s) going back to the hot sauce example, [13:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=806s) Sriracha can lead to garlic chili garlic [13:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=809s) sauce or sambell Ole and because these [13:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=813s) queries map onto like our catalog [13:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=816s) grounded semantic neighborhood uh that [13:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=820s) results in much more relevant queries [13:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=822s) that we're suggesting to users. Um [13:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=825s) ultimately if these are queries that we [13:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=827s) don't have inventory for on Door Dash, [13:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=830s) they're meaningless. And with the [13:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=832s) addition of semantic ids in this query [13:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=834s) graph, we were actually able to see [13:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=836s) pretty massive um MR gains as well uh [13:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=839s) for for this piece of work. [14:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=843s) So for now we've talked about you know [14:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=845s) item mapping and relevance and the next [14:08](https://www.youtube.com/watch?v=ACPEpji5NV4&t=848s) question is consumer context. What does [14:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=850s) the system know about the shopper and [14:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=854s) can that knowledge be reused across [14:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=856s) models again? And this is where uh our [14:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=859s) third primitive comes in memory. So [14:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=862s) memory in the agent context is pretty [14:25](https://www.youtube.com/watch?v=ACPEpji5NV4&t=865s) well understood now. Um we apply the [14:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=868s) same concepts to uh our recommendation [14:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=872s) systems. So user embeddings are clearly [14:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=875s) very useful. uh we have many [14:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=876s) representations of our consumers but [14:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=879s) they don't necessarily get to uh why a [14:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=883s) consumer may have certain intents um and [14:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=886s) why they might have certain preferences. [14:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=890s) Additionally, LLMs cannot uh readily use [14:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=893s) these uh embeddings. You could tokenize [14:55](https://www.youtube.com/watch?v=ACPEpji5NV4&t=895s) and like uh train these models to learn [14:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=897s) it. But oftentimes having some explicit [15:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=901s) LM native counterpart is very useful and [15:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=903s) that's what we found. So the idea of [15:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=906s) memory is to represent uh the consumer [15:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=909s) in multiple forms semantic in you know [15:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=913s) inspectable and reusable [15:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=916s) and uh the way we think about memory is [15:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=920s) uh in three time scales. So long-term [15:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=923s) memory captures durable preferences from [15:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=926s) orders, searches, browsing, support [15:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=929s) interactions. [15:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=931s) Real-time context captures uh agentic [15:34](https://www.youtube.com/watch?v=ACPEpji5NV4&t=934s) inter interactions um oh sorry in [15:37](https://www.youtube.com/watch?v=ACPEpji5NV4&t=937s) session interactions so cart state [15:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=940s) active searches and then we have stated [15:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=942s) preferences which come from agentic [15:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=945s) interactions so something like ask door [15:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=947s) dash for example where consumers are [15:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=950s) able to explicitly state their [15:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=951s) constraints and preferences as well [15:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=954s) and the way we represent consumers in uh [15:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=957s) in uh this long-term memory is through [16:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=961s) uh memory blocks. And these are [16:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=963s) structured in a way that allows uh us to [16:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=966s) add new dimensions of the user, you [16:08](https://www.youtube.com/watch?v=ACPEpji5NV4&t=968s) know, from dietary preferences to dining [16:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=970s) preferences to uh substitute pre uh [16:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=973s) preferences um as we learn more about [16:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=976s) the consumer and that is uh decoupled [16:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=979s) from like the downstream system that u [16:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=982s) that doesn't need to reinterpret uh this [16:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=984s) for their own use cases. [16:27](https://www.youtube.com/watch?v=ACPEpji5NV4&t=987s) So each consumer's memory is multi [16:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=989s) materialized in multiple forms. So first [16:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=992s) is text that's human readable. It's uh [16:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=995s) captures the consumer in a compact way. [16:37](https://www.youtube.com/watch?v=ACPEpji5NV4&t=997s) Second is are latent vectors embeddings [16:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1000s) of those memory blocks that can be fed [16:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1002s) into retrieval and ranking systems. And [16:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1005s) third are graph and tree or hierarchical [16:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1008s) approaches. So uh you know type [16:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1011s) relationships between consumers and [16:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1013s) brands taxonomies and memory uh revealed [16:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1017s) um preferences [17:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1020s) and these are assembled into you know [17:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1022s) composite memory objects that both ML [17:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1025s) models and LLM use downstream. So an [17:08](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1028s) example is you know this context graph [17:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1030s) uh where we connect consumers with [17:12](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1032s) extracted memory concepts and a graph [17:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1036s) needs fits neatly into shopping journeys [17:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1038s) because you know oftent times uh [17:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1041s) consumer item interactions are sparse [17:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1043s) and multihop. And in this case these [17:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1046s) context graphs are able to link [17:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1048s) consumers across memory concepts that we [17:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1050s) previously did not have uh relationships [17:34](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1054s) for. And it's particularly helpful uh at [17:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1058s) fine grain taxonomy levels. In our case, [17:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1061s) we're seeing uh for retrieval uh uh [17:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1064s) using uh graph uh based embeddings [17:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1067s) outperforming our uh existing taxonomy [17:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1071s) based embeddings. [17:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1074s) And today this memory framework shows up [17:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1076s) in three places. The first is [17:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1079s) personalized collections. I'll talk [18:01](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1081s) about that a bit later. Uh second is [18:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1083s) agentic personalization. So ask door [18:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1085s) dash for example uses this to [18:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1087s) personalize its sessions uh for you. And [18:10](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1090s) third in uh retrieval and ranking models [18:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1093s) as I said we encode these memory blocks [18:15](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1095s) and feed them as features in in [18:18](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1098s) downstream models as well. [18:22](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1102s) So the fourth primitive is steerable [18:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1104s) content generation. We heard quite a bit [18:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1106s) about it in the previous talk and this [18:30](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1110s) slide is my take on how we're p piecing [18:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1112s) all these uh primitives together. So all [18:37](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1117s) of these inputs, semantic ids, memory [18:39](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1119s) blocks um and graded relevance or um LLM [18:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1123s) supervision feed into multiple models [18:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1126s) you know be it LLMs or small language [18:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1128s) models or um uh you know traditional [18:52](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1132s) models and from these inputs we can [18:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1134s) generate different output shapes you [18:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1136s) know ranked semantic ID lists, carousel [18:58](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1138s) titles, subcopies [19:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1140s) and these show up across different [19:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1142s) surfaces today on Gord Dash and A direct [19:06](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1146s) example of this is uh our um [19:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1149s) personalized collections on store pages. [19:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1151s) So historically collections on store [19:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1154s) pages uh on Door Dash have been a fixed [19:17](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1157s) library or attribute-based [19:20](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1160s) um and we've been able to kind of expand [19:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1163s) that through uh consumer level [19:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1166s) collection generation. And today this is [19:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1169s) an offline uh LLM process where we take [19:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1172s) in consumer memory semantic ids and syni [19:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1175s) syn synthesize uh collections um all the [19:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1178s) way from title subtitle to the actual [19:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1180s) items. [19:42](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1182s) And the important part is that control [19:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1184s) and steerability. We're able to react to [19:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1187s) occasions and moments um and build those [19:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1190s) collections as needed uh for different [19:54](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1194s) consumers at serving time. uh because [19:57](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1197s) this is all batch and uh generated [20:00](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1200s) offline through LLMs, we are able to [20:03](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1203s) still use our uh existing retrieval and [20:05](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1205s) ranking stack for item hydration, [20:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1207s) collection ranking, etc. [20:11](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1211s) And you know this is for example what a [20:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1213s) shopper actually sees, right? For a [20:14](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1214s) shopper with plant-based affinity, the [20:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1216s) system can generate plant-based pantry [20:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1219s) rows. For cat-only households, it can [20:21](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1221s) generate cat dry food rows. And if a [20:24](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1224s) session indicates that you know a person [20:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1226s) is going through pantry restocking it [20:28](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1228s) can bias towards pantry staples. And our [20:31](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1231s) early tests show that you know consumers [20:33](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1233s) are feeling the benefits of these [20:35](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1235s) tailored experiences. Uh an example is [20:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1238s) uh within our pets vertical we've been [20:40](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1240s) able to drive uh close to 1% uh order [20:44](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1244s) rate increases and 6% in uh active [20:46](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1246s) users. [20:48](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1248s) So to conclude uh there are three [20:51](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1251s) takeaways I would love for you to take [20:53](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1253s) away from this talk and how LLM fit into [20:56](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1256s) search and recommendations. First [20:59](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1259s) discovery is a semantic understanding [21:02](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1262s) problem. It's not only about engagement. [21:04](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1264s) LMS give us a way to reason about item [21:07](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1267s) meaning and shopper intent. Uh in in [21:09](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1269s) Door Dash's case for example. Second uh [21:13](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1273s) distill LM reasoning into primitives. [21:16](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1276s) capture you know reasoning uh offline as [21:19](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1279s) labels, semantic ids, memory and then [21:23](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1283s) let smaller and faster models serve it. [21:26](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1286s) The online LLM call is often not the [21:29](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1289s) product architecture you need. Third, [21:32](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1292s) shared representations create many use [21:34](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1294s) cases. Uh once you have these [21:36](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1296s) primitives, they can power, you know, [21:38](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1298s) retrieval, ranking, content generation, [21:41](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1301s) etc. [21:43](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1303s) So last but not least, thank you and [21:45](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1305s) thank you to all the collaborators at [21:47](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1307s) Door Dash who have helped ship a lot of [21:50](https://www.youtube.com/watch?v=ACPEpji5NV4&t=1310s) these things as well.