[
  {
    "startMs": 12480,
    "text": "Thank you everyone for coming to this"
  },
  {
    "startMs": 14799,
    "text": "talk and Devanch for organizing this. Uh"
  },
  {
    "startMs": 18080,
    "text": "I'm Ragav. I'm a staff uh machine"
  },
  {
    "startMs": 20320,
    "text": "learning engineer at Door Dash. I work"
  },
  {
    "startMs": 22080,
    "text": "on search and personalization"
  },
  {
    "startMs": 24640,
    "text": "and uh today I'm going to talk about uh"
  },
  {
    "startMs": 28000,
    "text": "the ways that we're integrating LLMs for"
  },
  {
    "startMs": 30560,
    "text": "you know our marketplace discovery uh"
  },
  {
    "startMs": 33040,
    "text": "specifically in four pieces and um how"
  },
  {
    "startMs": 36160,
    "text": "these primitives are kind of uh uh"
  },
  {
    "startMs": 38960,
    "text": "working towards our integration of LLMs"
  },
  {
    "startMs": 42000,
    "text": "into traditional uh Rexus as well. So"
  },
  {
    "startMs": 45760,
    "text": "I'll start off uh today uh with a simple"
  },
  {
    "startMs": 48399,
    "text": "claim. Uh the claim is that to build"
  },
  {
    "startMs": 51920,
    "text": "effective discovery in this case for a"
  },
  {
    "startMs": 55039,
    "text": "marketplace like Door Dash the real"
  },
  {
    "startMs": 57440,
    "text": "bottleneck is semantic understanding. So"
  },
  {
    "startMs": 62160,
    "text": "that is knowing what you know actually"
  },
  {
    "startMs": 65760,
    "text": "what items mean uh for users given some"
  },
  {
    "startMs": 70320,
    "text": "context and what a shopper actually"
  },
  {
    "startMs": 73439,
    "text": "intends to do on you know an app like"
  },
  {
    "startMs": 76240,
    "text": "Door Dash. And historically we have"
  },
  {
    "startMs": 79200,
    "text": "treated these uh as engagement"
  },
  {
    "startMs": 82080,
    "text": "optimization problems. Um however LLMs"
  },
  {
    "startMs": 85840,
    "text": "give us a genuinely new lever to extend"
  },
  {
    "startMs": 89520,
    "text": "this. So this talk is about uh you know"
  },
  {
    "startMs": 92720,
    "text": "the four primitives that have allowed us"
  },
  {
    "startMs": 95520,
    "text": "to start leveraging LLMs for problems at"
  },
  {
    "startMs": 98960,
    "text": "Door Dash."
  },
  {
    "startMs": 101040,
    "text": "As you may know Door Dash is now grown"
  },
  {
    "startMs": 104479,
    "text": "uh well beyond restaurants. So we're in"
  },
  {
    "startMs": 107119,
    "text": "grocery, retail, pets, gifting and more."
  },
  {
    "startMs": 110720,
    "text": "And our goal is to capture every"
  },
  {
    "startMs": 113520,
    "text": "shoppable moment. Uh and we know that"
  },
  {
    "startMs": 116799,
    "text": "across these verticals, the users"
  },
  {
    "startMs": 119119,
    "text": "frequently have very broad shopping"
  },
  {
    "startMs": 121680,
    "text": "missions, right? So for example, a"
  },
  {
    "startMs": 124320,
    "text": "consumer may say when they come to our"
  },
  {
    "startMs": 126799,
    "text": "app is, you know, I just adopted a"
  },
  {
    "startMs": 128959,
    "text": "puppy."
  },
  {
    "startMs": 130479,
    "text": "What do I need to get started this week?"
  },
  {
    "startMs": 133360,
    "text": "And this consumer start may start on"
  },
  {
    "startMs": 135760,
    "text": "like searching for puppy food because"
  },
  {
    "startMs": 138319,
    "text": "they just adopted the puppy. That one"
  },
  {
    "startMs": 141200,
    "text": "query could unfold into further query"
  },
  {
    "startMs": 144000,
    "text": "reformulations."
  },
  {
    "startMs": 145760,
    "text": "Um they land on collections for you know"
  },
  {
    "startMs": 148640,
    "text": "other shopping needs, complimentary"
  },
  {
    "startMs": 151120,
    "text": "items such as crates, leashes, uh"
  },
  {
    "startMs": 155040,
    "text": "training pads all the way to checkout."
  },
  {
    "startMs": 158560,
    "text": "And this could happen over multiple"
  },
  {
    "startMs": 160560,
    "text": "sessions, mult multiple days, right? And"
  },
  {
    "startMs": 164000,
    "text": "capturing that whole arc is pretty hard"
  },
  {
    "startMs": 166400,
    "text": "because it spans multiple models. Um,"
  },
  {
    "startMs": 169680,
    "text": "and this is where the four primitives I"
  },
  {
    "startMs": 172720,
    "text": "will cover come in."
  },
  {
    "startMs": 175440,
    "text": "Specifically, supervision, catalog"
  },
  {
    "startMs": 177840,
    "text": "semantics, semantic personalization,"
  },
  {
    "startMs": 181280,
    "text": "and steerable content generation. As we"
  },
  {
    "startMs": 183519,
    "text": "heard previously as well, it's a system"
  },
  {
    "startMs": 186560,
    "text": "of shared representations that help us"
  },
  {
    "startMs": 188959,
    "text": "map this customer's journey uh with the"
  },
  {
    "startMs": 192080,
    "text": "use of LLMs."
  },
  {
    "startMs": 194879,
    "text": "Now, with supervision, the question"
  },
  {
    "startMs": 196720,
    "text": "we're asking or looking to address is"
  },
  {
    "startMs": 199360,
    "text": "how do we teach our retrieval and"
  },
  {
    "startMs": 201200,
    "text": "ranking systems what good means for"
  },
  {
    "startMs": 204080,
    "text": "different tasks? And here's a simple"
  },
  {
    "startMs": 206080,
    "text": "example, right? A shopper searches for"
  },
  {
    "startMs": 208480,
    "text": "gluten-free pasta. And if we only rank"
  },
  {
    "startMs": 211200,
    "text": "from engagement, very popular regular"
  },
  {
    "startMs": 213680,
    "text": "spaghettes might show up and because it"
  },
  {
    "startMs": 216159,
    "text": "sells well, you know, it may continue to"
  },
  {
    "startMs": 218879,
    "text": "show up. Even gluten-free white bread"
  },
  {
    "startMs": 221680,
    "text": "might be retrieved and, you know, show"
  },
  {
    "startMs": 224000,
    "text": "up because it overlaps with gluten-free."
  },
  {
    "startMs": 226480,
    "text": "But neither of these are clearly the"
  },
  {
    "startMs": 228799,
    "text": "right answer. What we actually want is"
  },
  {
    "startMs": 231599,
    "text": "graded relevance. A true, you know,"
  },
  {
    "startMs": 234400,
    "text": "gluten-free pasta should be a high"
  },
  {
    "startMs": 236959,
    "text": "relevance item. Chickpea pasta might be"
  },
  {
    "startMs": 239360,
    "text": "a reasonable substitute and regular"
  },
  {
    "startMs": 242159,
    "text": "spaghetti while it's popular it fails"
  },
  {
    "startMs": 244480,
    "text": "the constraint that the user has. Now to"
  },
  {
    "startMs": 247680,
    "text": "solve this there are obviously two"
  },
  {
    "startMs": 250000,
    "text": "sources of truth. We could look at human"
  },
  {
    "startMs": 252480,
    "text": "annotation"
  },
  {
    "startMs": 254239,
    "text": "but that's expensive slow and becomes"
  },
  {
    "startMs": 257759,
    "text": "quickly stale because our catalog"
  },
  {
    "startMs": 259359,
    "text": "changes quite rapidly."
  },
  {
    "startMs": 262079,
    "text": "Second is behavioral signals. They're"
  },
  {
    "startMs": 263680,
    "text": "abundant but obviously as I mentioned"
  },
  {
    "startMs": 265919,
    "text": "biased by exposure, position, price,"
  },
  {
    "startMs": 269759,
    "text": "promotions"
  },
  {
    "startMs": 271360,
    "text": "and you know previous model choices. So"
  },
  {
    "startMs": 273759,
    "text": "the missing signal is a scalable"
  },
  {
    "startMs": 276080,
    "text": "reasoning signal. This is where LLMs are"
  },
  {
    "startMs": 279280,
    "text": "very useful as they offer a way to"
  },
  {
    "startMs": 281680,
    "text": "produce that supervision for such tasks."
  },
  {
    "startMs": 284639,
    "text": "So in our case, you know, the PIP"
  },
  {
    "startMs": 286320,
    "text": "pipeline starts with human ground"
  },
  {
    "startMs": 288880,
    "text": "truths. uh we build a high quality seed"
  },
  {
    "startMs": 291840,
    "text": "set of example query item pairs and on a"
  },
  {
    "startMs": 295440,
    "text": "three three uh level relevant scale 012."
  },
  {
    "startMs": 300320,
    "text": "We audit the suspicious cases. For"
  },
  {
    "startMs": 302880,
    "text": "example, if a human label says you know"
  },
  {
    "startMs": 304800,
    "text": "that an item is irrelevant but that item"
  },
  {
    "startMs": 307199,
    "text": "performs very well on add to cart or"
  },
  {
    "startMs": 309759,
    "text": "conversions, we send that case to a"
  },
  {
    "startMs": 312479,
    "text": "stronger LLM uh to re-evaluate with more"
  },
  {
    "startMs": 315440,
    "text": "granular prompts."
  },
  {
    "startMs": 317520,
    "text": "We also reconcile with other models. You"
  },
  {
    "startMs": 320479,
    "text": "know, for example, our query to taxonomy"
  },
  {
    "startMs": 322560,
    "text": "or category models. And if a query, you"
  },
  {
    "startMs": 325440,
    "text": "know, query maps to a set of valid"
  },
  {
    "startMs": 327759,
    "text": "categories and the items belong to those"
  },
  {
    "startMs": 329759,
    "text": "valid categories, we may adjust the"
  },
  {
    "startMs": 332080,
    "text": "label accordingly. And through this"
  },
  {
    "startMs": 334560,
    "text": "process, we're able to achieve a pretty"
  },
  {
    "startMs": 336320,
    "text": "high precision golden data set that we"
  },
  {
    "startMs": 339520,
    "text": "then fine-tune a lightweight LLM, let's"
  },
  {
    "startMs": 342400,
    "text": "say, for example, a GPD 40 mini on top"
  },
  {
    "startMs": 345919,
    "text": "of it. And that's when it, you know,"
  },
  {
    "startMs": 348800,
    "text": "really pays off. We are able to use that"
  },
  {
    "startMs": 351840,
    "text": "fine-tuned labeler offline across, you"
  },
  {
    "startMs": 354320,
    "text": "know, our full catalog to generate uh"
  },
  {
    "startMs": 357360,
    "text": "full graded query item pairs. And those"
  },
  {
    "startMs": 359440,
    "text": "labels become one shared target for both"
  },
  {
    "startMs": 362400,
    "text": "retrieval and ranking systems. This"
  },
  {
    "startMs": 365039,
    "text": "pattern has been pretty important for us"
  },
  {
    "startMs": 367280,
    "text": "over the past uh few years. um use you"
  },
  {
    "startMs": 370880,
    "text": "know expensive reasoning once offline"
  },
  {
    "startMs": 374880,
    "text": "then distill it into models that can you"
  },
  {
    "startMs": 377520,
    "text": "know be served cheaply and quickly."
  },
  {
    "startMs": 382160,
    "text": "Now let's take retrieval first. The"
  },
  {
    "startMs": 384240,
    "text": "challenge is you know standard"
  },
  {
    "startMs": 385600,
    "text": "embeddings um at e-commerce scale"
  },
  {
    "startMs": 388880,
    "text": "quickly collapse the relevance uh"
  },
  {
    "startMs": 391360,
    "text": "distinction. items that are merely"
  },
  {
    "startMs": 393759,
    "text": "related sit close to items that actually"
  },
  {
    "startMs": 396400,
    "text": "match and the model uh may know two"
  },
  {
    "startMs": 399520,
    "text": "items are related but not be able to"
  },
  {
    "startMs": 401759,
    "text": "separate uh exact matches, substitutes,"
  },
  {
    "startMs": 404319,
    "text": "compliments."
  },
  {
    "startMs": 406000,
    "text": "So our fix is a two-stage contrastive"
  },
  {
    "startMs": 408160,
    "text": "method um trained on uh the same graded"
  },
  {
    "startMs": 411360,
    "text": "labels. Stage one, which we call mining,"
  },
  {
    "startMs": 415199,
    "text": "does a global geometry shaping. We use"
  },
  {
    "startMs": 418000,
    "text": "two tower encoders uh with multi-level"
  },
  {
    "startMs": 420560,
    "text": "supervised contrastive loss. Um after"
  },
  {
    "startMs": 423759,
    "text": "that we use that base model to mine"
  },
  {
    "startMs": 427039,
    "text": "harder negatives and uh these are items"
  },
  {
    "startMs": 430639,
    "text": "that the model confuses hard negatives"
  },
  {
    "startMs": 432880,
    "text": "that are ranked too high or strong"
  },
  {
    "startMs": 435199,
    "text": "positives ranked too low. And the second"
  },
  {
    "startMs": 437840,
    "text": "stage after that is where we really put"
  },
  {
    "startMs": 442319,
    "text": "the model through curriculum training."
  },
  {
    "startMs": 444400,
    "text": "uh use those hard negatives, relabel"
  },
  {
    "startMs": 446400,
    "text": "them and our uh you know relevance of"
  },
  {
    "startMs": 450400,
    "text": "the retrieval layer improves sharply"
  },
  {
    "startMs": 453039,
    "text": "after that. So after stage one we see"
  },
  {
    "startMs": 455840,
    "text": "overlaps between you know relevant and"
  },
  {
    "startMs": 458479,
    "text": "uh moderately relevant but after stage"
  },
  {
    "startMs": 460960,
    "text": "two there is a significant difference"
  },
  {
    "startMs": 463120,
    "text": "and this has been one of like our"
  },
  {
    "startMs": 464960,
    "text": "biggest uh levers in retrieval"
  },
  {
    "startMs": 467680,
    "text": "improvement uh across the board. um you"
  },
  {
    "startMs": 470880,
    "text": "know we've improved NDCG of relevance by"
  },
  {
    "startMs": 473199,
    "text": "2.3%."
  },
  {
    "startMs": 474879,
    "text": "And uh that's also been true with our"
  },
  {
    "startMs": 477199,
    "text": "downstream business metrics"
  },
  {
    "startMs": 479919,
    "text": "and we did the same with uh uh exercise"
  },
  {
    "startMs": 483280,
    "text": "with our ranking models as well"
  },
  {
    "startMs": 484879,
    "text": "distilling LM reason labels into our"
  },
  {
    "startMs": 487759,
    "text": "rankers. However, combining this with uh"
  },
  {
    "startMs": 491840,
    "text": "business and engagement objectives."
  },
  {
    "startMs": 495039,
    "text": "Here we add a new tower uh in this case"
  },
  {
    "startMs": 497680,
    "text": "an ordinal uh relevance tower on top of"
  },
  {
    "startMs": 500720,
    "text": "the LLM graded labels and uh this"
  },
  {
    "startMs": 505360,
    "text": "prediction sits right alongside existing"
  },
  {
    "startMs": 507599,
    "text": "engagement towers uh click add to cart"
  },
  {
    "startMs": 510240,
    "text": "and conversion and since they share the"
  },
  {
    "startMs": 512719,
    "text": "same uh bottom layers the semantic fit"
  },
  {
    "startMs": 515919,
    "text": "and relevance fit gets distilled in the"
  },
  {
    "startMs": 518159,
    "text": "same backward pass and uh our model is"
  },
  {
    "startMs": 521599,
    "text": "able to predict uh you know probability"
  },
  {
    "startMs": 523839,
    "text": "of relevance across the different levels"
  },
  {
    "startMs": 527120,
    "text": "and we are able to blend uh through a"
  },
  {
    "startMs": 529600,
    "text": "value function on on top of this that"
  },
  {
    "startMs": 532399,
    "text": "allows us to uh fine-tune between"
  },
  {
    "startMs": 534880,
    "text": "engagement and relevance for different"
  },
  {
    "startMs": 536720,
    "text": "surfaces."
  },
  {
    "startMs": 539760,
    "text": "Now these two examples are part of a"
  },
  {
    "startMs": 542080,
    "text": "larger theme of our work across similar"
  },
  {
    "startMs": 544880,
    "text": "projects uh where it seems right. We're"
  },
  {
    "startMs": 547760,
    "text": "not replacing the retrieval and ranking"
  },
  {
    "startMs": 549600,
    "text": "system with an LLM, but rather"
  },
  {
    "startMs": 551120,
    "text": "distilling the uh reasoning and"
  },
  {
    "startMs": 553360,
    "text": "understanding into some production"
  },
  {
    "startMs": 555200,
    "text": "ranking architecture. And this allows us"
  },
  {
    "startMs": 558000,
    "text": "to kind of quickly test and also"
  },
  {
    "startMs": 560320,
    "text": "scalably improve our models uh with LLMs"
  },
  {
    "startMs": 563360,
    "text": "while keeping uh you know the business"
  },
  {
    "startMs": 565360,
    "text": "and system objectives uh of these"
  },
  {
    "startMs": 568160,
    "text": "surfaces in mind."
  },
  {
    "startMs": 571279,
    "text": "So once we have supervision, the next"
  },
  {
    "startMs": 573360,
    "text": "question is how do we represent a very"
  },
  {
    "startMs": 576080,
    "text": "large and constantly changing catalog uh"
  },
  {
    "startMs": 579360,
    "text": "in a way that every model be it language"
  },
  {
    "startMs": 581680,
    "text": "models or uh you know traditional"
  },
  {
    "startMs": 584080,
    "text": "retrieval and ranking models uh that"
  },
  {
    "startMs": 586080,
    "text": "they can understand and at you know Door"
  },
  {
    "startMs": 589360,
    "text": "Dash's scale uh you can imagine we're"
  },
  {
    "startMs": 591680,
    "text": "kind of surfacing items across stores"
  },
  {
    "startMs": 593760,
    "text": "across internationally actually uh now"
  },
  {
    "startMs": 597440,
    "text": "and you know that's easily a few billion"
  },
  {
    "startMs": 599600,
    "text": "items at the store item level and for"
  },
  {
    "startMs": 602640,
    "text": "these items you obviously have unique"
  },
  {
    "startMs": 604640,
    "text": "skew ids but it says nothing"
  },
  {
    "startMs": 606800,
    "text": "semantically. So we also have a human"
  },
  {
    "startMs": 609519,
    "text": "curated taxonomy that helps us classify"
  },
  {
    "startMs": 611600,
    "text": "and categorize these items into"
  },
  {
    "startMs": 614160,
    "text": "meaningful spaces. But often this is too"
  },
  {
    "startMs": 617040,
    "text": "coarse and too rigid."
  },
  {
    "startMs": 619519,
    "text": "So for example something like hot sauces"
  },
  {
    "startMs": 622000,
    "text": "may just have like sauces hot sauces in"
  },
  {
    "startMs": 624640,
    "text": "the taxonomy"
  },
  {
    "startMs": 626240,
    "text": "uh but it doesn't tell you anything"
  },
  {
    "startMs": 627680,
    "text": "about the relationship between hoof or"
  },
  {
    "startMs": 630800,
    "text": "franks or tobasco."
  },
  {
    "startMs": 633680,
    "text": "So that's where we've also adapted uh or"
  },
  {
    "startMs": 636320,
    "text": "adopted um semantic ids. Uh it's been"
  },
  {
    "startMs": 640240,
    "text": "introduced previously already. It's a"
  },
  {
    "startMs": 642320,
    "text": "backbone for a lot of our uh work uh"
  },
  {
    "startMs": 646000,
    "text": "now. And uh what it does is it gives us"
  },
  {
    "startMs": 650160,
    "text": "a you know a short hierarchical code"
  },
  {
    "startMs": 652800,
    "text": "that's analogous to our taxonomy but uh"
  },
  {
    "startMs": 656480,
    "text": "the ability to control the fine grain"
  },
  {
    "startMs": 659040,
    "text": "nature of it. And the result of this"
  },
  {
    "startMs": 661839,
    "text": "learn taxonomy is that each item gets"
  },
  {
    "startMs": 665680,
    "text": "this hierarchical code and the prefix"
  },
  {
    "startMs": 667839,
    "text": "captures broad neighborhoods and you"
  },
  {
    "startMs": 671279,
    "text": "know the later tokens capture finer"
  },
  {
    "startMs": 673279,
    "text": "distinctions."
  },
  {
    "startMs": 675120,
    "text": "So for example in the map over on on the"
  },
  {
    "startMs": 678160,
    "text": "slide you'll see hot sauces and the"
  },
  {
    "startMs": 680640,
    "text": "structure emerging from the data with"
  },
  {
    "startMs": 683279,
    "text": "zero labels. each one share the same"
  },
  {
    "startMs": 686480,
    "text": "first and second prefix in this uh ID"
  },
  {
    "startMs": 689920,
    "text": "sequence but then split by specialtity."
  },
  {
    "startMs": 692640,
    "text": "So now you've got Mexican, Caribbean,"
  },
  {
    "startMs": 694880,
    "text": "Korean hot sauces uh being split up. And"
  },
  {
    "startMs": 699360,
    "text": "the important property is that this code"
  },
  {
    "startMs": 701279,
    "text": "that's learned is you know comparable by"
  },
  {
    "startMs": 703920,
    "text": "as I said by prefix but also usable by"
  },
  {
    "startMs": 707519,
    "text": "all of our downstream models"
  },
  {
    "startMs": 710959,
    "text": "and that unlocks a few things for us. So"
  },
  {
    "startMs": 713360,
    "text": "the first is cross-category comparisons"
  },
  {
    "startMs": 715839,
    "text": "or relationships. You know chips, salsa,"
  },
  {
    "startMs": 718560,
    "text": "guacamole may live in different taxonomy"
  },
  {
    "startMs": 721680,
    "text": "branches but a shared semantic ID"
  },
  {
    "startMs": 723600,
    "text": "neighborhood can uh indicate that they"
  },
  {
    "startMs": 725920,
    "text": "belong together and maybe a shopping"
  },
  {
    "startMs": 727680,
    "text": "mission. Second cold start problems. uh"
  },
  {
    "startMs": 731360,
    "text": "oftentimes we get new catalog items"
  },
  {
    "startMs": 733920,
    "text": "added by stores and with techniques like"
  },
  {
    "startMs": 736800,
    "text": "engram and bite parent coding of these"
  },
  {
    "startMs": 739200,
    "text": "tokens we're able to scalably add these"
  },
  {
    "startMs": 741279,
    "text": "to our models as sparse ID features"
  },
  {
    "startMs": 744880,
    "text": "third tail coverage so sparse items uh"
  },
  {
    "startMs": 749760,
    "text": "and uh inherit a lot of signal from"
  },
  {
    "startMs": 752480,
    "text": "semantically related items and uh you"
  },
  {
    "startMs": 755920,
    "text": "know we don't have to wait for the"
  },
  {
    "startMs": 757279,
    "text": "volume or exposure uh to you um for to"
  },
  {
    "startMs": 760800,
    "text": "consumers for these items. And then"
  },
  {
    "startMs": 763519,
    "text": "lastly, nice to have is the fact that we"
  },
  {
    "startMs": 766079,
    "text": "can actually also do a reverse audit. So"
  },
  {
    "startMs": 768160,
    "text": "we can audit our catalog and how well"
  },
  {
    "startMs": 770399,
    "text": "our human labels agree with the"
  },
  {
    "startMs": 772720,
    "text": "semantically learned labels."
  },
  {
    "startMs": 776160,
    "text": "So a couple of examples of where we're"
  },
  {
    "startMs": 778160,
    "text": "using it today uh just to get give some"
  },
  {
    "startMs": 780880,
    "text": "real metrics. Um it has been one of our"
  },
  {
    "startMs": 784320,
    "text": "biggest improvements to our ranker. Um"
  },
  {
    "startMs": 786959,
    "text": "in in this case it's actually improved"
  },
  {
    "startMs": 789120,
    "text": "our uh ranker by uh improving MRR"
  },
  {
    "startMs": 792480,
    "text": "between four to 5% and that's translated"
  },
  {
    "startMs": 795360,
    "text": "to pretty big conversion uh wins as"
  },
  {
    "startMs": 797519,
    "text": "well. The second one that I'm"
  },
  {
    "startMs": 800079,
    "text": "particularly"
  },
  {
    "startMs": 801760,
    "text": "fond of is query reformulation. So again"
  },
  {
    "startMs": 804800,
    "text": "going back to the hot sauce example,"
  },
  {
    "startMs": 806399,
    "text": "Sriracha can lead to garlic chili garlic"
  },
  {
    "startMs": 809200,
    "text": "sauce or sambell Ole and because these"
  },
  {
    "startMs": 813040,
    "text": "queries map onto like our catalog"
  },
  {
    "startMs": 816000,
    "text": "grounded semantic neighborhood uh that"
  },
  {
    "startMs": 820320,
    "text": "results in much more relevant queries"
  },
  {
    "startMs": 822560,
    "text": "that we're suggesting to users. Um"
  },
  {
    "startMs": 825360,
    "text": "ultimately if these are queries that we"
  },
  {
    "startMs": 827279,
    "text": "don't have inventory for on Door Dash,"
  },
  {
    "startMs": 830000,
    "text": "they're meaningless. And with the"
  },
  {
    "startMs": 832399,
    "text": "addition of semantic ids in this query"
  },
  {
    "startMs": 834160,
    "text": "graph, we were actually able to see"
  },
  {
    "startMs": 836160,
    "text": "pretty massive um MR gains as well uh"
  },
  {
    "startMs": 839360,
    "text": "for for this piece of work."
  },
  {
    "startMs": 843120,
    "text": "So for now we've talked about you know"
  },
  {
    "startMs": 845839,
    "text": "item mapping and relevance and the next"
  },
  {
    "startMs": 848160,
    "text": "question is consumer context. What does"
  },
  {
    "startMs": 850480,
    "text": "the system know about the shopper and"
  },
  {
    "startMs": 854000,
    "text": "can that knowledge be reused across"
  },
  {
    "startMs": 856240,
    "text": "models again? And this is where uh our"
  },
  {
    "startMs": 859920,
    "text": "third primitive comes in memory. So"
  },
  {
    "startMs": 862800,
    "text": "memory in the agent context is pretty"
  },
  {
    "startMs": 865199,
    "text": "well understood now. Um we apply the"
  },
  {
    "startMs": 868160,
    "text": "same concepts to uh our recommendation"
  },
  {
    "startMs": 872160,
    "text": "systems. So user embeddings are clearly"
  },
  {
    "startMs": 875040,
    "text": "very useful. uh we have many"
  },
  {
    "startMs": 876959,
    "text": "representations of our consumers but"
  },
  {
    "startMs": 879279,
    "text": "they don't necessarily get to uh why a"
  },
  {
    "startMs": 883600,
    "text": "consumer may have certain intents um and"
  },
  {
    "startMs": 886560,
    "text": "why they might have certain preferences."
  },
  {
    "startMs": 890000,
    "text": "Additionally, LLMs cannot uh readily use"
  },
  {
    "startMs": 893360,
    "text": "these uh embeddings. You could tokenize"
  },
  {
    "startMs": 895519,
    "text": "and like uh train these models to learn"
  },
  {
    "startMs": 897760,
    "text": "it. But oftentimes having some explicit"
  },
  {
    "startMs": 901680,
    "text": "LM native counterpart is very useful and"
  },
  {
    "startMs": 903839,
    "text": "that's what we found. So the idea of"
  },
  {
    "startMs": 906560,
    "text": "memory is to represent uh the consumer"
  },
  {
    "startMs": 909360,
    "text": "in multiple forms semantic in you know"
  },
  {
    "startMs": 913600,
    "text": "inspectable and reusable"
  },
  {
    "startMs": 916639,
    "text": "and uh the way we think about memory is"
  },
  {
    "startMs": 920480,
    "text": "uh in three time scales. So long-term"
  },
  {
    "startMs": 923199,
    "text": "memory captures durable preferences from"
  },
  {
    "startMs": 926480,
    "text": "orders, searches, browsing, support"
  },
  {
    "startMs": 929440,
    "text": "interactions."
  },
  {
    "startMs": 931040,
    "text": "Real-time context captures uh agentic"
  },
  {
    "startMs": 934399,
    "text": "inter interactions um oh sorry in"
  },
  {
    "startMs": 937440,
    "text": "session interactions so cart state"
  },
  {
    "startMs": 940160,
    "text": "active searches and then we have stated"
  },
  {
    "startMs": 942959,
    "text": "preferences which come from agentic"
  },
  {
    "startMs": 945600,
    "text": "interactions so something like ask door"
  },
  {
    "startMs": 947600,
    "text": "dash for example where consumers are"
  },
  {
    "startMs": 950240,
    "text": "able to explicitly state their"
  },
  {
    "startMs": 951839,
    "text": "constraints and preferences as well"
  },
  {
    "startMs": 954880,
    "text": "and the way we represent consumers in uh"
  },
  {
    "startMs": 957519,
    "text": "in uh this long-term memory is through"
  },
  {
    "startMs": 961199,
    "text": "uh memory blocks. And these are"
  },
  {
    "startMs": 963199,
    "text": "structured in a way that allows uh us to"
  },
  {
    "startMs": 966160,
    "text": "add new dimensions of the user, you"
  },
  {
    "startMs": 968480,
    "text": "know, from dietary preferences to dining"
  },
  {
    "startMs": 970800,
    "text": "preferences to uh substitute pre uh"
  },
  {
    "startMs": 973519,
    "text": "preferences um as we learn more about"
  },
  {
    "startMs": 976560,
    "text": "the consumer and that is uh decoupled"
  },
  {
    "startMs": 979600,
    "text": "from like the downstream system that u"
  },
  {
    "startMs": 982000,
    "text": "that doesn't need to reinterpret uh this"
  },
  {
    "startMs": 984959,
    "text": "for their own use cases."
  },
  {
    "startMs": 987600,
    "text": "So each consumer's memory is multi"
  },
  {
    "startMs": 989920,
    "text": "materialized in multiple forms. So first"
  },
  {
    "startMs": 992240,
    "text": "is text that's human readable. It's uh"
  },
  {
    "startMs": 995040,
    "text": "captures the consumer in a compact way."
  },
  {
    "startMs": 997839,
    "text": "Second is are latent vectors embeddings"
  },
  {
    "startMs": 1000720,
    "text": "of those memory blocks that can be fed"
  },
  {
    "startMs": 1002800,
    "text": "into retrieval and ranking systems. And"
  },
  {
    "startMs": 1005600,
    "text": "third are graph and tree or hierarchical"
  },
  {
    "startMs": 1008399,
    "text": "approaches. So uh you know type"
  },
  {
    "startMs": 1011040,
    "text": "relationships between consumers and"
  },
  {
    "startMs": 1013440,
    "text": "brands taxonomies and memory uh revealed"
  },
  {
    "startMs": 1017680,
    "text": "um preferences"
  },
  {
    "startMs": 1020079,
    "text": "and these are assembled into you know"
  },
  {
    "startMs": 1022720,
    "text": "composite memory objects that both ML"
  },
  {
    "startMs": 1025280,
    "text": "models and LLM use downstream. So an"
  },
  {
    "startMs": 1028400,
    "text": "example is you know this context graph"
  },
  {
    "startMs": 1030799,
    "text": "uh where we connect consumers with"
  },
  {
    "startMs": 1032720,
    "text": "extracted memory concepts and a graph"
  },
  {
    "startMs": 1036079,
    "text": "needs fits neatly into shopping journeys"
  },
  {
    "startMs": 1038640,
    "text": "because you know oftent times uh"
  },
  {
    "startMs": 1041760,
    "text": "consumer item interactions are sparse"
  },
  {
    "startMs": 1043919,
    "text": "and multihop. And in this case these"
  },
  {
    "startMs": 1046559,
    "text": "context graphs are able to link"
  },
  {
    "startMs": 1048160,
    "text": "consumers across memory concepts that we"
  },
  {
    "startMs": 1050960,
    "text": "previously did not have uh relationships"
  },
  {
    "startMs": 1054320,
    "text": "for. And it's particularly helpful uh at"
  },
  {
    "startMs": 1058559,
    "text": "fine grain taxonomy levels. In our case,"
  },
  {
    "startMs": 1061280,
    "text": "we're seeing uh for retrieval uh uh"
  },
  {
    "startMs": 1064640,
    "text": "using uh graph uh based embeddings"
  },
  {
    "startMs": 1067919,
    "text": "outperforming our uh existing taxonomy"
  },
  {
    "startMs": 1071120,
    "text": "based embeddings."
  },
  {
    "startMs": 1074480,
    "text": "And today this memory framework shows up"
  },
  {
    "startMs": 1076960,
    "text": "in three places. The first is"
  },
  {
    "startMs": 1079280,
    "text": "personalized collections. I'll talk"
  },
  {
    "startMs": 1081039,
    "text": "about that a bit later. Uh second is"
  },
  {
    "startMs": 1083760,
    "text": "agentic personalization. So ask door"
  },
  {
    "startMs": 1085679,
    "text": "dash for example uses this to"
  },
  {
    "startMs": 1087679,
    "text": "personalize its sessions uh for you. And"
  },
  {
    "startMs": 1090960,
    "text": "third in uh retrieval and ranking models"
  },
  {
    "startMs": 1093840,
    "text": "as I said we encode these memory blocks"
  },
  {
    "startMs": 1095760,
    "text": "and feed them as features in in"
  },
  {
    "startMs": 1098000,
    "text": "downstream models as well."
  },
  {
    "startMs": 1102000,
    "text": "So the fourth primitive is steerable"
  },
  {
    "startMs": 1104559,
    "text": "content generation. We heard quite a bit"
  },
  {
    "startMs": 1106400,
    "text": "about it in the previous talk and this"
  },
  {
    "startMs": 1110080,
    "text": "slide is my take on how we're p piecing"
  },
  {
    "startMs": 1112880,
    "text": "all these uh primitives together. So all"
  },
  {
    "startMs": 1117200,
    "text": "of these inputs, semantic ids, memory"
  },
  {
    "startMs": 1119280,
    "text": "blocks um and graded relevance or um LLM"
  },
  {
    "startMs": 1123600,
    "text": "supervision feed into multiple models"
  },
  {
    "startMs": 1126880,
    "text": "you know be it LLMs or small language"
  },
  {
    "startMs": 1128880,
    "text": "models or um uh you know traditional"
  },
  {
    "startMs": 1132240,
    "text": "models and from these inputs we can"
  },
  {
    "startMs": 1134720,
    "text": "generate different output shapes you"
  },
  {
    "startMs": 1136400,
    "text": "know ranked semantic ID lists, carousel"
  },
  {
    "startMs": 1138559,
    "text": "titles, subcopies"
  },
  {
    "startMs": 1140559,
    "text": "and these show up across different"
  },
  {
    "startMs": 1142240,
    "text": "surfaces today on Gord Dash and A direct"
  },
  {
    "startMs": 1146240,
    "text": "example of this is uh our um"
  },
  {
    "startMs": 1149360,
    "text": "personalized collections on store pages."
  },
  {
    "startMs": 1151520,
    "text": "So historically collections on store"
  },
  {
    "startMs": 1154000,
    "text": "pages uh on Door Dash have been a fixed"
  },
  {
    "startMs": 1157440,
    "text": "library or attribute-based"
  },
  {
    "startMs": 1160400,
    "text": "um and we've been able to kind of expand"
  },
  {
    "startMs": 1163760,
    "text": "that through uh consumer level"
  },
  {
    "startMs": 1166960,
    "text": "collection generation. And today this is"
  },
  {
    "startMs": 1169679,
    "text": "an offline uh LLM process where we take"
  },
  {
    "startMs": 1172640,
    "text": "in consumer memory semantic ids and syni"
  },
  {
    "startMs": 1175679,
    "text": "syn synthesize uh collections um all the"
  },
  {
    "startMs": 1178799,
    "text": "way from title subtitle to the actual"
  },
  {
    "startMs": 1180720,
    "text": "items."
  },
  {
    "startMs": 1182240,
    "text": "And the important part is that control"
  },
  {
    "startMs": 1184559,
    "text": "and steerability. We're able to react to"
  },
  {
    "startMs": 1187200,
    "text": "occasions and moments um and build those"
  },
  {
    "startMs": 1190720,
    "text": "collections as needed uh for different"
  },
  {
    "startMs": 1194400,
    "text": "consumers at serving time. uh because"
  },
  {
    "startMs": 1197600,
    "text": "this is all batch and uh generated"
  },
  {
    "startMs": 1200480,
    "text": "offline through LLMs, we are able to"
  },
  {
    "startMs": 1203200,
    "text": "still use our uh existing retrieval and"
  },
  {
    "startMs": 1205919,
    "text": "ranking stack for item hydration,"
  },
  {
    "startMs": 1207919,
    "text": "collection ranking, etc."
  },
  {
    "startMs": 1211039,
    "text": "And you know this is for example what a"
  },
  {
    "startMs": 1213039,
    "text": "shopper actually sees, right? For a"
  },
  {
    "startMs": 1214640,
    "text": "shopper with plant-based affinity, the"
  },
  {
    "startMs": 1216880,
    "text": "system can generate plant-based pantry"
  },
  {
    "startMs": 1219039,
    "text": "rows. For cat-only households, it can"
  },
  {
    "startMs": 1221360,
    "text": "generate cat dry food rows. And if a"
  },
  {
    "startMs": 1224480,
    "text": "session indicates that you know a person"
  },
  {
    "startMs": 1226400,
    "text": "is going through pantry restocking it"
  },
  {
    "startMs": 1228480,
    "text": "can bias towards pantry staples. And our"
  },
  {
    "startMs": 1231360,
    "text": "early tests show that you know consumers"
  },
  {
    "startMs": 1233760,
    "text": "are feeling the benefits of these"
  },
  {
    "startMs": 1235760,
    "text": "tailored experiences. Uh an example is"
  },
  {
    "startMs": 1238559,
    "text": "uh within our pets vertical we've been"
  },
  {
    "startMs": 1240640,
    "text": "able to drive uh close to 1% uh order"
  },
  {
    "startMs": 1244080,
    "text": "rate increases and 6% in uh active"
  },
  {
    "startMs": 1246720,
    "text": "users."
  },
  {
    "startMs": 1248880,
    "text": "So to conclude uh there are three"
  },
  {
    "startMs": 1251280,
    "text": "takeaways I would love for you to take"
  },
  {
    "startMs": 1253679,
    "text": "away from this talk and how LLM fit into"
  },
  {
    "startMs": 1256960,
    "text": "search and recommendations. First"
  },
  {
    "startMs": 1259919,
    "text": "discovery is a semantic understanding"
  },
  {
    "startMs": 1262000,
    "text": "problem. It's not only about engagement."
  },
  {
    "startMs": 1264559,
    "text": "LMS give us a way to reason about item"
  },
  {
    "startMs": 1267440,
    "text": "meaning and shopper intent. Uh in in"
  },
  {
    "startMs": 1269919,
    "text": "Door Dash's case for example. Second uh"
  },
  {
    "startMs": 1273440,
    "text": "distill LM reasoning into primitives."
  },
  {
    "startMs": 1276159,
    "text": "capture you know reasoning uh offline as"
  },
  {
    "startMs": 1279360,
    "text": "labels, semantic ids, memory and then"
  },
  {
    "startMs": 1283360,
    "text": "let smaller and faster models serve it."
  },
  {
    "startMs": 1286080,
    "text": "The online LLM call is often not the"
  },
  {
    "startMs": 1289280,
    "text": "product architecture you need. Third,"
  },
  {
    "startMs": 1292159,
    "text": "shared representations create many use"
  },
  {
    "startMs": 1294320,
    "text": "cases. Uh once you have these"
  },
  {
    "startMs": 1296480,
    "text": "primitives, they can power, you know,"
  },
  {
    "startMs": 1298960,
    "text": "retrieval, ranking, content generation,"
  },
  {
    "startMs": 1301760,
    "text": "etc."
  },
  {
    "startMs": 1303039,
    "text": "So last but not least, thank you and"
  },
  {
    "startMs": 1305919,
    "text": "thank you to all the collaborators at"
  },
  {
    "startMs": 1307760,
    "text": "Door Dash who have helped ship a lot of"
  },
  {
    "startMs": 1310480,
    "text": "these things as well."
  }
]