How Instacart transformed its search and discovery using an LLM-driven approach

Talk transcript

1. 0:00

   [on hold music] Hi.

2. 0:16

   Good afternoon, everyone. Uh, my name is Vinesh, and he's [REDACTED:username]. We are part of the search and machine learning team at Instacart. Uh, so today we'd like to talk to you about how we are using LLMs to transform our search and discovery.

3. 0:30

   Um, so yeah. So first, a little bit about ourselves. Yeah, as I mentioned, we are part of the search and discovery ML team at Instacart. And for those of you who may not be par-- who may not be familiar with Instacart, it's the leader in online grocery in North America.

4. 0:45

   Uh, and our mission is to create a world where everyone has access to the food they love and more time to enjoy it together.

5. 0:52

   So coming to what we'll actually talk about today, uh, first we'll talk about the importance of search, uh, in grocery e-commerce. Uh, then we'll look into some of the challenges facing conventional search engines, uh, and then actually get to the meat of the talk today, which is how we are using LLMs to solve some of these problems.

6. 1:10

   Uh, finally, we'll, uh, finish with some key takeaways from today's talk.

7. 1:15

   So coming to the importance of search in grocery commerce, uh, I think we've all gone grocery shopping. Customers come with long shopping lists, uh, and it's the same on the platform as well.

8. 1:25

   People are looking for tens of items, uh, and of these, the majority of them are just restocking purchases. That is, things that the customer has bought in the past.

9. 1:35

   Uh, and the remaining are items that the user is trying out for the first time. So, uh, and, and the majority of these purchases come from search. So, uh, search has a dual role.

10. 1:45

    It needs to both support, uh, quick and efficient... Uh, it needs to help the customer quickly and efficiently find the product they are looking for and also enable this new product discovery.

11. 1:56

    Uh, and new product discovery isn't just important for the customer. It's also great for our advertisers because it helps them showcase new products, uh, and it's also good for the platform because overall it increases larger basket sizes.

12. 2:10

    Uh, so, so let's see what some problems are with our existing setup that sort of, uh, makes this hard. Uh, so, so to begin with, uh, we have two classes of queries that are, are generally more challenging, especially from an e-commerce perspective.

13. 2:24

    Uh, the first are overly broad queries, uh, in this case, like on the left, the snacks query, where there are tons of products that map to that query. Uh, and now because our models are trained on engagement data, if we aren't exposing these products, uh, to the user, it's hard to actually collect engagement data to, to them

14. 2:43

    rank them up high. So the traditional cold start problem in a way. Uh, then, uh, as you can see on the query on the right, we have very specific queries like unsweetened plant-based yogurt, where the user is looking for something very specific, and these queries, uh, don't happen very frequently, which means that we just don't have enough

15. 3:01

    engagement data to train the models on. Uh, and while we have, uh, done quite a bit of work to sort of, um, improve this, uh, the challenge that we continually keep facing is that while recall improves, uh, precision is still a challenge, especially in a pre-LLM world.

16. 3:18

    Uh, the next class of problems is how do we actually support that new item discovery, as we spoke about. So when a customer walks into a, a grocery store, uh, let's say into a pasta aisle, they might see new brands of pasta that they would want to try out.

17. 3:32

    Uh, along with that, they would also see pasta sauce and every other thing that's needed to make a bowl of pasta. Uh, and customers would want a similar experience on our site.

18. 3:42

    Uh, we have heard multiple feedback, uh, multiple rounds of feedback from our customers that, "Hey, I can find the product that I'm-- that I want via search, but when I'm trying to find any other related products, it's, it's a bit of a dead end.

19. 3:55

    I would need to make multiple searches to get to where I want to." So this was a problem that we wanted to solve as well. And yeah, as I mentioned, uh, pre-LLMs, this was a, a hard problem because of the lack of engagement data, et cetera.

20. 4:09

    So let's see how we actually use the LLMs to sort of solve these problems. Uh, I'll sort of talk specifically about how we use the LLMs to up-level our query understanding module.

21. 4:20

    Now, query understanding, as I'm sure most of you know, uh, is the most upstream part of the search stack, uh, and very accurate outputs are needed to sort of enable better retrieval and recall, uh, and finally improve our ranking results.

22. 4:35

    Uh, so our query understanding module has multiple models in them, like, uh, query normalization, query tagging, query classification, uh, category classification, et cetera. So in the interest of time, uh, I'll just, uh, pick a couple of models and talk about how we, uh, sort of really improved them.

23. 4:52

    Uh, the first is our query to our category-- a product category classifier. Uh, essentially, we are taking a query and mapping it to a category in our taxonomy. Uh, so as an example, if you take a query like watermelon, that maps to categories like fruits, organic food, foods, et cetera.

24. 5:10

    Uh, and our taxonomy has about ten thousand labels, of it six thousand are more commonly used. So because a product-- a query can map to multiple labels, this is essentially a multi-label classification problem.

25. 5:23

    Um, and in the past, our traditional models, uh, which were... We actually had a, a couple of different models. One was a fast text-based, uh, neural network, which essentially modeled the semantic relationship between the query and the category.

26. 5:39

    Uh, and then as a fallback, we had an NPMI model, which was a statistical co-occurrence model between the query and the category. Now, while these, uh, techniques work great for the, for the head and torso queries, we had really low coverage for our tail queries because, again, we just didn't have enough engagement data to train the models

27. 5:57

    on. Um, and to be honest, we actually tried more sophisticated BERT-based models as well. Uh, and while we did see some improvement, the lack of engagement data, uh, meant that for the increased latency, we didn't see the wins that we actually hoped for.

28. 6:13

    So, um, so this is where we actually tried to use an LLM. Uh, first, we took all of our queries, uh, and we-- along with the taxonomy, we fed it into an LLM and asked it to predict the most relevant categories for that query.

29. 6:26

    Now, the output that came back was decent. Uh, actually, when we all looked at it, it made a lot of sense. Uh, but when we actually ran an online A/B test, the results weren't as great.

30. 6:36

    Uh, and one particular example that illustrates this point very well is a query like protein. Uh, our users that come to Instacart when they type something like protein, they're looking for maybe protein shakes, uh, protein bars, or other protein supplements.

31. 6:51

    The LLM, on the other hand, thinks that pro-- when a user types protein, they're looking for maybe chicken, tofu, or other protein foods. So this mismatch, wherein the LLM doesn't truly understand Instacart user behavior, was really the cause of the problem.

32. 7:08

    So to sort of maybe improve our results, we sort of switched the problem around. We took the most commonly converting categories or the top-K converting categories for each query and fed that as additional context to the LLM.

33. 7:22

    Uh, and then I'm sort of simplifying this a bit. There's a bunch of, uh, ranking and downstream validation that happens, but essentially that, that is what we did. We generated a bunch of candidates, uh, ranked candidates, and this greatly simplified the problem for the LLM as well.

34. 7:38

    Uh, and again, to illustrate this with an example, uh, take a query like Werner Soda. Uh, our previous model actually identified this as a, as a brand of fruit fla-- or a fruit-flavored soda, which is not incorrect, uh, but it's not very precise either.

35. 7:53

    Now, the LLM, uh, did a much better job. It identified it as a brand of ginger ale. And with this, our downstream retrieval and ranking improved greatly as well.

36. 8:02

    And as you can see from, uh, the results below, uh, especially for tail queries, we saw a big improvement. Our precision improved by over eighteen percentage points and our recall improved by over seventy percentage points, which is actually pretty significant for our tail queries.

37. 8:17

    Um, and maybe to, to very briefly look at our prompt. As you can see, it's very simple. Uh, we are essentially passing in the ca-ca-- the top converted categories as context.

38. 8:27

    Uh, there are a bunch of guidelines about what the LLM should actually outdo, and, and that's it. So this was all that was needed to sort of enable this.

39. 8:35

    Uh, again, I'm simplifying the overall flow, but, uh, the general concepts are pretty straightforward.

40. 8:43

    So coming to the-- another model. Uh, the query rewrites model is actually pretty important as well, uh, from, uh, a g-- uh, from an e-commerce perspective, especially at Instacart, because not all retailers are created equal.

41. 8:57

    Some have large catalogs, some have very small catalogs. The same query may not always return results, and that is where a rewrite is really helpful. For example, going from a query like one percent milk to just milk will at least return results that the customer can decide to buy or not.

42. 9:12

    Uh, and again, our previous approach, which was trained on, uh, engagement data, didn't do too well. It suffered... Or it did decently well on head and torso queries, but it suffered from a lack of engagement data on tail queries.

43. 9:26

    Uh, so by using an LLM similar to how we did for the product category classifier, uh, we were able to generate very precise rewrites. Uh, uh, in the example here, you can see that there's a, a substitute, a broad, and a synonymous rewrite.

44. 9:40

    So for the case of avocado oil, a substitute is olive oil, a broader rewrite is, um, healthy cooking oil, and a synonymous rewrite is just avocado extract. And again, uh, just, just looking at the results from this, and if you-- we saw a bunch of offline improvements and just moving from, uh, from thir-- using third-party LLMs here,

45. 10:02

    just going from more simpler models to better models improved, uh, the results quite a bit. Uh, this is based off of our human evaluation data. Uh, so as you can see, just improving the models itself improved the overall performance of the task.

46. 10:16

    And in terms of online improvements, we actually saw a large drop in the number of queries without any results. This is pretty significant again because, uh, we could now actually show results to users where they previously saw empty results, uh, which was great for the business.

47. 10:34

    So, uh, coming to the so-sort of the important part of this, which is how we actually scored and served the, the data.

48. 10:43

    Uh, the thing is that Instacart has a pre-- uh, has a pretty idiosyncratic, uh, query pattern. There's a very fat head and torso set of queries, and we have a sort of a long tail.

49. 10:55

    So by compu-- pre-computing the outputs for, for all of the head and torso queries offline in a batch mode, we were able to sort of, uh, cache all of this data and then at online, when a query comes in, we could just serve it off of the cache with very low impact on latency, uh, and fall back

50. 11:14

    to our existing models for the long tail of queries.

51. 11:17

    And again, this worked really well because it didn't, uh, impact our latency while it greatly improved our coverage for the long tail of queries. Now, for the, the really long tail where I said we would fall back to our existing models, we're actually trying to replace them with our distilled Llama Eight B model, uh, so that we

52. 11:38

    can actually do a much better job compared to the existing models. Um, so yeah, to sort of summarize, uh, essentially what we saw was that, uh, from a query understanding perspective, we have a bunch of models, uh, and just using our hybrid approach greatly improved their performance.

53. 11:56

    But what's actually more interesting is that today, query understanding consists of a bunch of models, and as Yuzu was talking about in the Netflix talk, managing all of these models is actually complex from a system perspective.

54. 12:08

    So Uh, consolidating all of these into an SLM or a, or maybe a large language model, uh, can make the results a lot more consistent. And I'll finish it off by giving an example here.

55. 12:20

    Uh, there's a query, hum, that we sort of saw some interesting issues with, uh, which is pr- pr- which is spelled H-U-M-M. Uh, the actual query, the-- our, our query brand Tiger identified the brand correctly as a brand of, uh, kombucha, but then our spell corrector unfortunately corrected it as hummus.

56. 12:41

    So, the results were really confusing to users, uh, and was pretty bad. But by using a more unified model, I think the results were much better. The second is by passing in...

57. 12:51

    by using an LLM for query understanding, uh, we can actually pass an extra context. Um, so instead of just generating results for that query in isolation, we can tr- really try to understand what the customer's mission is.

58. 13:05

    Um, so for example, detect if they're actually here to buy ingredients for a recipe, et cetera, and then generate the content for that. So, to talk more about that, uh, I have [REDACTED:username] here.

59. 13:17

    Thank you, Vinesh. Uh, now I'll quickly talk about how we used LLMs for showing more discovery-oriented content in search results page. Uh, just to re- uh, restate the problem, uh, our users found that while our search engine was very good at showing exactly the, the, the results that they exactly wanted to see, once they added an item

60. 13:35

    to the cart, they couldn't do anything useful with the search results page. They either had to do, like, another search or go to another page to fulfill their next intent to some sort.

61. 13:44

    Uh, tradit-- Solving this with traditional methods would require, like, a lot of feature engineering or manual work. Uh, LLMs solved this problem for us, and I will talk about how.

62. 13:53

    Uh, so this is how it looked in the end. So, for queries like swordfish, uh, let's say there are no exact results. We used LLMs to generate substitute results like other seafood alternatives, meaty fish like tilapia and whatnot.

63. 14:07

    Um, and similarly for queries like sushi, where there were a lot of e-exact results, let's say, uh, we would show at the bottom of the search results page, we would show things like Asian cook-cooking ingredients or Japanese drinks and so on, uh, in order to, like, you know, uh, get the users to engage.

64. 14:23

    Uh, I'll talk about the techniques here, but, uh, both of these, uh, both of these discovery-oriented results we saw, like, improv-- uh, led to, like, improvement in engagement as well as improvement in revenue, uh, for our-- for each search.

65. 14:37

    Uh, cool. Uh, like I said, I'll get into the techniques, but let's first talk about the requirements to generate such content. Uh, first, uh, obviously, we want to generate content that is incremented to the current solutions.

66. 14:47

    We don't want duplicates to what we were already showing. And the second requirement, and the most important one, is we wanted all of the LLM answers or, or the generation to be aligned with Instacart's domain knowledge.

67. 14:59

    What does this mean? So if a queries-- If a user searches for a query called dishes, LLM should understand that it refers to, like, cookware and not food, uh, and vice versa for a query like Thanksgiving dishes, right?

68. 15:10

    So, with these requirements in mind, we set up with-- We started with, like, a very basic generation approach. So, what did we do? We took the query, and we told the LLM, "Hey, you are an AI assistant, and your job is to generate two shopping lists.

69. 15:23

    One is a list of complementary items, and another is a list of, like, uh, substitute items for a given query," right? Um, looked good. Uh, I mean, like, so, so we, we saw the results.

70. 15:32

    They looked pretty good. Uh, our PMs vetted everything. We looked at everything. Uh, and, and like Vinesh said in, in, like, QU, we-- when we launched this to our users, uh, we saw that the results were good, but users weren't engaging it as much as we would have liked it to.

71. 15:48

    So, we went back to the drawing board, and we were like, uh, we, we tried to analyze what was going on. And what we realized quickly was, while LLM's answers, uh, were like common sense-like answers and so on and such, they weren't really what users were looking for.

72. 16:02

    Uh, taking the protein example again, like, uh, users, when they search for protein, they look for protein bars and protein shakes rather than what LLM would give as an answer, which is chicken, turkey, and tofu and whatnot, right?

73. 16:14

    So, uh, so what we did was we augmented the prompt with Instacart domain knowledge. So, f- uh, in one case, what we did was we took the query, and then we augmented it with, like, "Hey, here is the query, and here are the top converting categories, uh, for this particular query," along with any annotations from the query

74. 16:32

    understanding model like, "Hey, here is a brand present in the query. Here is, like, a, a dietary attribute present, present in the query," and so on and such. Uh, in another case, we were like, "Here is the query, and here are the subsequent queries that users did once they issued this particular query."

75. 16:47

    So, once you augmented the prompt with this additional metadata about how Instacart users behave, the, the, the results were far more better. Uh, I don't have the time to show, like, like, the before and after, but like I said, we definitely saw, like, a huge improvement in both engagement as well as revenue.

76. 17:04

    Uh, I'll quickly talk about, like, how we served, uh, all of these contents. Uh, like, very similar to QU. It's impractical to call the LLM in real time because of latency and maybe cost concerns sometimes.

77. 17:16

    So, what we did was, uh, we took all of our s- uh, historical search logs. We called LLM in, like, a batch mode and stored everything. So query, content metadata, along with even the products that could potentially show up in the carousel.

78. 17:29

    And online, it's just a very quick lookup from a feature store. Uh, and that's how we were able to, like, uh, serve all of these recommendations in, like, blazing fast time.

79. 17:39

    Uh, again, things weren't as simple as, as we making them out to be. The, the, the... Like Vinesh said, the overall concept is simple. The, the prompt itself is very simple.

80. 17:48

    But there were three key challenges that we solved along the way. Uh, one is aligning generation with business metrics like revenue. Uh, this was very important to see, like, top-line wins.

81. 17:57

    So we iterated over the prompts and the kind of metadata that we would fe- that we would feed to the LLM in order to achieve this. Second, we spent a lot of time on ranking, uh, on improving, improving the ranking of the content itself and so on and such.

82. 18:11

    So our traditional PCTR, PCVR models did not work, so we had to like employ strategies like, uh, diversity-based re-ranking and so on and so forth to get users to engage with the content.

83. 18:21

    Uh, and then the third thing is evaluating, uh, the content itself. So one is making sure that, hey, whatever LLM is giving, uh, is one, right, it's not hallucinating something.

84. 18:30

    Uh, and second, it adhered to like what Instacart or what we need as a product, right? Cool. Uh, so summarizing the, the key take-takeaways from our talk. Uh, LLMs world knowledge was super important, uh, to improve, uh, query understanding predictions for, especially for the tail queries.

85. 18:47

    Uh, while LLMs were super helpful, we really found success by combining the domain knowledge of Instacart with LLMs, uh, in order to see the top-line wins that we saw.

86. 18:57

    Uh, and the third and the last one is evaluating the content a-as well as the Q predictions and so on and such was far more important and far more difficult, uh, than we anticipated.

87. 19:07

    We, we used LLMs as a judge in order to make this happen, but very, very important step we-- and we realized that kind of late. So yeah, that's all from us.

88. 19:15

    We'll take questions now. [audience applauding] Thank you, [REDACTED:username]-[REDACTED:username], Vinesh. Um, yeah, we'll take questions at the mic while the next speaker gets set up.

89. 19:24

    Hi. Uh, thanks for the talk. Um, have you also been trying around queries which are very long in natural language? Like, uh, I want these three items and these five items, like what we would do it on Chat-ChatGPT.

90. 19:38

    Or it's still like single item that's the focus?

91. 19:43

    Uh, yeah. I, I think we have... We actually launched something in the past, uh, like Ask Instacart, if you've heard, heard of it, which essentially takes natural language queries and tries to map that to search intent.

92. 19:55

    So for example, you might ask-- you might say, "Healthy foods for a [REDACTED:age] baby," or something like that. And so that would map to things like fruit slices. Uh, I don't know if [REDACTED:age] toddlers can eat popcorn, but something along those lines.

93. 20:09

    And, and then we had our usual ranking, uh, recall and ranking stack sort of retrieve those results.

94. 20:16

    So, uh, any learnings from that experiment for you?

95. 20:19

    Yeah. So, so I think we actually have a lot of learnings from that. Essentially, uh, as [REDACTED:username] already mentioned, uh, we need to inject a lot of Instacart context into the model to be able to get re-decent results.

96. 20:31

    The evaluation part is really key, uh, so having a robust automated evaluation pipeline was important. And lastly, passing context. That is, for example, if it's a...

97. 20:42

    Let's say it's a Mother's Day query, and let's say we come up with the individual search intent as perfumes. You really want women's perfumes to be in there. Whereas when we just had perfumes, we could see all kinds of items.

98. 20:56

    So passing that context from the LLM to the downstream systems is really important.

99. 21:00

    Cool. Thanks.

100. 21:01

     Yeah, we have a lot of examples where we failed we can talk about. [upbeat music]