# How Instacart transformed its search and discovery using an LLM-driven approach

Tejaswi Tenneti · Vinesh Gudla

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/PjaVHm_3Ljg-instacart-transformed-its-search-discovery-using) · [Video](https://www.youtube.com/watch?v=PjaVHm_3Ljg)

## How Instacart transformed its search and discovery using an LLM-driven approach

Vinesh Gudla and Tejaswi Tenneti explain how conversion data, batch generation and careful evaluation made LLMs useful for grocery search and product discovery.

### Search must support both restocking and discovery

[1:15](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=75s)

A grocery shopper arrives with a list of tens of items. Most replenish products they have bought before; a few are things they want to try for the first time. Search has to support both activities without making an already long shopping trip harder. In Instacart's account, most purchases come through search, so quickly finding a familiar product and discovering something new are responsibilities of the same interface.

Discovery also matters beyond the individual purchase. It gives advertisers a way to introduce products and gives the platform an opportunity to increase basket sizes. A search engine that efficiently retrieves an item but offers no useful next action satisfies only part of the shopping journey.

### Broad queries, rare queries and discovery dead ends

[2:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=130s)

Two kinds of query expose weaknesses in models trained on engagement data:

- **Broad queries**, such as `snacks`, match many products. Products that are not shown cannot collect the engagement signals that would help them rank higher. This creates a cold-start problem within an apparently well-covered query.
- **Highly specific queries**, such as `unsweetened plant-based yogurt`, occur too infrequently to provide much training evidence. The intent is narrow, but the model has few examples from which to learn it.

Instacart had made progress retrieving more candidates, improving recall. Precision—whether those candidates actually fit the request—remained difficult.

A physical pasta aisle illustrates a different gap. A shopper encounters unfamiliar pasta brands alongside sauce and other ingredients needed for a meal. Online, customers could find the product they requested, yet related products remained difficult to discover. They had to issue another search, then another, to assemble what the store aisle naturally brought together. Sparse engagement data made that related-product experience difficult to build with the existing methods.

### Query understanding inherits the long-tail data problem

[4:09](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=249s)

Gudla begins with query understanding, the upstream layer whose interpretations guide retrieval and ranking. It contains several models, including normalization, tagging and category classification. The product-category classifier maps a query into the catalog taxonomy: `watermelon`, for example, can map to fruits, organic food and broader food categories. Instacart's taxonomy contained about 10,000 labels, roughly 6,000 of them commonly used. Because one query can receive several labels, this is a multi-label classification task.

The previous system combined a **[fastText](https://fasttext.cc/docs/en/supervised-tutorial.html)**-based neural model for semantic relationships with an **NPMI** statistical co-occurrence fallback. These approaches worked well for head and torso queries—the frequent and moderately frequent searches—but covered the tail poorly. The team also tried **[BERT](https://arxiv.org/abs/1810.04805)**-based models. Those produced some improvement, but the shortage of engagement evidence still constrained learning, and the gains did not meet the team's expectations for the added latency. Increasing model sophistication had not removed the underlying data problem.

### Ground category predictions in shopping behavior

[6:13](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=373s)

The first LLM experiment supplied queries and the taxonomy, then asked for the most relevant categories. The outputs looked sensible to the team. An online A/B test was less encouraging: plausible category assignments were not necessarily the assignments Instacart shoppers needed.

The query `protein` makes the mismatch concrete. A general model interpreted it through foods such as chicken and tofu. Instacart shoppers often meant protein shakes, bars or supplements. The missing input was shopping behavior. The team therefore supplied each query's top-K converting categories as additional context, giving the LLM a commercially grounded set of candidates to work with. Candidate ranking and downstream validation remained part of the process.

Protein interpreted through two contexts
Top-converting categories supply commercial context; ranking and validation still follow.
- protein — The query is unchanged across both interpretations.
- Query and taxonomy — Without shopping context, the model applies general food knowledge.
- Top-converting categories — Commercial behavior supplies additional context for the same query.
- Chicken and tofu — The speaker's examples of the general-model interpretation.
- Protein bars and shakes — The speaker's examples of what Instacart shoppers are seeking.
- protein → Query and taxonomy (Same query)
- protein → Top-converting categories (Added shopping context)
- Query and taxonomy → Chicken and tofu (General-model interpretation)
- Top-converting categories → Protein bars and shakes (Align with shoppers)

The gain was also visible in specificity. For `Vernors soda`, the previous model returned a fruit-flavored-soda category. The LLM identified ginger ale, a more precise category that helped downstream retrieval and ranking. For tail-query product-category classification, Gudla reports gains of more than 18 percentage points in precision and more than 70 percentage points in recall.

The prompt itself was compact: top-converting categories supplied context, and guidelines specified the expected output. Its simplicity did not eliminate the surrounding processing. It made the model's part of the task easier by supplying useful candidates and a clearer output contract.

### Rewrite queries when a retailer cannot satisfy the original

[8:43](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=523s)

Understanding a request does not guarantee that a particular retailer can fulfill it. Instacart serves retailers with very different catalog sizes, so the same query can succeed at one store and return nothing at another. Broadening `one percent milk` to `milk` can expose available choices while leaving the purchase decision to the shopper. The previous engagement-trained rewrite model again worked better on head and torso queries than on the data-poor tail.

The LLM approach generated distinct kinds of rewrite rather than treating every alternative as equivalent. For `avocado oil`, the talk illustrates three relationships:

| Rewrite relationship | Generated example | Purpose |
| --- | --- | --- |
| Substitute | `olive oil` | Offer a different item that might serve the shopper's purpose. |
| Broader query | `healthy cooking oil` | Expand the request to a larger category. |
| Synonymous rewrite | `avocado extract` | Propose another expression for the same intended item. |

The last entry is a generated synonym candidate to validate, not a basis for treating avocado extract and avocado oil as interchangeable products. Keeping these relationships distinct matters because broadening a request and preserving its exact meaning are different operations.

On the team's human evaluation data, more capable third-party LLMs improved query-rewrite quality. In online testing, the team reported a large reduction in queries returning no results. That changed the user experience directly: searches that previously ended at an empty page could now offer products to consider.

### Precompute outputs and keep live search inexpensive

[10:34](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=634s)

Instacart's query distribution made batch processing useful: a large head and torso of recurring searches sits alongside a long tail. The serving design separated expensive generation from the live request:

1. **Offline:** precompute LLM outputs for head and torso queries and cache the results.
2. **Online, on a cache hit:** retrieve the stored output instead of generating it again.
3. **Online, on a cache miss:** fall back to the existing models for queries outside the cached set.

The work performed during a live search was therefore usually a lookup for the precomputed queries, while the existing system handled uncovered requests.

Gudla reports improved coverage with little impact on search latency. For the extreme tail, the team was trying a distilled **Llama 8B** model as a replacement for the existing-model fallback. That was a proposed improvement to the uncovered-query path; the described serving system still used the existing models there.

### Separate models can disagree about the same query

[11:56](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=716s)

Beyond individual prediction quality, a collection of query-understanding models creates coordination problems. For `Humm`, Instacart's brand tagger correctly recognized a kombucha brand, while its spell corrector changed the query to `hummus`. The same request thus acquired incompatible interpretations before reaching the rest of the search stack. Gudla presents consolidation into a small or large language model as a way to make these outputs more consistent, as well as reduce the complexity of managing many models.

A shared model also creates an opportunity to interpret more than an isolated query. Additional context might reveal that the customer is gathering ingredients for a recipe, allowing the system to generate content for that broader shopping mission. This is the transition from Gudla's query-understanding work to Tenneti's discussion of discovery on the results page.

### Give shoppers a useful next action

[13:17](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=797s)

Tenneti starts where the exact-match experience ends. Once customers added an item to their cart, they often had to search again or leave the page to continue shopping. Discovery content offered two ways forward. If `swordfish` returned no exact results, the page could suggest seafood alternatives; tilapia was one example shown. If `sushi` already had plenty of results, complementary groups such as Asian cooking ingredients or Japanese drinks could appear near the bottom of the page. Tenneti reports improved engagement and revenue per search for these discovery experiences.

The content had to add something beyond the existing results, rather than repeat products already shown. It also had to reflect Instacart's domain. In the stated shopping context, `dishes` refers to cookware, while `Thanksgiving dishes` refers to food. A useful generator must preserve that difference before it can suggest the next thing to buy.

### Plausible shopping lists still need behavioral context

[15:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=910s)

The initial generation prompt asked for two shopping lists: complementary items and substitutes for the query. Product managers and other internal reviewers liked the results. After launch, however, users engaged less than the team expected. The answers were plausible in everyday terms but did not consistently match what customers were trying to do. Tenneti returns to the earlier `protein` example: bars and shakes were the shopping intent, while the model suggested chicken, turkey and tofu.

The team augmented the prompt with behavioral context in two forms:

- Top-converting categories, together with query-understanding annotations such as a detected brand or dietary attribute.
- Subsequent queries that users issued after the current query.

The first supplied evidence about how to interpret the request. The second supplied evidence about where shoppers went next, which was directly relevant to generating a useful continuation.

Tenneti reports that adding this behavioral metadata improved both engagement and revenue. The prompt still asked the model to generate related shopping content, but it now included evidence of how Instacart customers actually shopped.

### Batch discovery generation, then rank the stored content

[17:04](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1024s)

Discovery used a similar separation between generation and serving. Calling an LLM during every search was considered impractical because of latency and, sometimes, cost. Instead, the team processed historical search logs in batches and stored the query, generated content metadata and even products that could appear in a carousel. At request time, recommendations came from a quick feature-store lookup. The stored result included material needed to present the discovery experience, not just an abstract related-query suggestion.

The simple batch-and-lookup outline concealed substantial iteration. One challenge was aligning generation with business outcomes such as revenue. The team changed both prompts and the metadata supplied to the LLM to improve that alignment. Another was deciding which generated content to show and in what order. Generating a relevant set of possibilities did not settle the ranking problem.

Traditional predicted click-through-rate and conversion-rate models—`pCTR` and `pCVR`—did not work for this discovery content. The team used strategies including diversity-based reranking to encourage engagement. The new surface therefore needed work on its selection and ordering logic as well as its generation prompt.

### Evaluate correctness and product fit

[18:21](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1101s)

The third implementation challenge was evaluation. It required two separate judgments: whether the generated content was correct rather than hallucinated, and whether it met Instacart's product requirements. World knowledge helped with predictions, especially for tail queries, but combining it with domain knowledge was what produced the business gains the speakers described. Semantic plausibility alone was an incomplete test of usefulness.

Evaluation covered both discovery content and query-understanding predictions, and it proved more difficult and more important than the team had anticipated. They used LLM-as-judge evaluation to support that work. Tenneti's closing admission is that they recognized the importance of this step late: evaluation needed to be a central part of developing the system.

### Carry natural-language intent into retrieval and ranking

[19:24](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1164s)

An audience member asks whether the approach extends to long conversational requests containing multiple items, rather than searches focused on one item. Gudla points to [Ask Instacart](https://company.instacart.com/updates/bringing-inspirational-ai-powered-search-to-the-instacart-app-with-ask-instacart), which mapped natural-language requests into search intent. The existing recall and ranking stack then retrieved the products. The answer describes an intent-translation layer feeding conventional search.

Asked what the team learned from that experience, Gudla again emphasizes injecting Instacart context and building a robust automated evaluation pipeline. He then identifies a further requirement: context must survive the handoff from the language model to downstream systems.

In his Mother's Day example, extracting only `perfumes` loses the intended restriction to women's perfumes. The retrieval system can then return perfumes of all kinds, even though the original request contained a more specific shopping context. Preserving that context through retrieval and ranking is part of fulfilling the request; passing along the generic item term alone is not enough.

## Resources

- [Building the Intent Engine](https://company.instacart.com/tech-innovation/building-the-intent-engine-how-instacart-is-revamping-query-understanding-with-llms) — A November 2025 follow-up from the team, with later details on validation and the deployed long-tail model.
- [Search infrastructure on Postgres](https://company.instacart.com/tech-innovation/how-instacart-built-a-modern-search-infrastructure-on-postgres) — Both speakers' related account of the retrieval infrastructure underneath Instacart search.
- [Ask Instacart](https://company.instacart.com/updates/bringing-inspirational-ai-powered-search-to-the-instacart-app-with-ask-instacart) — The natural-language grocery search product discussed in the closing Q&A; the linked account was last updated in October 2023.
- [Related intent generation for product discovery](https://arxiv.org/abs/2607.27172) — A July 2026 paper coauthored by both speakers extends this line of work; its experiments postdate the talk.

## Slides

### Instacart’s LLM-driven approach to Search and Discovery

Vinesh Gudla and Tejaswi Tenneti introduce Instacart’s use of LLMs for search and discovery.

[0:23](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=23s) [Title slide introducing Instacart’s search and discovery talk.](https://assets.ai.engineer/sites/aiecode2025/sha256/88c2175fea49d0b04ba5343ae7eac99db72c57fdc6deaf7b0840afa9f6c44b22.webp)

Instacart’s LLM-driven approach to Search and Discovery

Speaker Information:
Vinesh Gudla, Staff MLE @Instacart
Tejaswi Tenneti, Director of ML @ Instacart

[illegible]acart
AI.Engineer Jun 3, 2025

### Agenda

The talk moves from the importance of search to challenges in query interpretation and discovery content, then explores LLM-based approaches.

[0:52](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=52s) [Agenda covering search, conventional search limitations, LLM solutions and summary, beside a cooking photograph.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc6e5fce61d5bfac09b6483bb506075a7b3fc5d6381c4fa2eee58ceeda6bb3fe.webp)

Agenda

• Importance of Search
• Key limitations of conventional search engines
  • Challenges in query interpretation
  • Lack of Discovery-driven content
• Solutions -
  • LLM as QU
  • Better discovery-oriented content using LLMs
• Summary

### Limitations of conventional search

The talk turns to problems in the existing search setup that make product discovery harder.

[2:11](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=131s) [Section title: Limitations of conventional search.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7b199e687f664255996d19e2e90fa1b6a0211d7ad738bff7aff2a49789a19a0.webp)

Limitations of conventional search

6

### Lack of Discovery-driven Content

Search returned unrelated alternatives for “croissant” and unhelpful related items for “vegan burger.” Customers said finding related products could require multiple searches.

[3:29](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=209s) [Grocery search examples show unrelated alternatives for croissant and limited related items for vegan burger.](https://assets.ai.engineer/sites/aiecode2025/sha256/ffe3728201cb219d36ad4076fad379dea8e63d798b278d8b0ab3f0861e87a82c.webp)

Lack of Discovery-driven content

We returned irrelevant alternative results for queries where no exact matches were found

Even if users could find what they were looking for at the top, scrolling down the page would not be very helpful for satisfying follow up intents

Interface text:
croissant
$11.81
Thomas’ Original English Muffins, 24-count
Many in stock
$6.21
Kirkland Signature Eggs, Large, 24-count
Cage-Free
Many in stock
$10.93
El Camino Real Bakery Organic Cinnamon Raisin Bread, 28 oz,...
2 x 28 oz
Organic
Many in stock
$9.32
Kirkland Signature Fresh Mozzarella, 18 oz, 2-count
2 ct
Many in stock

vegan burger
Results for "vegan burger"
Best seller
$18.67
Buy 1, get $5 off
Impossible Plant-Based Burger Patties, 4 oz, 10-count
Related items
Buy it again
$8.70
House Foods Organic Tofu, Firm, 16 oz, 4-count
4 ct
Many in stock
$22.38
Trident Seafoods Alaskan Salmon Burgers, 48 oz
12 ct
Many in stock

### Query Understanding Overview

A raw query passes through spell correction and normalization, then branches into models for classification, rewrites, brand recognition, query tagging, and more.

[4:19](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=259s) [Flowchart from a raw query through spelling correction and normalization to classification, rewrites, recognition and tagging.](https://assets.ai.engineer/sites/aiecode2025/sha256/e60bbca2a9f9cb551b67bf1e1905cfe729ac4cf61dd224bf53e994bff7aec7df.webp)

Query Understanding Overview

raw query

Preprocessing:
spell correction
query normalization

category classification
query rewrites
brand recognition
query tagging
aisle classification
and more

### Query to Product Category Classification: Previous Model

The previous classifier mapped queries to multiple taxonomy categories using fastText and an nPMI fallback. Sparse engagement data left it with low coverage for tail queries.

[4:51](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=291s) [Previous multilabel model details alongside product and category examples for a pepper and cut watermelon.](https://assets.ai.engineer/sites/aiecode2025/sha256/87cd9ff308c6aed0968b60fce8f8cdc0aa229597eec0636682f2697bd048e260.webp)

Query to Product Category CLS

Previous model
Multilabel classification
Labels: ~6000
Models: Fastext (shallow NN), nPMI
Training data: Search logs

product | category

Orange Bell Pepper
$2.99
1 each
Many in stock
Best seller
+ Add
Food>Fresh Produce>Fresh Vegetables>Bell Peppers

Fresh Cut Watermelon Cup
$4.69
12 oz container
Best seller
+ Add
Food>Fresh Produce>Fresh Fruit>Pre-Cut Fruits,
Food>Fresh Produce>Fresh Fruit>Melons>Watermelon

### Query to Product Category Classification: LLM Model

Instacart feeds top-converting categories to an LLM for reranking, then applies relevance guardrails. For tail queries, the slide reports precision up 18 pp and recall up 70 pp.

[6:12](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=372s) [Diagram of category candidate generation, LLM ranking and guardrails, with example outputs and reported tail-query precision and recall gains.](https://assets.ai.engineer/sites/aiecode2025/sha256/c34c9e1ab274b66e6283c3419444aca9c724a348fc80b0a32c965bcca59563d6.webp)

Query to Product Category CLS

New LLM-based model

Context-aware candidate generation:
raw query → User engagement data → Top 5 converted categories

Rank by LLM:
prompt engineering:
1. Human in the loop
2. Chain of thoughts
3. Few shot learning
LLM → Top 5 reranked by LLM

Post-processing:
Relevance guardrails:
1. OpenAI Text Embedding
2. Search Embedding
query categories

Output
query | Fasttext output | LLM output
beef short ribs | Meat, Beef | Meat>Beef>Beef Ribs>Short Ribs
butter milk | Dairy, Milk | Dairy>Milk>Buttermilk
vernors soda | Soda, Beverages, Fruit Flavored Soda | Beverages>Soda>Ginger Ale
vegan roast | None | Meat > Meat Alternatives

Metrics improvement for tail queries
Precision | +18 pp
Recall | +70 pp

### Query to Product Category Classification: Prompt

The prompt uses top-converted categories as context, then asks the model to rank specific categories by search intent, account for brands, and question conversions when the percentage is low.

[8:18](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=498s) [Prompt asks a shopping assistant to rank categories by search intent, specificity and brand context.](https://assets.ai.engineer/sites/aiecode2025/sha256/b62e9cec0d5faf994ea27942dd8adcfbdff37075652ec6901953a8d605b34b2b.webp)

Query to Product Category CLS

You are an AI shopping assistant for the online grocery store Instacart.

I will provide a list of top converted categories for this search term at Instacart. Your job is to rank product categories based on if they exactly match with the intent of the given search term.

Here are the detailed guidelines:

1. Ensure the category is as specific or granular as possible.
2. If the search term contains a brand intent, you should take brand information into account when finding relevant categories.
3. Don't directly return top converted categories since they could be irrelevant, especially when conversion pct is low.

### Query Rewrites

Three agents generate substitute, broader and synonym rewrites, then deduplicate and sort them. Broadening “1% milk” to “milk” can return results when a retailer has no matches for the original query.

[8:42](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=522s) [Multi-agent rewrite framework generates substitute, broader and synonym rewrites, with examples in a table.](https://assets.ai.engineer/sites/aiecode2025/sha256/eee265a9ad6863e71279f301a5a076313594a748e80da02eaf2561955e4e9439.webp)

Query Rewrites

Multi-agent framework
raw query
Substitute Agent
LLM
substitute rewrites
Broader Agent
LLM
broader rewrites
Synonym Agent
LLM
synonym rewrites
dedeup and sorting
rewrites

query | strong substitute rewrite | broader rewrite | synonym rewrite
1% milk | 2% milk | milk | low-fat milk
avocado oil | olive oil | healthy cooking oil | avocado extract
block cheese | shredded cheese | cheese | solid cheese

### Query Rewrites

In human evaluation, substitute rewrites rose from 85% with GPT-3.5 to 98% with GPT-4 few-shot learning. Online, average recall size rose 0.5%, while the percentage of queries with no results fell 4.4%.

[9:50](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=590s) [Offline rewrite quality by model and online changes in recall size and queries with no results.](https://assets.ai.engineer/sites/aiecode2025/sha256/1310373d17651fd3f46b1d96c2c3949a589d57c5e35df23e49ac12ee46028d56.webp)

Query Rewrites

Offline improvements

gpt 3.5 | gpt 4 | gpt 4, few shot learning
Substitute | 85% | 93% | 98%
Exact | 10% | 5% | 2%
Irrelevant | 5% | 2% | 0%

% of rewrites that are Substitute vs Exact vs Irrelevant to the original query

Online improvements

Avg Recall size | +0.5%
% Queries with No Results | -4.4%

### Scoring and Serving

Instacart precomputes and caches labels for head and torso queries. The serving diagram routes cache misses to a real-time LLM call; a small fine-tuned model is under test for the long tail.

[10:35](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=635s) [Flowcharts show offline generation and caching of query understanding labels and online serving with a real-time LLM fallback.](https://assets.ai.engineer/sites/aiecode2025/sha256/89ea94d8ee4882ec15597c9fa77236cf920a6d4c45827360b4337b4d594d2c4b.webp)

Scoring and Serving

Offline Scoring, Daily incremental
Raw queries
Offline QU labels
cache hit?
Yes → TTL expired?
TTL expired? Yes → Generate QU labels
cache hit? No → Bot query?
Bot query? No → Generate QU labels
Augment with engagement data
Prompt
LLM
Update

Online Serving
Query → Search Orchestrator → QU labels → cache hit
cache hit: Yes → QU predictions
cache hit: No → Real-time LLM call
Ongoing test with a small fine-tuned model

### What's next

Instacart is exploring a unified model for consistent query understanding and session context to identify a shopper’s mission. Separate models, for example, tagged humm as a brand but corrected it to hummus.

[11:41](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=701s) [Future work includes unified query understanding and session-level understanding of reformulations, complementary intents, and recipes or meals.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cd16cef517df0b0f8e9c0d28bf6f4a6c1f54de473fe1cdb8034c8eaf72282cd.webp)

What’s next

Exploring how LLMs can help with

Consistent reasoning across all QU tasks
a. A unified model that performs all tasks in a single shot

Session-level Task understanding
a. Query reformulations, Complementary intents, Recipe or meal detection

### LLMs for product discovery

Instacart used LLMs to add discovery-oriented content to search results, helping users continue shopping after adding an item to their cart.

[13:18](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=798s) [Section title: LLMs for product discovery.](https://assets.ai.engineer/sites/aiecode2025/sha256/61e9f212aa1e512736c66675aad0d81b1b24e356eeb0f996aba143fc9581241b.webp)

LLMs for product discovery

19

### Generating Discovery-oriented content

Instacart used LLMs to suggest ingredients and drinks for sushi and seafood substitutes for swordfish. The slide reports more products added per search for complements and higher conversion for substitutes.

[13:51](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=831s) [Grocery search examples show complementary items for sushi and substitute seafood results; the slide reports improved product additions and conversion.](https://assets.ai.engineer/sites/aiecode2025/sha256/f1132674c92e3d4da01e5f16ee659188b3506be73c3e389738991274a431625a.webp)

Generating Discovery-oriented content

LLMs helped create inspirational and discovery-driven content to enrich our search results page.

High quality Complementary results improved number of products added per search

Showing highly relevant substitute results led to higher conversion rates

Interface text:
sushi
Goes well with
Asian cooking ingredients
$4.69 Kikkoman Marinade & Sauc... 10 fl oz
$5.99 Kikkoman Less Sodium Soy Sauce 10 fl oz
$4.69 Organic Nakano Sea... Rice Vinega... 12 fl oz
Many in stock
Sponsored
Japanese drinks
$49.44 Basil Hayden Kentucky Straigh... 750 ml
$28.74 Maker's Mark Kentucky Straigh... 750 ml
$10.34 Gekkeikan S... 50.7 fl oz
Many in stock
Sponsored
View cart
Shop · Buy It Again · Account

2 results for “swordfish”
[illegible product details]

No results for “swordfish”
Browse related items or try another search.
Try these instead
Seafood alternatives
[illegible product details]
Other meaty fish
[illegible product details]

### Requirements

Generated content should add to existing search results, reflect Instacart’s domain knowledge, and appear when and where customers need it. For example, “dishes” and “thanksgiving dishes” imply different meanings.

[14:36](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=876s) [Requirements call for incremental content, alignment with domain knowledge, and timely presentation.](https://assets.ai.engineer/sites/aiecode2025/sha256/4155c8a25f396f7f40afe3effcd8dc27e9bfefaf55a6da43558950b38e950a77.webp)

Requirements

1. Generate content that is incremental to current solutions
   a. Avoid duplication with products that are already shown in search results

2. Alignment with Instacart domain knowledge
   a. LLM should understand that a user query for "dishes" refers to "cookware" not "food", and vice versa for “thanksgiving dishes”

3. Be well timed
   a. The experience should appear when and where a customer needs it

### Approach: Basic Generation

The query goes to an LLM prompted to generate two five-item shopping lists: substitutes and complementary products.

[15:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=910s) [A query goes to an LLM, which generates substitute and complementary shopping lists.](https://assets.ai.engineer/sites/aiecode2025/sha256/a25d2222c5dea0eebb60f8cc5b0e2e4c2a1077dad2d7a8166a3e465403fc713e.webp)

Approach: Basic Generation

Query → LLM → Substitutes
                 → Complements

As an AI assistant, you aid with online grocery shopping. You will first generate 2 shopping lists, each with 5 items. The 2 lists are:
1. substitute items;
2. complementary/bought-together product group that goes well with the search term;

### Basic Generation: Crab Example

For “crab,” the LLM suggests other seafood as substitutes and seasonings and sides as complements. After launch, users engaged less than the team hoped; the speaker says the answers missed their intent.

[15:29](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=929s) [Table of seafood substitutes, seasonings, and side dishes generated for the query crab.](https://assets.ai.engineer/sites/aiecode2025/sha256/fbbeb465ef3b24a91d587da9b53e1bc6e36e5fd8100a62303346a7cc2fb72195.webp)

Approach: Basic Generation

Content Type | Sample output for query ‘crab’

Substitute
Reason: Other seafood options
['Lobster', 'Shrimp', 'Scallops', 'Oysters', 'Mussels']

Complementary
Reason: Seafood seasonings:
['Old Bay Seasoning', 'Lemon Pepper Seasoning', 'Cajun Seasoning', 'Garlic Butter Seasoning', 'Seafood Boil Seasoning']

Complementary
Reason: Side dishes
['Corn on the Cob', 'Potato Salad', 'Cole Slaw', 'Garlic Bread', 'Steamed Vegetables']

### Query Augmented Generation

Instacart augments a query with purchase categories and query annotations before an LLM generates substitutes and complements, grounding suggestions in shopping behavior.

[16:15](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=975s) [A query is augmented with user engagement data before an LLM generates substitutes and complements.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc7e1aae1ccf5f3275bea73f59a33d26a65c43228284d11005fbbf658001c5d5.webp)

Query Augmented Generation (QUAG)

Query → Augment with user engagement data → LLM → Substitutes / Complements

As an AI assistant, you help with online grocery shopping. You generate search queries for ecommerce platforms like Instacart, utilizing the following info.
1. user query. 2. previous purchased categories. 3. query annotations.
Annotations help you accurately understand query intent, where <P> means product concept, <B> means brand name, <A> means attributes.

### Utilizing Next Converted Search Terms

The prompt uses items frequently bought after sour cream to ask for merchandising categories. The diagram shows user engagement data guiding an LLM toward substitutes and complements.

[16:38](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=998s) [An augmented generation diagram and a prompt asking how to categorize items bought after sour cream.](https://assets.ai.engineer/sites/aiecode2025/sha256/61bdb9c8c4feafb4525d195edc2b49973752f08b9b06a402c9fe66d0bc65be61.webp)

Utilizing next converted search terms

Query → Augment with user engagement data → LLM → Substitutes / Complements

You are an AI shopping assistant for a grocery search engine like Instacart. Here are the items and their respective frequencies that customers often bought after purchasing "sour cream", Can you provide suggestions on how we can improve the merchandising of these items by categorizing them more effectively?

### Content Generation Pipeline

Instacart processes historical search logs with an LLM in batches and stores the results. Online, a search orchestrator retrieves and ranks that content for the shopper’s query.

[17:03](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1023s) [Offline search logs and an LLM service populate a content store, which an online search orchestrator uses to serve recommendations.](https://assets.ai.engineer/sites/aiecode2025/sha256/b04c0552c7723ed16f581db3f2becdeb763590b94cfae00ab74093b3a0ddfaac.webp)

Content generation pipeline

Offline
Search logs
LLM service
Search API

Content store
<query,
content id,
content metada,
[Products]>

Online
Query: “blueberries”
Search Orchestrator
Handles retrieval and ranking

Interface text:
blueberries
Fruit Fiesta
Antioxidant powerhouses
Smoothie boosters
[illegible]

### Challenges That We Solved

Instacart tuned prompts and metadata for revenue, used diversity-based re-ranking after traditional models failed, and checked generated content for accuracy and product fit.

[17:33](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1053s) [Three challenges: aligning generation with business metrics, ranking for engagement, and evaluating content, beside a grocery search example.](https://assets.ai.engineer/sites/aiecode2025/sha256/5f3189bf6b8b2f62370eefae1d63d9d55d7bed06fb5658d0e30aace798892526.webp)

Challenges that we solved

1. Aligning generation with business metrics like revenue
2. Ranking the content to improve engagement
3. Content evaluation

Interface text:
blueberries

Fruit Fiesta
$1.84 each (est.); $2.05
Jumbo Navel Orange
$1.79 / lb
About 1.03 lb each
Many in stock
$0.38 each (est.)
Banana
$0.99 / lb
About 0.38 lb each
Many in stock
$10.60 /pkg (est.)
Green Seedless Grapes
$4.69 / lb
About 2.26 lb / [illegible]
Many in stock

Antioxidant powerhouses
$5.99; $7.99
Guittard Chocolate Semisweet Chocol...
12 oz
Many in stock
Sponsored
$12.99
Dove PROMISES Dark Chocolate Al...
12.67 oz
Sponsored
$4.99; $5.99
Signature SELECT Pomegranate, Arils
8 oz

Smoothie boosters
$2.39; $3.59
Zola Coconut Water, with Pulp
Vita Coco
[illegible] Water, Pineapple
$2.99
Signature Farms Spinach
6 View cart

### Summary

LLM world knowledge helped Instacart understand tail queries, while domain knowledge contributed to its success. Evaluating content and query predictions was harder than expected, so the team used LLMs as judges.

[18:34](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1114s) [Summary of takeaways: LLM world knowledge improves query understanding for tail queries; combining it with domain knowledge is crucial; and content evaluation was harder and more important than anticipated.](https://assets.ai.engineer/sites/aiecode2025/sha256/a06d5555479276899a451de53a085dfc7a33d97e39e0c78d5d4ef4f37970aabb.webp)

Summary

- LLMs world knowledge is very helpful to improve QU for tail queries
- Combining world knowledge of LLMs with domain knowledge is crucial for success
- Evaluating content was more important and harder than we anticipated

## Full transcript

[0:00](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=0s) [on hold music] Hi.

[0:16](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=16s) Good afternoon, everyone. Uh, my name is Vinesh, and he's [REDACTED:username]. We are part of the search and machine learning team at Instacart. Uh, so today we'd like to talk to you about how we are using LLMs to transform our search and discovery.

[0:30](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=30s) Um, so yeah. So first, a little bit about ourselves. Yeah, as I mentioned, we are part of the search and discovery ML team at Instacart. And for those of you who may not be par-- who may not be familiar with Instacart, it's the leader in online grocery in North America.

[0:45](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=45s) Uh, and our mission is to create a world where everyone has access to the food they love and more time to enjoy it together.

[0:52](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=52s) So coming to what we'll actually talk about today, uh, first we'll talk about the importance of search, uh, in grocery e-commerce. Uh, then we'll look into some of the challenges facing conventional search engines, uh, and then actually get to the meat of the talk today, which is how we are using LLMs to solve some of these problems.

[1:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=70s) Uh, finally, we'll, uh, finish with some key takeaways from today's talk.

[1:15](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=75s) So coming to the importance of search in grocery commerce, uh, I think we've all gone grocery shopping. Customers come with long shopping lists, uh, and it's the same on the platform as well.

[1:25](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=85s) People are looking for tens of items, uh, and of these, the majority of them are just restocking purchases. That is, things that the customer has bought in the past.

[1:35](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=95s) Uh, and the remaining are items that the user is trying out for the first time. So, uh, and, and the majority of these purchases come from search. So, uh, search has a dual role.

[1:45](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=105s) It needs to both support, uh, quick and efficient... Uh, it needs to help the customer quickly and efficiently find the product they are looking for and also enable this new product discovery.

[1:56](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=116s) Uh, and new product discovery isn't just important for the customer. It's also great for our advertisers because it helps them showcase new products, uh, and it's also good for the platform because overall it increases larger basket sizes.

[2:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=130s) Uh, so, so let's see what some problems are with our existing setup that sort of, uh, makes this hard. Uh, so, so to begin with, uh, we have two classes of queries that are, are generally more challenging, especially from an e-commerce perspective.

[2:24](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=144s) Uh, the first are overly broad queries, uh, in this case, like on the left, the snacks query, where there are tons of products that map to that query. Uh, and now because our models are trained on engagement data, if we aren't exposing these products, uh, to the user, it's hard to actually collect engagement data to, to then

[2:43](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=163s) rank them up high. So the traditional cold start problem in a way. Uh, then, uh, as you can see on the query on the right, we have very specific queries like unsweetened plant-based yogurt, where the user is looking for something very specific, and these queries, uh, don't happen very frequently, which means that we just don't have enough

[3:01](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=181s) engagement data to train the models on. Uh, and while we have, uh, done quite a bit of work to sort of, um, improve this, uh, the challenge that we continually keep facing is that while recall improves, uh, precision is still a challenge, especially in a pre-LLM world.

[3:18](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=198s) Uh, the next class of problems is how do we actually support that new item discovery, as we spoke about. So when a customer walks into a, a grocery store, uh, let's say into a pasta aisle, they might see new brands of pasta that they would want to try out.

[3:32](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=212s) Uh, along with that, they would also see pasta sauce and every other thing that's needed to make a bowl of pasta. Uh, and customers would want a similar experience on our site.

[3:42](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=222s) Uh, we have heard multiple feedback, uh, multiple rounds of feedback from our customers that, "Hey, I can find the product that I'm-- that I want via search, but when I'm trying to find any other related products, it's, it's a bit of a dead end.

[3:55](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=235s) I would need to make multiple searches to get to where I want to." So this was a problem that we wanted to solve as well. And yeah, as I mentioned, uh, pre-LLMs, this was a, a hard problem because of the lack of engagement data, et cetera.

[4:09](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=249s) So let's see how we actually use the LLMs to sort of solve these problems. Uh, I'll sort of talk specifically about how we use the LLMs to up-level our query understanding module.

[4:20](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=260s) Now, query understanding, as I'm sure most of you know, uh, is the most upstream part of the search stack, uh, and very accurate outputs are needed to sort of enable better retrieval and recall, uh, and finally improve our ranking results.

[4:35](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=275s) Uh, so our query understanding module has multiple models in them, like, uh, query normalization, query tagging, query classification, uh, category classification, et cetera. So in the interest of time, uh, I'll just, uh, pick a couple of models and talk about how we, uh, sort of really improved them.

[4:52](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=292s) Uh, the first is our query to our category-- a product category classifier. Uh, essentially, we are taking a query and mapping it to a category in our taxonomy. Uh, so as an example, if you take a query like watermelon, that maps to categories like fruits, organic food, foods, et cetera.

[5:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=310s) Uh, and our taxonomy has about ten thousand labels, of it six thousand are more commonly used. So because a product-- a query can map to multiple labels, this is essentially a multi-label classification problem.

[5:23](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=323s) Um, and in the past, our traditional models, uh, which were... We actually had a, a couple of different models. One was a fastText-based, uh, neural network, which essentially modeled the semantic relationship between the query and the category.

[5:39](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=339s) Uh, and then as a fallback, we had an NPMI model, which was a statistical co-occurrence model between the query and the category. Now, while these, uh, techniques work great for the, for the head and torso queries, we had really low coverage for our tail queries because, again, we just didn't have enough engagement data to train the models

[5:57](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=357s) on. Um, and to be honest, we actually tried more sophisticated BERT-based models as well. Uh, and while we did see some improvement, the lack of engagement data, uh, meant that for the increased latency, we didn't see the wins that we actually hoped for.

[6:13](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=373s) So, um, so this is where we actually tried to use an LLM. Uh, first, we took all of our queries, uh, and we-- along with the taxonomy, we fed it into an LLM and asked it to predict the most relevant categories for that query.

[6:26](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=386s) Now, the output that came back was decent. Uh, actually, when we all looked at it, it made a lot of sense. Uh, but when we actually ran an online A/B test, the results weren't as great.

[6:36](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=396s) Uh, and one particular example that illustrates this point very well is a query like protein. Uh, our users that come to Instacart when they type something like protein, they're looking for maybe protein shakes, uh, protein bars, or other protein supplements.

[6:51](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=411s) The LLM, on the other hand, thinks that pro-- when a user types protein, they're looking for maybe chicken, tofu, or other protein foods. So this mismatch, wherein the LLM doesn't truly understand Instacart user behavior, was really the cause of the problem.

[7:08](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=428s) So to sort of maybe improve our results, we sort of switched the problem around. We took the most commonly converting categories or the top-K converting categories for each query and fed that as additional context to the LLM.

[7:22](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=442s) Uh, and then I'm sort of simplifying this a bit. There's a bunch of, uh, ranking and downstream validation that happens, but essentially that, that is what we did. We generated a bunch of candidates, uh, ranked candidates, and this greatly simplified the problem for the LLM as well.

[7:38](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=458s) Uh, and again, to illustrate this with an example, uh, take a query like Vernors soda. Uh, our previous model actually identified this as a, as a brand of fruit fla-- or a fruit-flavored soda, which is not incorrect, uh, but it's not very precise either.

[7:53](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=473s) Now, the LLM, uh, did a much better job. It identified it as a brand of ginger ale. And with this, our downstream retrieval and ranking improved greatly as well.

[8:02](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=482s) And as you can see from, uh, the results below, uh, especially for tail queries, we saw a big improvement. Our precision improved by over eighteen percentage points and our recall improved by over seventy percentage points, which is actually pretty significant for our tail queries.

[8:17](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=497s) Um, and maybe to, to very briefly look at our prompt. As you can see, it's very simple. Uh, we are essentially passing in the ca-ca-- the top converted categories as context.

[8:27](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=507s) Uh, there are a bunch of guidelines about what the LLM should actually output, and, and that's it. So this was all that was needed to sort of enable this.

[8:35](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=515s) Uh, again, I'm simplifying the overall flow, but, uh, the general concepts are pretty straightforward.

[8:43](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=523s) So coming to the-- another model. Uh, the query rewrites model is actually pretty important as well, uh, from, uh, a g-- uh, from an e-commerce perspective, especially at Instacart, because not all retailers are created equal.

[8:57](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=537s) Some have large catalogs, some have very small catalogs. The same query may not always return results, and that is where a rewrite is really helpful. For example, going from a query like one percent milk to just milk will at least return results that the customer can decide to buy or not.

[9:12](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=552s) Uh, and again, our previous approach, which was trained on, uh, engagement data, didn't do too well. It suffered... Or it did decently well on head and torso queries, but it suffered from a lack of engagement data on tail queries.

[9:26](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=566s) Uh, so by using an LLM similar to how we did for the product category classifier, uh, we were able to generate very precise rewrites. Uh, uh, in the example here, you can see that there's a, a substitute, a broad, and a synonymous rewrite.

[9:40](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=580s) So for the case of avocado oil, a substitute is olive oil, a broader rewrite is, um, healthy cooking oil, and a synonymous rewrite is just avocado extract. And again, uh, just, just looking at the results from this, and if you-- we saw a bunch of offline improvements and just moving from, uh, from thir-- using third-party LLMs here,

[10:02](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=602s) just going from more simpler models to better models improved, uh, the results quite a bit. Uh, this is based off of our human evaluation data. Uh, so as you can see, just improving the models itself improved the overall performance of the task.

[10:16](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=616s) And in terms of online improvements, we actually saw a large drop in the number of queries without any results. This is pretty significant again because, uh, we could now actually show results to users where they previously saw empty results, uh, which was great for the business.

[10:34](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=634s) So, uh, coming to the so-sort of the important part of this, which is how we actually stored and served the, the data.

[10:43](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=643s) Uh, the thing is that Instacart has a pre-- uh, has a pretty idiosyncratic, uh, query pattern. There's a very fat head and torso set of queries, and we have a sort of a long tail.

[10:55](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=655s) So by compu-- pre-computing the outputs for, for all of the head and torso queries offline in a batch mode, we were able to sort of, uh, cache all of this data and then at online, when a query comes in, we could just serve it off of the cache with very low impact on latency, uh, and fall back

[11:14](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=674s) to our existing models for the long tail of queries.

[11:17](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=677s) And again, this worked really well because it didn't, uh, impact our latency while it greatly improved our coverage for the long tail of queries. Now, for the, the really long tail where I said we would fall back to our existing models, we're actually trying to replace them with our distilled Llama 8B model, uh, so that we

[11:38](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=698s) can actually do a much better job compared to the existing models. Um, so yeah, to sort of summarize, uh, essentially what we saw was that, uh, from a query understanding perspective, we have a bunch of models, uh, and just using our hybrid approach greatly improved their performance.

[11:56](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=716s) But what's actually more interesting is that today, query understanding consists of a bunch of models, and as Yuzu was talking about in the Netflix talk, managing all of these models is actually complex from a system perspective.

[12:08](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=728s) So Uh, consolidating all of these into an SLM or a, or maybe a large language model, uh, can make the results a lot more consistent. And I'll finish it off by giving an example here.

[12:20](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=740s) Uh, there's a query, humm, that we sort of saw some interesting issues with, uh, which is pr- pr- which is spelled H-U-M-M. Uh, the actual query, the-- our, our query brand tagger identified the brand correctly as a brand of, uh, kombucha, but then our spell corrector unfortunately corrected it as hummus.

[12:41](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=761s) So, the results were really confusing to users, uh, and was pretty bad. But by using a more unified model, I think the results were much better. The second is by passing in...

[12:51](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=771s) by using an LLM for query understanding, uh, we can actually pass in extra context. Um, so instead of just generating results for that query in isolation, we can tr- really try to understand what the customer's mission is.

[13:05](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=785s) Um, so for example, detect if they're actually here to buy ingredients for a recipe, et cetera, and then generate the content for that. So, to talk more about that, uh, I have [REDACTED:username] here.

[13:17](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=797s) Thank you, Vinesh. Uh, now I'll quickly talk about how we used LLMs for showing more discovery-oriented content in search results page. Uh, just to re- uh, restate the problem, uh, our users found that while our search engine was very good at showing exactly the, the, the results that they exactly wanted to see, once they added an item

[13:35](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=815s) to the cart, they couldn't do anything useful with the search results page. They either had to do, like, another search or go to another page to fulfill their next intent to some sort.

[13:44](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=824s) Uh, tradit-- Solving this with traditional methods would require, like, a lot of feature engineering or manual work. Uh, LLMs solved this problem for us, and I will talk about how.

[13:53](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=833s) Uh, so this is how it looked in the end. So, for queries like swordfish, uh, let's say there are no exact results. We used LLMs to generate substitute results like other seafood alternatives, meaty fish like tilapia and whatnot.

[14:07](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=847s) Um, and similarly for queries like sushi, where there were a lot of e-exact results, let's say, uh, we would show at the bottom of the search results page, we would show things like Asian cook-cooking ingredients or Japanese drinks and so on, uh, in order to, like, you know, uh, get the users to engage.

[14:23](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=863s) Uh, I'll talk about the techniques here, but, uh, both of these, uh, both of these discovery-oriented results we saw, like, improv-- uh, led to, like, improvement in engagement as well as improvement in revenue, uh, for our-- for each search.

[14:37](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=877s) Uh, cool. Uh, like I said, I'll get into the techniques, but let's first talk about the requirements to generate such content. Uh, first, uh, obviously, we want to generate content that is incremental to the current solutions.

[14:47](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=887s) We don't want duplicates to what we were already showing. And the second requirement, and the most important one, is we wanted all of the LLM answers or, or the generation to be aligned with Instacart's domain knowledge.

[14:59](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=899s) What does this mean? So if a queries-- If a user searches for a query called dishes, LLM should understand that it refers to, like, cookware and not food, uh, and vice versa for a query like Thanksgiving dishes, right?

[15:10](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=910s) So, with these requirements in mind, we set up with-- We started with, like, a very basic generation approach. So, what did we do? We took the query, and we told the LLM, "Hey, you are an AI assistant, and your job is to generate two shopping lists.

[15:23](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=923s) One is a list of complementary items, and another is a list of, like, uh, substitute items for a given query," right? Um, looked good. Uh, I mean, like, so, so we, we saw the results.

[15:32](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=932s) They looked pretty good. Uh, our PMs vetted everything. We looked at everything. Uh, and, and like Vinesh said in, in, like, QU, we-- when we launched this to our users, uh, we saw that the results were good, but users weren't engaging it as much as we would have liked it to.

[15:48](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=948s) So, we went back to the drawing board, and we were like, uh, we, we tried to analyze what was going on. And what we realized quickly was, while LLM's answers, uh, were like common sense-like answers and so on and such, they weren't really what users were looking for.

[16:02](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=962s) Uh, taking the protein example again, like, uh, users, when they search for protein, they look for protein bars and protein shakes rather than what LLM would give as an answer, which is chicken, turkey, and tofu and whatnot, right?

[16:14](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=974s) So, uh, so what we did was we augmented the prompt with Instacart domain knowledge. So, f- uh, in one case, what we did was we took the query, and then we augmented it with, like, "Hey, here is the query, and here are the top converting categories, uh, for this particular query," along with any annotations from the query

[16:32](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=992s) understanding model like, "Hey, here is a brand present in the query. Here is, like, a, a dietary attribute present, present in the query," and so on and such. Uh, in another case, we were like, "Here is the query, and here are the subsequent queries that users did once they issued this particular query."

[16:47](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1007s) So, once you augmented the prompt with this additional metadata about how Instacart users behave, the, the, the results were far more better. Uh, I don't have the time to show, like, like, the before and after, but like I said, we definitely saw, like, a huge improvement in both engagement as well as revenue.

[17:04](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1024s) Uh, I'll quickly talk about, like, how we served, uh, all of these contents. Uh, like, very similar to QU. It's impractical to call the LLM in real time because of latency and maybe cost concerns sometimes.

[17:16](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1036s) So, what we did was, uh, we took all of our s- uh, historical search logs. We called LLM in, like, a batch mode and stored everything. So query, content metadata, along with even the products that could potentially show up in the carousel.

[17:29](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1049s) And online, it's just a very quick lookup from a feature store. Uh, and that's how we were able to, like, uh, serve all of these recommendations in, like, blazing fast time.

[17:39](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1059s) Uh, again, things weren't as simple as, as we making them out to be. The, the, the... Like Vinesh said, the overall concept is simple. The, the prompt itself is very simple.

[17:48](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1068s) But there were three key challenges that we solved along the way. Uh, one is aligning generation with business metrics like revenue. Uh, this was very important to see, like, top-line wins.

[17:57](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1077s) So we iterated over the prompts and the kind of metadata that we would fe- that we would feed to the LLM in order to achieve this. Second, we spent a lot of time on ranking, uh, on improving, improving the ranking of the content itself and so on and such.

[18:11](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1091s) So our traditional PCTR, PCVR models did not work, so we had to like employ strategies like, uh, diversity-based re-ranking and so on and so forth to get users to engage with the content.

[18:21](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1101s) Uh, and then the third thing is evaluating, uh, the content itself. So one is making sure that, hey, whatever LLM is giving, uh, is one, right, it's not hallucinating something.

[18:30](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1110s) Uh, and second, it adhered to like what Instacart or what we need as a product, right? Cool. Uh, so summarizing the, the key take-takeaways from our talk. Uh, LLMs world knowledge was super important, uh, to improve, uh, query understanding predictions for, especially for the tail queries.

[18:47](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1127s) Uh, while LLMs were super helpful, we really found success by combining the domain knowledge of Instacart with LLMs, uh, in order to see the top-line wins that we saw.

[18:57](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1137s) Uh, and the third and the last one is evaluating the content a-as well as the QU predictions and so on and such was far more important and far more difficult, uh, than we anticipated.

[19:07](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1147s) We, we used LLMs as a judge in order to make this happen, but very, very important step we-- and we realized that kind of late. So yeah, that's all from us.

[19:15](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1155s) We'll take questions now. [audience applauding] Thank you, [REDACTED:username]-[REDACTED:username], Vinesh. Um, yeah, we'll take questions at the mic while the next speaker gets set up.

[19:24](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1164s) Hi. Uh, thanks for the talk. Um, have you also been trying around queries which are very long in natural language? Like, uh, I want these three items and these five items, like what we would do it on Chat-ChatGPT.

[19:38](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1178s) Or it's still like single item that's the focus?

[19:43](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1183s) Uh, yeah. I, I think we have... We actually launched something in the past, uh, like Ask Instacart, if you've heard, heard of it, which essentially takes natural language queries and tries to map that to search intent.

[19:55](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1195s) So for example, you might ask-- you might say, "Healthy foods for a [REDACTED:age] baby," or something like that. And so that would map to things like fruit slices. Uh, I don't know if [REDACTED:age] toddlers can eat popcorn, but something along those lines.

[20:09](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1209s) And, and then we had our usual ranking, uh, recall and ranking stack sort of retrieve those results.

[20:16](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1216s) So, uh, any learnings from that experiment for you?

[20:19](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1219s) Yeah. So, so I think we actually have a lot of learnings from that. Essentially, uh, as [REDACTED:username] already mentioned, uh, we need to inject a lot of Instacart context into the model to be able to get re-decent results.

[20:31](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1231s) The evaluation part is really key, uh, so having a robust automated evaluation pipeline was important. And lastly, passing context. That is, for example, if it's a...

[20:42](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1242s) Let's say it's a Mother's Day query, and let's say we come up with the individual search intent as perfumes. You really want women's perfumes to be in there. Whereas when we just had perfumes, we could see all kinds of items.

[20:56](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1256s) So passing that context from the LLM to the downstream systems is really important.

[21:00](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1260s) Cool. Thanks.

[21:01](https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=1261s) Yeah, we have a lot of examples where we failed we can talk about. [upbeat music]
