AI Engineer World's Fair 2025
Layering every technique in RAG, one query at a time
Read the talk
Layering RAG techniques, one failing query at a time
A CRM agent’s launch bar provides a practical way to choose RAG techniques: inspect failing queries, repair the relevant layer, and add complexity only when the product needs it.
From a talk by David Karam
Before you start: Familiarity with LLM context windows and the basic retrieve-then-generate pattern is helpful; retrieval and ranking mechanisms are explained as they arise.
What would make a CRM agent ready to launch?
What would convince you that a CRM agent is ready for customers? Choosing between BM25 and vector retrieval cannot answer that question. First, you need to know what customers will ask and what successful answers look like. David Karam brings a search engineer’s approach to this problem: he and his co-founder worked on Google Search before starting Pi Labs, and their habit is to inspect individual queries—what worked, what failed, and why. That makes the product’s launch bar, rather than a debate about whether RAG is dead, the starting point for engineering.
The launch bar becomes concrete through application-specific query sets. Start with easy cases, then add medium and hard ones as the product’s ambitions grow. Techniques are interventions intended to move performance on those cases toward the required quality. A benchmark disconnected from the application cannot tell you whether your CRM agent is ready; an evaluation built around its actual work can.
The quality engineering loop has three steps:
- Establish a baseline. Run the simplest system that could satisfy the initial query set.
- Analyze the losses. Inspect unsuccessful cases and identify what broke.
- Choose an intervention. Add a technique that addresses the observed failure, then evaluate again.
Asking whether you need BM25 or vector retrieval before examining the query stream reverses this order. You may build a component that solves none of your current problems.
Karam calls the decision criterion complexity-adjusted impact: weigh the likely improvement against the effort required. In a catalog of techniques, difficulty and impact are the most useful columns. BM25 is relatively easy to try; custom embeddings can demand substantial work. Harvey’s custom legal embeddings illustrate the latter kind of investment—a difficult domain can justify customization when generic retrieval does not meet the quality bar. The point is to earn that complexity through evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Begin with the documents in context
The simplest baseline may not need a separate retrieval system. Karam calls this in-memory retrieval: put all the documents into the LLM’s context and ask the question. He uses NotebookLM and a five-document question-answering scenario to make the interaction tangible. That is an illustration of a document-grounded product, not evidence that NotebookLM itself operates without retrieval. For your own small collection, however, supplying the whole collection is a straightforward place to start.
This baseline gives you specific failure modes to look for. The documents may no longer fit. Irrelevant material may crowd the context. Or the model may fail to attend to the documents that contain the answer. Those are different reasons to introduce selection, but each comes from an observed case. Once you can point to the documents or questions that break the baseline, retrieval has a concrete job: supply the useful evidence without carrying everything else along.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match retrieval to the language of the queries
BM25 offers a simple first retrieval layer. Its scoring considers query terms, their frequency in a document, document length, and how rare the terms are across the collection. It works well when the query and useful documents share discriminative words. When the user expresses the same need with different language, term matching can miss the connection.
Relevance embeddings address that gap by representing queries and documents in vector space. Distance or similarity between those representations can capture relationships beyond shared words. But vector retrieval has its own weaknesses, including cases where exact keyword matching matters. The choice depends on the language and failure patterns in your query set.
Karam uses examples generated with ChatGPT to contrast the two query styles:
| Query style | Example | Retrieval strength to investigate |
|---|---|---|
| Keyword phrase | iPhone battery life | Direct term matching |
| Conversational question | How long does an iPhone last before I need to charge it again? | Semantic matching across paraphrases |
The second question describes battery life without using that phrase. If your failed queries mostly look like that, Karam strongly recommends investing in vector retrieval. If your query stream consists of precise keyword phrases, lexical retrieval may already do the necessary work. These examples help classify failures; your evaluation determines whether another retrieval layer improves them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve broadly, then score the candidates together
After adding lexical and vector retrieval, you may have useful candidates but conflicting orderings. A retrieve-and-rerank pipeline separates finding candidates from judging them more carefully. With a bi-encoder, the query and document receive independent embeddings, which are then compared. A cross-encoder takes the query and document together and produces a score while attending to both inputs. That joint processing can make finer relevance judgments.
| Stage | What it processes | Role |
|---|---|---|
| Embedding retrieval | Independently encoded queries and documents | Find candidates efficiently |
| Cross-encoder reranking | Query–document pairs jointly | Improve candidate ordering |
Joint scoring is more expensive, so at corpus scale the usual pattern is to retrieve a broad candidate set and rerank a smaller subset. The restriction is about scale, not a prohibition: a small paragraph collection can be scored directly. Once the collection grows, spending the expensive model only on plausible candidates makes the pipeline practical.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognize when generic relevance stops being enough
Better relevance scoring still leaves a deeper problem: relevance is a proxy for the application’s information need. A document can be semantically close to a question without being the right document to use. Karam draws on Google Search’s long history of using signals beyond relevance, then turns to the legal retrieval example discussed by Harvey and LanceDB. In law, ordinary-looking words can carry specialized meanings that generic matching does not reliably capture.
Words such as regime and material have domain-specific significance; the required evidence may be a particular law or regulation. Custom embeddings adapt retrieval to those meanings. That does not require starting a model from scratch: Harvey’s work with Voyage adapted an existing legal embedding model. The engineering goal is to represent the distinctions that matter in the domain well enough to retrieve the right evidence.
Karam also asks ChatGPT for illustrative legal queries that could expose generic retrieval failures, including vocabulary such as moot and material. Such examples suggest what to investigate, but the decision to customize comes from your own query sets. Are failed retrievals consistently associated with specialized vocabulary that the generic model handles poorly? If so, domain adaptation has a specific target. If not, it may be an expensive answer to the wrong problem. Even successful custom embeddings primarily improve retrieval and recall; ordering those retrieved items remains another task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry structured meaning into ranking
Shopping makes the difference between relevance and usefulness especially clear. Karam reports asking Perplexity for cheap gifts for his son and receiving suggestions around $10. That interpretation of cheap was reasonable enough; he then clarified the intended price range. Karam reports that after specifying a budget of $50 or more, Perplexity still suggested items priced at $15 and $40. The failure in this individual interaction is concrete: both prices violate the stated lower bound.
The missing information is a price signal. Understanding the words is only the beginning: the price condition must survive query interpretation and affect which candidates are returned or ranked. A relevant gift below the lower bound is still an unsuitable result. Once the request has been interpreted as minimumPrice: 50, the constraint can be represented explicitly:
typescript
type Gift = {
id: string;
price: number;
};
const request = { minimumPrice: 50 };
const suggestions: Gift[] = [
{ id: "gift-a", price: 15 },
{ id: "gift-b", price: 40 },
];
const eligible = suggestions.filter(
gift => gift.price >= request.minimumPrice
);
console.log(eligible); // []
This small implementation of the constraint rejects both reported suggestions. An empty eligible set is a reason to retrieve additional candidates, rather than quietly drop the user’s requirement.
Other signals depend on the corpus and the product:
- Merchant information: Shopping results can depend on properties of the seller, not just the product description.
- Audience behavior: Podcast listen counts can help distinguish otherwise relevant candidates.
- Link structure: PageRank captures prominence through links pointing to a page. It reflects the structure of the web, rather than simply matching the page’s text to a query.
Karam distinguishes broad natural-language meaning from the vertical semantics of a particular domain. Shopping, CRM, and email each contain relationships and constraints that semantic relevance alone does not express. As the product’s quality bar rises, those domain signals become a larger part of ranking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let observed preferences change the score
Even a domain-aware ranking system can misunderstand its users. You may account for relevance, prices, and merchants, then discover that people click items you expected them to ignore and ignore items you expected them to choose. That is another loss pattern, requiring another source of evidence. Clicks, thumbs-up, and thumbs-down provide signals about preferences that the initial domain model did not capture.
Karam sketches a ranking function that combines a click-through prediction signal with relevance and structured domain signals. The final score balances natural-language relevance, price-related query understanding, and user preference. This adds a prediction problem: observed feedback must become a signal that can be applied to future candidates. The talk stops at the architecture rather than specifying how to train or weight that predictor. The practical distinction is that knowing the shopping domain and knowing what users prefer are separate capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control what the agent asks the search engine
The next failure can happen before ranking begins: the agent asks the wrong query. A search engine may be tuned for keyword phrases or another specific query form, while an LLM sends a broad request expressed in the language of the overall application. This mismatch is a tool-boundary problem. Improving the ranker will not necessarily repair a request the backend cannot interpret as intended.
Query fan-out gives the application more control. Decompose a complex request into narrower searches, retrieve their results, and use those results together. Karam points to Google AI Mode as an example. Karam describes seeing AI Mode’s brief animation indicate fifteen or twenty queries. Those counts are his observation, not a fixed fan-out setting; the mechanism is decomposition across subtopics and information sources.
An LLM may already attempt decomposition, but it does not automatically know which searches work well against your backend. Karam is enthusiastic about MCP while skeptical that prompting alone can communicate all of a search engine’s behavior. There are two possible directions: make the backend understand more complex requests, or give the agent enough knowledge to tailor its requests. Until that boundary works reliably, explicit orchestration provides control over query quality.
Consider What is David working on? The application may need to turn that into searches for David’s Jira tickets and Slack threads. The useful decomposition depends on knowing where work is recorded, not merely understanding the sentence.
| Assistant input | Narrower searches | Application knowledge required |
|---|---|---|
What is David working on? | Jira tickets David; Slack threads David | Tickets and conversations contain evidence of ongoing work |
Sending the original question unchanged leaves that domain interpretation to the search engine. If the engine lacks it, the request can fail despite a perfectly understandable user question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask more backends when intent is ambiguous
Correctly formed queries can still go to too few sources. Karam calls the remedy supplementary retrieval: search additional backends when the current set cannot provide enough recall. He sees teams minimize search calls too early, before they have established which information sources the task needs. Broadening retrieval gives the system more evidence to work with.
His example is the query falafel. In Karam’s telling, this apparently simple query stumped a Google Search organization of six thousand people. The difficulty was ambiguous intent. A person might want somewhere to eat falafel, making restaurants useful, or might want to see what it looks like, making images useful. The same short query supports multiple result types.
Karam describes Google’s response as querying all the relevant backends and combining the results. This is different from splitting one complex question into subquestions: here, a short query needs broader source coverage because its meaning is underdetermined. His recommendation is to avoid being stingy with retrieval while recall is the problem. Actual cost pressure is the reason to reconsider that breadth.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce serving cost without lowering the quality bar
Eventually, broad retrieval and query fan-out can become expensive. Karam introduces this stage with a server-rack image that he describes as a real advertisement involving a rack thrown from a roof. The joke gives way to an operational problem: more backends, more queries, and more model work increase the complexity and cost of serving each request. At that point, distillation becomes worth considering.
The objective is to reduce model size while holding the application’s quality bar constant. A general-purpose LLM can be overqualified for a narrow task: the product may only need one capability, such as question answering, rather than the full range of the model’s abilities. Distillation and task specialization require fine-tuning expertise, so this is a substantially harder intervention than trying another retrieval method.
Karam attributes Perplexity’s speed in some contexts to a model specialized for question answering, though he supplies no measured distillation result. His hypothetical retention example contrasts ten-second responses that cause users to leave with two-second responses that keep them. The reason to invest is the product consequence of latency: once response time threatens adoption, preserving quality with a smaller, faster model can justify the training work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the interface promise only what the system understands
What happens when you have applied the techniques and cases still fail? Karam jokes that engineers should blame the product manager, then gives the joke a serious meaning. Stochastic systems will retain failures. Product design must decide how much responsibility the system should take and what happens when it cannot reliably fulfill that responsibility. Engineering alone cannot make every product promise safe.
Customer support makes this responsibility explicit. Some cases can be handled by the bot; others need escalation to a human. The handoff is part of the user experience, not simply a missing model capability. The product must define when the machine can act on its own and when a person needs to participate.
Google Shopping provides a more visible spectrum. Where Google has strong structured understanding, it can offer a high-promise UI with product choices, reviews, and filters. Where it understands the material mainly as web documents, it offers a simpler experience that leaves more interpretation to the user.
| Understanding | Interface promise | User’s role |
|---|---|---|
| Strong product understanding | Structured choices, reviews, filters | Refine a well-understood result set |
| Limited document understanding | A list of possible results | Inspect and choose |
Karam contrasts offering ten possibilities with presenting one result when the system understands the need well enough. The interface should gracefully move up or down that spectrum. A less assertive experience can remain useful while avoiding a promise the underlying system cannot support.
That returns the engineering decision to the query set. Context windows versus RAG, or autonomous agents versus workflows, become empirical questions about what improves this application. Baseline the system, inspect its losses, and look for an easy intervention. Move to medium-complexity work when needed; hire for genuinely difficult work when the remaining failures justify it. The next layer should follow a demonstrated need, because sophistication added too far ahead of the product can consume substantial effort without moving its launch bar any closer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- Introducing NotebookLMArticle
Google's original explanation of document-grounded questions, summaries and source citations in NotebookLM.
Sentence Transformers explains bi-encoder retrieval and cross-encoder reranking, with runnable examples.
How Harvey adapted legal embeddings with case-law training and expert annotations, including its retrieval evaluation.
A production account of legal retrieval, evaluation, privacy, ingestion and vector database requirements.
The March 2025 announcement explains query fan-out across subtopics and multiple information sources.
Read the complete timestamped transcript
- 0:00
[on-hold music] Uh, I'll, I'll just give you all a, a little bit of context.
- 0:17
So, uh, my co-founder and I and a lot of our team were actually working on Google Search, and then we left and, like, started Pi Labs. And, uh, I, I loved, I loved the exit talk and, like, we're all nerds for information retrieval and search and, uh, so there's gonna be a little bit of that.
- 0:30
Uh, just gonna go through a whole bunch of ways you can actually shore up and improve your RAG systems. Uh, I think one thing that I personally, uh, sometimes struggle with is there's a lot of talk about things sometimes, like, too much in the weeds, like, oh, specific techniques, and you do RL this way, and you can
- 0:44
tune the model this way. And it's like, doesn't help me orient in the space. Like, what are all these things and how do I, like, h-hang on them? Uh, or you have the complete opposite, which is, like, a whole bunch of buzzwords and hype and such, and like, "RAG is dead.
- 0:55
No, RAG is not dead." It's like, "Agents?" Like, wait, what? Like, [chuckles] uh, so just, you know. I think a lot of what I'll do today is just, uh, what I call, like, plain English.
- 1:03
Uh, just trying to, like, set up a framework, right? Like, very centered around, like, okay, if you are trying to shore up the quality of your system, how do you do that?
- 1:10
And then where do all the things you hear about, like, day in, day out, like, fit? Uh, and then just how to approach that. And we'll give a lot of examples.
- 1:17
I think one thing that I always love, and we always did in Google, we always do in Pi Labs, uh, is just, like, look at things, look at cases, look at queries, see what's working, see what's not working.
- 1:26
And that's really the essence of, like, quality engineering, uh, as we used to call it at Google. If you do want the slides, there's like fifty slides, and I set my ch- a challenge for myself to go through fifty slides in nineteen minutes.
- 1:36
Uh, but you can catch the slides here if you want. Uh, I'll flash this towards the end as well, withpi.ai/rag-talk. Uh, it should point to the slides that we're going through.
- 1:44
And as I mentioned, plain English, no hype, no buzz, uh, no debates, no like... All right. So how to think about techniques. Before we go into techniques and get into the weeds of it, like, wh-wh-why does this even matter?
- 1:54
And the way we always think about it is like, always start with outcomes. You're always trying to solve some product problem. Uh, and generally, the best way to visualize something like this, you have a certain quality bar you wanna reach.
- 2:04
And there were a, a very interesting talk th-this, this week about like, you know, benchmarks aren't really helpful. And absolutely, evals are helpful. You're trying to launch a CRM agent, and you sort of have a launch bar, like a, a place where you feel comfortable that you can actually put it out into the world.
- 2:16
Uh, and techniques fit somewhere here. You have that, like, kinda end metric, and you're trying to, like, come up with different ways to shore up the quality, and those ways are, like, sort of the techniques there.
- 2:26
And, you know, this is sort of your own personal benchmark. You start with some of the easy te- uh, the easy, the easy bars you wanna hit, and then there's like medium benchmarks and hard benchmarks.
- 2:34
So these are query sets you're setting up. Uh, and then, you know, depending on what you wanna reach and in, at what timeframe, uh, then you end up trying different things.
- 2:43
Uh, and this is what we call, like, quality engineering loop. You sort of like baseline yourself. Okay, you wanna achieve, you know, you want CRM, and this is the easy query set, and your quality's there, uh, just through the simplest way you can try it.
- 2:54
Do a loss analysis. Okay, what's broken? There were a lot of eval talks this week. And then what we call quality engineering. Now, the reason I, I, I, I say this is because like, okay, techniques fit this, in this last bucket.
- 3:04
And one of the things that I think biggest problems is like people sometimes start there, and it doesn't make any sense because you say, "Oh, do I need BM25 or do I need like, uh, vector, vector retrieval?"
- 3:14
It's like, I don't know. What, what are you trying to do? And what is your query sets and where are things failing? 'Cause many times you actually don't need these things, and you end up implementing them, and it doesn't make a lot of sense.
- 3:22
Anyhow, so usually the thing I say is, like, what I call complexity-adjusted impact or, you know, stay lazy. Uh, in a sense of like always look at what's broken, and if it's not broken, don't fix it.
- 3:32
And if it is broken, do fix it. Uh, and we'll go through a lot of techniques, uh, today, but like this is a good way to think about them.
- 3:38
It's just a cluster. It's a catalog of stuff. The most important two columns are the ones to the right, difficulty and impact. And if it's easy, go ahead and try it.
- 3:45
And most times, like BM25. BM25 is pretty easy. You should absolutely try it, and it does like, you know, shore up your quality quite a bit. Um, but you know, should I build like custom embeddings for retrieval?
- 3:55
Like, I don't know. Let's take a look. This is actually really, really hard. Uh, Harvey gave a talk. They build custom embeddings, but you know, they have a really hard problem space and just, you know, relevance embeddings don't, don't do enough for them.
- 4:05
Uh, and then they're willing to put all that work and effort. All right, queries, examples. Lots of stuff. Let's start with first technique, in-memory retrieval. Uh, easiest thing, bring all your documents, shove them all to the LLM.
- 4:16
Uh, this is the whole like, is RAG dead? Is RAG not dead? Context windows. Well, context windows are pretty easy, so you should definitely start there. Uh, one example, NotebookLM.
- 4:24
Uh, very nice product. You actually, you know, put in five documents, you just ask questions about them. You don't need any RAG. Just shove the whole thing in. Now, questions might get cut too long, and this is where it breaks, right?
- 4:34
Maybe things don't fit in memory, uh, or maybe you just polluted the context window too much. So this is where you start to think like, "Oh, okay, that's what's happening.
- 4:42
I have too many documents. Oh, that's what's happening. The documents are not attended properly by the LLM, and here are like the five things that are breaking." Okay, great.
- 4:49
Let's move to the next one. So now you try something very simple, which is, can I retrieve just based on terms? So BM25. What is BM25? BM25 is kinda like four things: um, query terms, frequency of those query terms, uh, length of the document, and just how rare a certain term is.
- 5:05
It's a very nice thing. It actually works pretty well, and it's very easy to try. And, um, it has a problem that like when things are not have that nature, like the exa-exa-exit poll was saying, when they don't have that nature of like a keyword-based search, they don't work.
- 5:18
And this is where you bring in something like relevance embeddings. And relevance embeddings are pretty interesting 'cause now you're in vector space. And vector space can handle way more nuance, uh, than like keyword space.
- 5:28
Uh, but you know, they also fail in certain ways, especially when you're looking for keyword matching. And it's actually pretty easy to know when things work and when they don't.
- 5:35
Actually, this was created, like I went to ChatGPT and I asked like, "Hey, give me a bunch of keywords, ones that work for like standard term matching and ones that work for relevance embedding."
- 5:42
And you can see like exactly what's going on here, right? If your query stream looks like iPhone battery life, then you don't need ve-like re-, uh, vector search. But if they look something like, "Yo, how long does an iPhone like last before I need to charge it again?"
- 5:55
Then you absolutely need like things like vector search. And this is where you need to be like tuned to what, what every technique gives you before you go in and invest in it.
- 6:02
And when you do your loss analysis and you see, "Oh, most of my queries actually look like the ones on the right-hand side," then you should absolutely start investing in this area.
- 6:10
All right. Now you did BM25, you did vector because your query sets look exactly like that, and now you have con- conflicted candidate set. And this is where re-rankers help quite a bit.
- 6:20
And when people say re-rankers, they're usually referring to like cross-encoders. And this is a specific architecture. If you remember the architecture here for relevant-- for the relevance embeddings was you're getting a vector for the query, and you're getting a vector for a document, and then you're just measuring distance.
- 6:32
Now, cross-encoders are more sophisticated. They actually take both the query and the document, and they give you a score while attending to both at the same time. And that's why they're much more powerful.
- 6:41
Now, they are more powerful, but they're actually pretty expensive. And now this is a failure state as well. You can't do it on all your documents. So now you have to have like this fancy thing where you're retrieving a lot of things and then ranking a smaller set of things with a technique like that.
- 6:55
Uh, but it is really powerful and you should use it, and it fails in certain cases. And now when you hit those cases, then you move to the next thing.
- 7:02
Now, where does it fail? Uh, it's still relevance. And this is a big problem with like, you know, standard embeddings and standard re-rankers. They only measure semantic similarity. And there's a thing like these are all proxy metrics.
- 7:13
At the end, like your application is your application and your set of information needs is your set of information needs. And you try to proxy with relevance. But relevance is not ranking, and this is something, you know, we learned in Google Search sort of, uh, it's been like fifteen, twenty years where, you know, what brings the magic
- 7:26
of Google Search? Well, they look at a lot of other things than just relevance. Uh, and this is, you know, this came from like actually the talk from Harvey and Lance DeVey was really, really interesting.
- 7:35
And he gave the example of this query, right? Uh, and it's a really interesting query. Like, it's-- it has so much semantics for the legal, uh, domain that it's impossible to catch these with just relevance.
- 7:46
Um, and again, what does a word like regime mean? That's a very specific like legal term. Material, what does it mean? It actually very-- has a very specific meaning in the legal term.
- 7:54
Uh, and then there's like things that are very specific to the domain that need to be retrieved, like laws and regulations and such. And this is where you get to building things like custom embeddings.
- 8:02
And you say, "You know what? Just fetching on relevance is not enough for me. And now I need to go and like model my own domain and its own vector space, and now I can actually fetch some of these things."
- 8:12
Now again, go back to ChatGPT like, "Is this interesting? Should I actually even do it?" So I asked it to give me a list of things that would fail in a standard relevance search in the legal domain, and you start to see like, oh, all these things would fail.
- 8:24
The words like moot don't mean the same thing. Words like material don't mean the same thing. And when you have a vocabulary that is so specific and just off, you will not get good results, right?
- 8:35
So now how do you, how do you match that? Like, you need to have, again, you need to have evals. You need to have query sets. You need to look at things that are breaking and decide that, oh, the things that are breaking have to do with the vocabulary just being out of distribution of a standard relevance
- 8:47
model. And that's how you decide, right? So don't like... Again, don't think too much about it. Like, "Oh, should I do it? Should I not do it?" Like what is your, what are your queries telling you?
- 8:55
What is your data telling you? And then go and try to do it or not do it. There's also an example from shopping. Um, so the embeddings are very interesting 'cause they help you a lot with retrieval and recall.
- 9:05
Uh, but you still goo- need good ranking, right? So now if, if, if, if, if you think relevance doesn't work with retrieval, it also probably doesn't work with ranking.
- 9:13
Uh, this is an example I pulled from Perplexity. I was trying, you know, I was just trying to break it today. It didn't take too much to break it.
- 9:18
Uh, I asked like, "Give me cheap gifts, uh, for a gift for my son." And then I follow up with this query like, "But I have a budget of fifty bucks or more."
- 9:24
'Cause when I said cheap, it started giving me like ten dollars. Well, you know, cheap for me is like fifty dollars. Uh, but it didn't know that, so it's fine.
- 9:30
I told it that. But when I said fifty dollars or more, it still gave me fifteen dollars and forty dollars, both of which are actually below, uh, fifty dollars.
- 9:38
Uh, and this is kinda interesting, right? Because what we call like in, you know, in standard terms like for information retrieval, this is a signal. It's a price signal.
- 9:44
And it's not being caught, and it's not being translated into the query, and it's definitely not being translated into the ranking. So now you have to like think of, okay, I have ranking, and I need the ranking to see the semantics of my corpus and my queries.
- 9:56
And this is-- has a very specific meaning. Like whe- when you think of your corpus and queries, again, it's not just relevance. Relevance helps you with natural language. But things like price signals, things like merchant signals, uh, if you're doing like podcasts, how many times has it been listened to is a very important signal.
- 10:10
Has nothing to do with relevance, right? In, in, in, in many, many applications, you will see things that are, for example, more popular tend to rank more highly. Uh, and there's a talk you mentioned like, uh, the PageRank algorithm.
- 10:21
PageRank is not about relevance. It's about prominence. How many things outside of my, uh, document point to me? That has nothing to do with relevance and everything to do with the structure of the web corpus.
- 10:31
So that's the shape of the data. So this is a signal about the shape of the data and not a signal about like the relevance. Um, and you know, best way to think about it, think of it like you have horizontal semantics and then you have vertical semantics.
- 10:42
And if you're in vertical domain where the semantics are very verticalized, right? Let's say you're in-- doing a CRM or you're doing emails, uh, and it's a very complex bar you're, you're trying to hit, uh, that is way beyond just natural language.
- 10:54
Understand that relevance will be a very tiny, tiny part of the semantic universe. And the harder you try to go, the more you're gonna hit this wall and the more you...
- 11:01
All right. This breaks again. Things keep breaking. I'm sorry. [laughs] At, at sufficient complexity, things will keep breaking. So now the thing that breaks with even custom semantics is user preference.
- 11:12
Uh, because even when you get to all this, okay, you're saying, "I'm doing relevance, and I'm doing price signals and merchant signals. I'm doing everything. I-- Now I know the shopping domain."
- 11:19
Now you don't know the shopping domain 'cause now users are using your product. They're clicking on stuff you thought they're not gonna click on, and they're clicking-- they're not clicking on thoughts-- on things you thought they were gonna click on.
- 11:29
Uh, and this is where you need to like bring in the click signal, thumbs up, thumbs down signal. Now, um, these things get very complex, so we're not gonna talk about how to implement them, uh, just because again, in this case, for example, you have to be able-- a click-through, uh, signal, uh, prediction signal, and then you
- 11:43
take that signal and then you combine it with all your other signals. So now if you look at your ranking function, it's doing, okay, I want it to be relevant.
- 11:50
I want it to have this like semi-structured price signal and like query understanding related to that. Plus, I wanna get the user preference in that. And then you take all these signals and you add them, and that becomes your ranking score.
- 12:01
So it becomes a very balanced function. And this is how you go from like, "Oh, it's just relevance," to, "Oh, no, it's not just relevance," to, "Oh, no, it's not just relevance and, and my semantics and my user preferences all rolled up into one."
- 12:14
I'll mention two more things. Um- You calling the wrong queries. This is happening a lot because this go, this goes into more orchestration and you're trying to do complex things.
- 12:23
Uh, especially now when you have agents, uh, and you're telling them to use a certain tool. This is happening quite a bit because there is an impedance mismatch, uh, between what the search engine expects, right?
- 12:33
Let's say you tune the search engine and it expects like keyword queries or expects, uh, you know, even like more complex queries. But you cannot describe all of that to the LLM, and the LLM is reasoning about your application and then making queries by itself.
- 12:45
And this is a big problem. So one thing that we've seen many companies do, we've done this also at Google, you actually take more control of the actual orchestration.
- 12:52
So you take the big query and you make N smaller queries out of it. Uh, I took this screenshot from AI Mode in Google, and it's, it's very brief.
- 13:01
You have to catch it because after, after the animation, it goes away. But you see it's actually, it's making X queries. It's making 15 queries. It's making 20 queries.
- 13:08
Um, so with what, what we call fan-out. Take very complex thing, try to figure out what are all the sub-queries in it, and then fan them out. Now you might think, "Hey, why isn't the LLM doing it?"
- 13:17
The LLM is kind of doing it, but the LLM doesn't know about your tool. It doesn't know enough about your search engine. Uh, I love MCP, but I'm not a big believer that you can actually teach the LLM in like just through prompting what to expect from the search on the other back end.
- 13:30
This is why people still like, "Oh, is this agent autonomous? Do I need to do workflows?" This is very, very complicated, uh, and it will take a while for this to be solved because again, it's unclear where the boundary is.
- 13:40
Is it, uh, is it the search engine should be able to handle more complex things and then the LLM will just throw anything its way? Or is it the other way around?
- 13:46
The LLM has to have more information about what the search engine can support so it can k- tailor it. And right now, uh, you need control because the quality is still not there.
- 13:56
Uh, so this looks like this. Um, you have sort of like this assistant input, and you're turning it into these narrow queries. Like for example, "What is David working on?"
- 14:03
This has very, very specific semantics, and it's more like, oh, Jira tickets David, Slack threads David. Uh, and it's very, very hard to know without knowing enough here about your application that these are the queries that matter and not on the, the ones on the left-hand side.
- 14:15
And if you send the thing on the left-hand side to a search engine, it will absolutely tip over unless it understands your domain. And this is where like, you know, they need to calibrate the boundary.
- 14:25
Okay, so now you're asking all the right queries. Are you asking them to all the right back ends? And this is another place where it all fails. Um, and this is what we call like one technique is we call supplementary retrieval.
- 14:33
This is something you notice like clients do quite a bit, which is they don't call search enough, uh, and sometimes people try to over-optimize. When, when, when you're trying to get high recall, you should always be searching more.
- 14:44
Like I always tell, like just search more. Like this is similar to what we talked like about dynamic content, like the in-memory, uh, uh, retrieval. Just like, just give more things.
- 14:52
So it never fails to give more things. I know in the, in the description we said like there was this query falafel, which was really hard to, uh, to do.
- 14:59
And then you would think like, "Oh, we're in Google Search," and it's a very simple Middle Eastern dish, and it stumped an organization of six thousand people. Like, oh my God, what, what's so hard about this query?
- 15:07
What's so hard about this query is like it's, it's, it's an ambiguous intent. Uh, so you need to reach to a lot of back ends to actually understand enough about it, right?
- 15:14
Because you might be asking about food, at which point I'm gonna show you restaurants. You might be asking this for, for pictures, at which point I'm gonna show you images.
- 15:21
Uh, now what Google ended up doing is that they asked, they, you know, query all the back ends and then they put the whole thing in. And, and I think, you know, I would recommend like this is a great technique to just even increase the recall more.
- 15:31
Just call more things. Um, and don't try to be skimpy unless you're running through like some real cost overload. And that's the last one. You're running into cost overload.
- 15:40
GPUs are melting. I tried to generate an image, but then I realized there's actually a pretty good image that is real. Somebody took a server rack and threw it from the roof.
- 15:48
Um, this was [laughs] just like I didn't need to go to ChatGPT and generate this image. Uh, apparently this was an advertisement, pretty expensive one. Um, all right, so this happens a lot like when, when you get to a certain scale and you have all these back ends and you're making all these queries and it, it's just getting
- 16:04
very, very complex. And this, you know, I mean, Google's there, Perplexity's there. I mean, Sam Altman keeps, keeps complaining about GPUs melting. Um, I think this is the part where like you need to start doing distillation.
- 16:14
And distillation is a very interesting thing 'cause it, like to do that, you have to learn how to fine-tune models. And this, this gets t- to be a little bit complex.
- 16:21
You sort of have to hold the quality bar constant while you decrease the size of the model. Um, the reason you can do that like is, is, is kind of like in that, in that graph.
- 16:29
Like, "Hey, hire me, I know everything." Actually, I'm pranking you. Uh, it's overqualified. Like an LLM, a very like large language model is actually over, mostly overqualified for the task you're, you want to do.
- 16:39
Uh, because what you really want to do is just one thing. Like Perplexity, they're, they're doing question answering. Uh, and they're pretty fast. I mean, when you use Perplexity in certain context, they're really, really fast, which is amazing 'cause they trained this one model to do this one very specific thing, which is just be really, really good
- 16:53
at question answering. Um, and you know, this is very hard, so I wouldn't do it unless, you know, latency becomes a really important thing for your users, right? Like, oh, the thing is taking ten seconds, users churn.
- 17:04
If I can make it in two seconds, users don't churn. Actually, that's a really great place to be because then you can use this technique and like just bring everything down.
- 17:11
Um, all right. You've done everything you can. Things are still failing. This is, uh, [laughs] everybody. Okay, what do you do? Like we have a bunch of engineers here. What do you do when, when everything fails?
- 17:22
Um, yes, you, you, you blame the product manager. [laughs]
- 17:26
It's, uh, [laughs] it's the last trick in the book. Uh, when everything fails, uh, make sure it's not your fault. But I, I'll say there's something really important here. Q-quality engineering will never, like it'll never be a hundred percent.
- 17:36
Things will always fail. These are stochastic systems. So then you have to punt the problem. You have to punt it upwards. Uh, so it's, it's kind of a joke, but it's not a joke.
- 17:43
Like the design of the product matters a lot to how much, how magical it can seem. 'Cause if you try to be more magical than your product surface, uh, can, can absorb, you will, you will run into, into a bunch of problems.
- 17:54
Um, this is... I, I use a very simple example. Uh, probably a more complex one would be, uh, sort of a human-in-the-loop for customer support, where you're like, okay, some cases the bot can handle by its own, but then you need to like punt to a human.
- 18:06
This is basically your X design, right? Like when, when do you trust the machine to do what the machine needs to do, and when does a human need to be in the loop?
- 18:12
This is a much simpler example from like Google Shopping. Um- There's some cases where Google has a lot of great data, so what we call like high understanding. The fidelity of the understanding is really high.
- 18:22
And then it shows like what we call a high-promise UI. Like I'll show you things, you can click on them, there's reviews, there's filters, because I just understand this really well.
- 18:29
And there's things Google does not understand at all, mostly as web documents, bag of words. And what's really interesting about the UI is the UI changes. If you understand more, you show a more kind of like filterable high-promise UI.
- 18:41
If you don't understand enough, you actually degrade your experience, but you degrade it to something that is still workable. Like I'll show you ten things, you choose. Oh, no, I know exactly what you want.
- 18:49
I'll show you one thing. Uh, and this is really, really important. It has to be like part of every... And this is sort of like always understand, like there's only so much engineering you can do until you have to like actually change your product to accommodate this sort of stochastic nature.
- 19:01
So gracefully degrade, gracefully upgrade, depending on like the, the level of your understanding. And again, I'll flash these two slides at the end. Like always remember what you're doing because you can absolutely get into theoretical debates.
- 19:11
Again, context window versus RAG, uh, this versus that. Like is, you know, agents versus I don't know, like just everything is empirical. In this domain when you're doing like this, this sort of thing, "Oh, I have my, my evals, I'm trying to like step by step go up.
- 19:24
I have like a toolbox under my disposal." Everything, everything is empirical. So again, baseline, analyze your losses, and then look at your toolbox and see, are there easy things here I can do?
- 19:36
If not, are there at least medium things I could do? If not, you know, should I hire more people and do like some really, really hard things? Uh, but always remember like the choice is on you, and you should be principled because this can be an absolute waste of time, uh, if you're doing it too far ahead
- 19:49
of the curve. All right. Again, the slides are here, I think. Oh, I, I, I achieved it. Thirty seconds left. Uh, and if you want the slides, they're here again.
- 19:58
And, um, reach out to us. We're always happy to talk. I think I was very happy with the exit talk because it's always nice to find like friends who are nerds in information retrieval.
- 20:07
Um, we are also such, so reach out and happy to talk about, you know, RAG challenges and such, and some of the models we are building. Um, all right.
- 20:15
Thank you so much. [upbeat music]