Layering every technique in RAG, one query at a time

What would convince you that a CRM agent is ready for customers? Choosing between BM25 and vector retrieval cannot answer that question. First, you need to know what customers will ask and what successful answers look like. David Karam brings a search engineer’s approach to this problem: he and his co-founder worked on Google Search before starting Pi Labs, and their habit is to inspect individual queries—what worked, what failed, and why. That makes the product’s launch bar, rather than a debate about whether RAG is dead, the starting point for engineering.

The launch bar becomes concrete through application-specific query sets. Start with easy cases, then add medium and hard ones as the product’s ambitions grow. Techniques are interventions intended to move performance on those cases toward the required quality. A benchmark disconnected from the application cannot tell you whether your CRM agent is ready; an evaluation built around its actual work can.

The quality engineering loop has three steps:

1. Establish a baseline. Run the simplest system that could satisfy the initial query set.
2. Analyze the losses. Inspect unsuccessful cases and identify what broke.
3. Choose an intervention. Add a technique that addresses the observed failure, then evaluate again.

Asking whether you need BM25 or vector retrieval before examining the query stream reverses this order. You may build a component that solves none of your current problems.

Karam calls the decision criterion complexity-adjusted impact: weigh the likely improvement against the effort required. In a catalog of techniques, difficulty and impact are the most useful columns. BM25 is relatively easy to try; custom embeddings can demand substantial work. Harvey’s custom legal embeddings illustrate the latter kind of investment—a difficult domain can justify customization when generic retrieval does not meet the quality bar. The point is to earn that complexity through evidence.

The simplest baseline may not need a separate retrieval system. Karam calls this in-memory retrieval: put all the documents into the LLM’s context and ask the question. He uses NotebookLM and a five-document question-answering scenario to make the interaction tangible. That is an illustration of a document-grounded product, not evidence that NotebookLM itself operates without retrieval. For your own small collection, however, supplying the whole collection is a straightforward place to start.

This baseline gives you specific failure modes to look for. The documents may no longer fit. Irrelevant material may crowd the context. Or the model may fail to attend to the documents that contain the answer. Those are different reasons to introduce selection, but each comes from an observed case. Once you can point to the documents or questions that break the baseline, retrieval has a concrete job: supply the useful evidence without carrying everything else along.

BM25 offers a simple first retrieval layer. Its scoring considers query terms, their frequency in a document, document length, and how rare the terms are across the collection. It works well when the query and useful documents share discriminative words. When the user expresses the same need with different language, term matching can miss the connection.

Relevance embeddings address that gap by representing queries and documents in vector space. Distance or similarity between those representations can capture relationships beyond shared words. But vector retrieval has its own weaknesses, including cases where exact keyword matching matters. The choice depends on the language and failure patterns in your query set.

Karam uses examples generated with ChatGPT to contrast the two query styles:

The second question describes battery life without using that phrase. If your failed queries mostly look like that, Karam strongly recommends investing in vector retrieval. If your query stream consists of precise keyword phrases, lexical retrieval may already do the necessary work. These examples help classify failures; your evaluation determines whether another retrieval layer improves them.

After adding lexical and vector retrieval, you may have useful candidates but conflicting orderings. A retrieve-and-rerank pipeline separates finding candidates from judging them more carefully. With a bi-encoder, the query and document receive independent embeddings, which are then compared. A cross-encoder takes the query and document together and produces a score while attending to both inputs. That joint processing can make finer relevance judgments.

Joint scoring is more expensive, so at corpus scale the usual pattern is to retrieve a broad candidate set and rerank a smaller subset. The restriction is about scale, not a prohibition: a small paragraph collection can be scored directly. Once the collection grows, spending the expensive model only on plausible candidates makes the pipeline practical.

Better relevance scoring still leaves a deeper problem: relevance is a proxy for the application’s information need. A document can be semantically close to a question without being the right document to use. Karam draws on Google Search’s long history of using signals beyond relevance, then turns to the legal retrieval example discussed by Harvey and LanceDB. In law, ordinary-looking words can carry specialized meanings that generic matching does not reliably capture.

Words such as regime and material have domain-specific significance; the required evidence may be a particular law or regulation. Custom embeddings adapt retrieval to those meanings. That does not require starting a model from scratch: Harvey’s work with Voyage adapted an existing legal embedding model. The engineering goal is to represent the distinctions that matter in the domain well enough to retrieve the right evidence.

Karam also asks ChatGPT for illustrative legal queries that could expose generic retrieval failures, including vocabulary such as moot and material. Such examples suggest what to investigate, but the decision to customize comes from your own query sets. Are failed retrievals consistently associated with specialized vocabulary that the generic model handles poorly? If so, domain adaptation has a specific target. If not, it may be an expensive answer to the wrong problem. Even successful custom embeddings primarily improve retrieval and recall; ordering those retrieved items remains another task.

Shopping makes the difference between relevance and usefulness especially clear. Karam reports asking Perplexity for cheap gifts for his son and receiving suggestions around $10. That interpretation of cheap was reasonable enough; he then clarified the intended price range. Karam reports that after specifying a budget of $50 or more, Perplexity still suggested items priced at $15 and $40. The failure in this individual interaction is concrete: both prices violate the stated lower bound.

The missing information is a price signal. Understanding the words is only the beginning: the price condition must survive query interpretation and affect which candidates are returned or ranked. A relevant gift below the lower bound is still an unsuitable result. Once the request has been interpreted as minimumPrice: 50, the constraint can be represented explicitly:

type Gift = {
  id: string;
  price: number;
};

const request = { minimumPrice: 50 };
const suggestions: Gift[] = [
  { id: "gift-a", price: 15 },
  { id: "gift-b", price: 40 },
];

const eligible = suggestions.filter(
  gift => gift.price >= request.minimumPrice
);

console.log(eligible); // []

This small implementation of the constraint rejects both reported suggestions. An empty eligible set is a reason to retrieve additional candidates, rather than quietly drop the user’s requirement.

Other signals depend on the corpus and the product:

• Merchant information: Shopping results can depend on properties of the seller, not just the product description.
• Audience behavior: Podcast listen counts can help distinguish otherwise relevant candidates.
• Link structure: PageRank captures prominence through links pointing to a page. It reflects the structure of the web, rather than simply matching the page’s text to a query.

Karam distinguishes broad natural-language meaning from the vertical semantics of a particular domain. Shopping, CRM, and email each contain relationships and constraints that semantic relevance alone does not express. As the product’s quality bar rises, those domain signals become a larger part of ranking.

Even a domain-aware ranking system can misunderstand its users. You may account for relevance, prices, and merchants, then discover that people click items you expected them to ignore and ignore items you expected them to choose. That is another loss pattern, requiring another source of evidence. Clicks, thumbs-up, and thumbs-down provide signals about preferences that the initial domain model did not capture.

Karam sketches a ranking function that combines a click-through prediction signal with relevance and structured domain signals. The final score balances natural-language relevance, price-related query understanding, and user preference. This adds a prediction problem: observed feedback must become a signal that can be applied to future candidates. The talk stops at the architecture rather than specifying how to train or weight that predictor. The practical distinction is that knowing the shopping domain and knowing what users prefer are separate capabilities.

The next failure can happen before ranking begins: the agent asks the wrong query. A search engine may be tuned for keyword phrases or another specific query form, while an LLM sends a broad request expressed in the language of the overall application. This mismatch is a tool-boundary problem. Improving the ranker will not necessarily repair a request the backend cannot interpret as intended.

Query fan-out gives the application more control. Decompose a complex request into narrower searches, retrieve their results, and use those results together. Karam points to Google AI Mode as an example. Karam describes seeing AI Mode’s brief animation indicate fifteen or twenty queries. Those counts are his observation, not a fixed fan-out setting; the mechanism is decomposition across subtopics and information sources.

An LLM may already attempt decomposition, but it does not automatically know which searches work well against your backend. Karam is enthusiastic about MCP while skeptical that prompting alone can communicate all of a search engine’s behavior. There are two possible directions: make the backend understand more complex requests, or give the agent enough knowledge to tailor its requests. Until that boundary works reliably, explicit orchestration provides control over query quality.

Consider What is David working on? The application may need to turn that into searches for David’s Jira tickets and Slack threads. The useful decomposition depends on knowing where work is recorded, not merely understanding the sentence.

Sending the original question unchanged leaves that domain interpretation to the search engine. If the engine lacks it, the request can fail despite a perfectly understandable user question.

Correctly formed queries can still go to too few sources. Karam calls the remedy supplementary retrieval: search additional backends when the current set cannot provide enough recall. He sees teams minimize search calls too early, before they have established which information sources the task needs. Broadening retrieval gives the system more evidence to work with.

His example is the query falafel. In Karam’s telling, this apparently simple query stumped a Google Search organization of six thousand people. The difficulty was ambiguous intent. A person might want somewhere to eat falafel, making restaurants useful, or might want to see what it looks like, making images useful. The same short query supports multiple result types.

Karam describes Google’s response as querying all the relevant backends and combining the results. This is different from splitting one complex question into subquestions: here, a short query needs broader source coverage because its meaning is underdetermined. His recommendation is to avoid being stingy with retrieval while recall is the problem. Actual cost pressure is the reason to reconsider that breadth.

Eventually, broad retrieval and query fan-out can become expensive. Karam introduces this stage with a server-rack image that he describes as a real advertisement involving a rack thrown from a roof. The joke gives way to an operational problem: more backends, more queries, and more model work increase the complexity and cost of serving each request. At that point, distillation becomes worth considering.

The objective is to reduce model size while holding the application’s quality bar constant. A general-purpose LLM can be overqualified for a narrow task: the product may only need one capability, such as question answering, rather than the full range of the model’s abilities. Distillation and task specialization require fine-tuning expertise, so this is a substantially harder intervention than trying another retrieval method.

Karam attributes Perplexity’s speed in some contexts to a model specialized for question answering, though he supplies no measured distillation result. His hypothetical retention example contrasts ten-second responses that cause users to leave with two-second responses that keep them. The reason to invest is the product consequence of latency: once response time threatens adoption, preserving quality with a smaller, faster model can justify the training work.

What happens when you have applied the techniques and cases still fail? Karam jokes that engineers should blame the product manager, then gives the joke a serious meaning. Stochastic systems will retain failures. Product design must decide how much responsibility the system should take and what happens when it cannot reliably fulfill that responsibility. Engineering alone cannot make every product promise safe.

Customer support makes this responsibility explicit. Some cases can be handled by the bot; others need escalation to a human. The handoff is part of the user experience, not simply a missing model capability. The product must define when the machine can act on its own and when a person needs to participate.

Google Shopping provides a more visible spectrum. Where Google has strong structured understanding, it can offer a high-promise UI with product choices, reviews, and filters. Where it understands the material mainly as web documents, it offers a simpler experience that leaves more interpretation to the user.

Karam contrasts offering ten possibilities with presenting one result when the system understands the need well enough. The interface should gracefully move up or down that spectrum. A less assertive experience can remain useful while avoiding a promise the underlying system cannot support.

That returns the engineering decision to the query set. Context windows versus RAG, or autonomous agents versus workflows, become empirical questions about what improves this application. Baseline the system, inspect its losses, and look for an easy intervention. Move to medium-complexity work when needed; hire for genuinely difficult work when the remaining failures justify it. The next layer should follow a demonstrated need, because sophistication added too far ahead of the product can consume substantial effort without moving its launch bar any closer.