Information Retrieval from the Ground Up

Does retrieval-augmented generation require vector search? No: retrieval supplies the right context to generation, and vector search is one way to find that context. In Philipp Krenn’s workshop, more hands go up for experience with vector search than with keyword search—a reversal that reminds him of early NoSQL adoption, when developers sometimes encountered MongoDB before SQL. Vector search is a feature of retrieval, alongside lexical matching, filters and ranking. The demonstrations use Elasticsearch, but the underlying ideas also apply to Lucene- and Tantivy-based systems.

The recurring example is a familiar sentence: “These are not the droids you're looking for.” What should a search engine retain from it? Should Droid match droids? Should robot find the sentence even though that word never appears? Those questions lead from ordinary word matching to learned representations and, eventually, combinations of both.

The workshop provides a code gist and a shared cloud instance, so attendees can run the examples without individual accounts. The essential precaution is to replace Krenn’s handle in index names: otherwise everyone writes to the same documents. The recording’s login instructions use “Login with Elasticsearch” and the temporary workshop/workshop credentials; these describe the workshop setup, not a current service.

Before indexing anything, Krenn calls _analyze. This endpoint previews the tokens an analyzer produces; it does not store a document. Tokenization separates text into searchable units. Whitespace and punctuation are useful starting points for the Western-language examples here, although languages without those word boundaries need different segmentation.

A token carries more than its text:

Offsets support highlighting. Positions let a phrase query check whether one token follows another at positions N and N + 1, without reconstructing the sentence. Search indexing moves this preparation into ingestion: when the relevant metadata is stored, later queries can use it instead of repeatedly analyzing a long source field. The analyzer output also exposes token types; synonym processing is another configurable part of analysis.

The custom pipeline strips HTML, tokenizes the remaining text, lowercases it, removes stop words and applies Snowball stemming. The following request body for POST /_analyze expresses that sequence:

{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "filter": ["lowercase", "stop", "snowball"],
  "text": "These are not the <em>droids</em> you're looking for."
}

The demonstrated analysis leaves droid, you and look: plural droids and inflected looking collapse to shared stems. Compatible analysis at query time makes those normalized terms match directly. Token positions remain zero-based.

These are language rules, not universal transformations. The German analyzer leaves droid, den, such; the French example leaves droid, la, rechercher. Applying English rules to German can remove endings for the wrong reason. A pipeline can run successfully and still produce unsuitable search terms because its language configuration is wrong.

Stop-word removal makes another trade-off. Krenn describes the default English list as short—he estimates roughly 33 entries—and emphasizes that it can be overridden. Removing not does not mean the engine understands negation; it simply means that token is excluded. The extreme example is “to be or not to be,” which disappears entirely under this analysis. Keeping representations both with and without stop words can preserve phrase-sensitive searches while avoiding common-word noise elsewhere.

A mapping attaches analysis to a stored field, much as a database schema defines field behavior. Krenn creates a named analyzer and applies it to quote. In the shared environment, index names must retain the user_ prefix while using a unique handle. This request body for PUT /user_reader_quotes captures the lexical setup:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "quote_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "quote": {
        "type": "text",
        "analyzer": "quote_analyzer"
      }
    }
  }
}

After creation, analyzing through the field itself checks the actual mapping rather than a separately reconstructed pipeline. Send this body to POST /user_reader_quotes/_analyze:

{
  "field": "quote",
  "text": "These are not the droids you're looking for."
}

The corpus contains three quotations:

1. These are not the droids you're looking for.
2. No, I am your father.
3. Obi-Wan never told you what happened to your father.

The second sentence exposes an easy mistake: guessing the stop list from intuition. After correcting the live analysis request, Krenn finds that only no is removed; i, am, your and father remain. The third sentence tests token boundaries: the standard tokenizer splits Obi-Wan into two tokens, whereas a whitespace tokenizer preserves the hyphenated form. URLs and email addresses provide another reason to choose a dedicated tokenizer instead of accepting ordinary word boundaries.

Now Droid retrieves the first document. The query becomes lowercase droid, and the stored plural was already stemmed to droid. This is an exact match between normalized tokens. A search body for POST /user_reader_quotes/_search is correspondingly small:

{
  "query": {
    "match": {
      "quote": "Droid"
    }
  }
}

The underlying inverted index maps terms to their occurrences. Krenn’s diagram arranges the extracted vocabulary alphabetically and associates each term with document IDs, occurrence counts and positions. The droid entry points to document 1, with its occurrence at position 4; documents 2 and 3 do not contain it. The query follows that posting rather than scanning every sentence. Positions support phrase checks, while offsets provide the original character span for highlighting.

A search for robot finds nothing in this corpus. A synonym mapping could connect it to droid, and an LLM could help generate that mapping, but the search engine would still be applying explicit term relationships. Homonyms expose the limitation: bat can mean an animal or sporting equipment, and a context-free string mapping cannot reliably choose between them.

Phrase matching adds positional constraints. I am your father matches the second quotation. I am not your father does not: although the analyzer removes not, it preserves the gap that token occupied. Removing a token does not renumber everything that follows it.

The slop parameter permits positional movement. In this example, I am father fails with the default slop: 0 but matches with slop: 1, accommodating the intervening your:

{
  "query": {
    "match_phrase": {
      "quote": {
        "query": "I am father",
        "slop": 1
      }
    }
  }
}

That tolerance does not make arbitrary words interchangeable. I am his father still fails because the required term his does not match the stored sentence.

Contractions need their own handling: the demonstrated analyzer does not make I'm your father equivalent to I am your father. Normalization rules, synonyms, dictionaries or learned representations can bridge that difference. Krenn describes dictionary availability and licensing as practical obstacles, rather than something basic stemming solves.

A stem is not necessarily a dictionary word. Both blackberry and blackberries become blackberri, which is useful because both inputs meet at the same token. But searching for black does not automatically find blackberry. Compound nouns, common in languages such as German and Korean, motivate dictionaries or substring techniques. Krenn also raises licensing as a concern for language analyzers, tentatively recalling restrictions on a Hebrew analyzer; linguistic resources and model weights both require checking their actual licenses.

An audience question about smaller tokens leads to character n-grams. In the quick fox demonstration, one- and two-character grams produce overlapping pieces: q, qu, u, ui, and so on. This increases ingestion work and disk usage, then increases query work because many more terms and documents can match. It can also return substring matches that surprise users.

Several representations can coexist. Keep a normal analyzed field, a stop-word-preserving alternative and an n-gram field, then weight the n-gram matches lower so strong whole-term matches win. The granularity is configurable:

The edge-n-gram demonstration emits only q and qu from quick. Prefix lengths such as two through five are tuning examples, not universal defaults. Prefix-only matching cannot retrieve blackberry from berry.

For multilingual data, the next question is where analysis and statistics live. Mixing English and Hebrew into one undifferentiated field mixes both language rules and term-frequency distributions. Separate language fields or separate indices allow each language to use its appropriate analyzer. Krenn describes language identification as a way to route incoming text, including short inputs, although its suitability still depends on the actual material being classified.

Fuzziness widens matching through edit distance: inserting, deleting or substituting characters can connect a misspelling to an indexed term. Elasticsearch’s documented fuzziness settings allow explicit distances of 0, 1 or 2. The corrected AUTO explanation is:

The important surprise is that this applies per token. A misspelled Obi-Wan can contain an error on both sides of the hyphen and still match, because standard tokenization creates two independently checked terms. Krenn illustrates the underlying Levenshtein automaton with food and two edits. Like n-grams, fuzziness is a powerful but potentially costly expansion mechanism.

Finding matching terms produces candidates; scoring orders them. Krenn introduces TF-IDF and then BM25, Elasticsearch’s standard lexical ranking approach, through three useful intuitions:

• Term frequency: more occurrences of the query term can increase relevance. BM25 applies diminishing returns; Krenn’s repeated-droid example illustrates saturation, not a fixed occurrence threshold.
• Inverse document frequency: a term found in fewer documents generally contributes more than a ubiquitous term.
• Field length: a match in a short field can carry more weight than the same occurrence in a long field.

These factors explain much of the behavior without requiring a reader to calculate scores manually.

For father, both matching quotations contain the term once, and the query term has the same document frequency in both calculations. The shorter No, I am your father therefore ranks above the longer Obi-Wan sentence. Adding explain: true exposes the score components:

{
  "explain": true,
  "query": {
    "match": {
      "quote": "father"
    }
  }
}

Use that output to investigate a specific ranking rather than routinely returning it with every search. Krenn then demonstrates random scoring before discussing more useful custom signals, such as a product’s margin or rating. Text relevance can be one component of a broader ranking function.

Multi-term queries introduce unequal term contributions. In Krenn’s two-axis illustration, father is more informative than your; a document containing both aligns with the intended query, while one containing only father is a better partial match than one containing only your. He follows this with a historical coordination-factor illustration: matching three of three terms yields a factor of 3/3, two yields 2/3, and one yields 1/3. The angular diagram and coordination factor explain classic vector-space and TF-IDF ideas; they should not be combined with the preceding BM25 discussion into a single current scoring formula.

The next experiment shows why search scores are not confidence percentages. These are my father's machines analyzes to my, father, machine. Searching for those same three terms looks like a perfect match. The demonstrated score is about 3.2, but that does not make 3.2 a reusable definition of 100%. Adding or deleting documents changes the corpus statistics used in scoring.

Delete that document and another result becomes first. Is it now 100%? Add These droids are my father's father's machines and the repeated father changes term frequency. Could it exceed the supposed perfect score? The contradiction comes from assigning probability-like meaning to a ranking value. Scores order results within a query; they are not directly comparable across different queries or changing corpus states.

To move beyond manually maintained synonyms, Krenn connects an OpenAI text-embedding model through Elasticsearch’s inference API. He removes the API key from the example; an independent setup needs its own credentials. Two inference services are configured: one for dense embeddings and one for sparse embeddings.

The dense-vector explanation begins with deliberately interpretable Star Wars axes: realistic versus cartoonish, then human versus machine. Leia falls toward realistic and human; a cartoon Darth Vader lies elsewhere. A character with a mechanical hand moves slightly toward the machine side. This illustrates proximity in a vector space, but actual learned dimensions do not have such clean human-readable labels. A document or chunk becomes an array of floating-point values, and retrieval searches for nearby representations. Whether proximity captures useful meaning depends on the model.

Krenn reports a 128-dimensional configuration for the dense demonstration. That is the demonstration’s output size, not an inherent size of all OpenAI embeddings. The displayed representation of the droids sentence is a floating-point array whose individual coordinates are difficult to interpret directly.

SPLADE takes a different approach: learned sparse representations over a vocabulary. Krenn introduces ELSER, Elastic’s learned sparse encoder, in that family of techniques. The model expands an input into weighted terms it considers relevant. These weights are learned, rather than ordinary TF-IDF counts or a hand-authored synonym list. The demonstrated ELSER model is English-only, and its sentence representation contains roughly a hundred or more weighted terms—not the entire vocabulary.

A query receives its own sparse representation. Shared terms contribute products of query and document weights; those contributions combine into the document’s score. The terms make the representation more inspectable than a dense array, but not necessarily intuitive. Large expansions also create many OR-style matches: in a large corpus, the query may touch and score a substantial set of documents. Interpretability and query cost are separate properties.

A new index retains the lexical configuration and adds semantic_text fields for the sparse and dense models. Krenn copies the three documents into it, with quote feeding both semantic representations. Ordinary document retrieval shows the source text without returning the derived sparse weights or dense arrays; those structures support searching in the background without enlarging every result payload.

Searching the original lexical field for machine still returns nothing. Searching through ELSER retrieves the droids quotation, with a demonstrated score around 0.9, and also a weak match for No, I am your father. The Obi-Wan sentence does not appear. A learned expansion has bridged machine to the droids sentence, but it has also introduced a less obvious connection to the father sentence.

A minimum score can suppress weak results, but a fixed threshold does not automatically transfer across queries. Krenn’s example moves from a possible cutoff of 0.2 to a weak result scoring about 0.4 after the query changes. The threshold needs to reflect the query and dataset. He expects such remote matches to be less visible in the top results of a large corpus containing better candidates; the three-document corpus makes every marginal connection conspicuous.

The missing third result has a simpler explanation: its expanded terms have no overlap with the query’s expansion. For the surprising father result, Krenn inspects explain: true and finds puppet as an apparent overlapping term. That identifies the mechanical reason for the match, even if the learned association is not persuasive to a human reader. Explaining how a score was produced and deciding whether a result is useful are different tasks.

The dense search for machine returns all three quotations, with the droids sentence first and the scores close together. Unlike sparse term overlap, a distance or similarity calculation can place every vector relative to the query. Without an appropriate cutoff or other restrictions, nearest-neighbor retrieval can return candidates that look unrelated to the user.

Krenn explicitly rejects treating this tiny, 128-dimensional example as a model benchmark. Its purpose is to expose behavior. Lexical retrieval demands careful choices about language, tokenization and stemming; dense retrieval shifts much of that responsibility toward selecting and evaluating a model against the actual documents and queries. Returning something can solve an empty-results problem while creating a relevance problem.

How much that matters depends on the domain. Krenn recounts an unnamed European retailer’s report that it accidentally deleted roughly two-thirds of its product data and lost almost no revenue. The anecdote motivates offering plausible alternatives quickly in commerce; it does not supply a measured rule for other systems. Legal-case retrieval has a very different tolerance for remotely related material. Search quality depends on the corpus, user requests and consequences of a bad result. Tuning one query can worsen others, which makes a representative evaluation set essential.

An audience member asks how to enable semantic enrichment for only some customers in a shared index. Separate indices and mappings are one option, but they complicate upgrades between customer tiers. A colleague offers a simpler alternative: leave the semantic field unpopulated for ineligible documents. An application condition or ingest processor can copy text into the vector-indexed field only when the document qualifies. Documents without a vector do not become candidates in that vector index. The colleague describes absent-field overhead as negligible, without presenting measurements.

Another attendee reports CPU-heavy HNSW writes and advice to freeze writes after building an index. The response focuses on the interaction between HNSW and Elasticsearch’s immutable segments. Merge-scheduler settings inherited from lexical workloads may be unsuitable for frequent vector updates. Tuning should reflect actual update patterns and available IOPS, rather than assuming a static index is the only workable arrangement.

The underlying difficulty is that merging graph-based vector indices is not as simple as concatenating postings. Segment merges can require rebuilding HNSW structures. One optimization Krenn describes reuses the largest suitable segment and adds the remaining documents to it, instead of rebuilding everything from scratch. Both speakers stress version sensitivity, specifically urging users of Elasticsearch 8.11 to upgrade and mentioning further work around 9.0; their recollections do not pin each optimization to an exact release. The attendee then says they moved to Pinecone and are now curious about these changes.

A request such as planning a fun weekend does not necessarily resemble the documents needed to answer it. Generation can therefore happen before retrieval. Krenn’s recipe example starts with someone who remembers an Italian dish containing layers of pasta and meat but has forgotten its name. An LLM identifies lasagna, then the search engine retrieves an existing recipe. There is no need to regenerate the recipe itself.

HyDE addresses a related mismatch by generating a hypothetical document from the query, encoding that document-shaped text, and using it to retrieve real corpus documents. More involved requests can require multiple searches: shopping for a 1920s theme party first requires identifying appropriate items, then retrieving them. An audience member also suggests conditioning query and document embeddings with instructions describing the task. These approaches change the retrieval input, rather than expecting every raw user request to already be a good search query.

Embedding dimensions are another evaluation choice. A 512-dimensional model is not necessarily better for a task than a 128-dimensional one. More dimensions may represent useful distinctions, but also require more storage and computation; fewer can be sufficient. Dimension reduction and reducing the precision of each coordinate are separate resource levers. The decisive question is whether the resulting representation improves retrieval for the use case.

To assess that, start with representative queries and relevance judgments rather than attempting to interpret similarity scores as percentages. Krenn describes several sources of evidence:

• Human judgments: experts rate results for a golden query-and-document dataset, allowing changes to be compared against a stable reference.
• LLM-assisted judgments: models help assess results, potentially with human review.
• Behavioral signals: early-result clicks followed by sustained visits suggest usefulness; abandonment, immediate returns and deep pagination can indicate trouble.

Behavior is a proxy, not a direct reading of intent, but it can provide practical feedback when a fully judged dataset is unavailable.

After more semantic searches against the father and droids examples, Krenn switches to a much longer passage with multiple speakers. A search concerning murder in the Skywalker saga retrieves the document and highlights a matching fragment. The fragment contains kill, rather than the literal query word murder: semantic retrieval has located a related passage, and highlighting makes that location visible.

Chunking controls how much material one representation must summarize. Pages, paragraphs and sentences are possible boundaries, with or without overlap. A book-sized field can contain too many different subjects for one vector to represent usefully; smaller units give retrieval a chance to isolate the relevant passage and link back to its source. In this demonstration, chunk creation and inference happen behind the semantic field interface. Krenn does not manually submit individual chunk vectors, even though those derived units are doing important work.

Retrievers provide a composition mechanism for the pieces developed so far. Lexical matching can combine with rank features such as ratings, purchase history or margin. Boolean filters have a different role: they admit or exclude documents without contributing relevance scores. Learned sparse and dense retrieval add model-based evidence. In Krenn’s broad usage, hybrid search combines multiple retrieval approaches—lexical and dense, sparse and dense, or potentially multiple dense searches.

Reciprocal rank fusion, or RRF, combines positions rather than raw scores. A document might rank fourth in lexical retrieval and second in dense retrieval; both positions contribute to its fused ranking. Filters remain hard constraints, such as a user ID, a geographic radius or a release-date range. Other signals—book ratings, stock availability, margin and click history—can affect ordering rather than eligibility.

Raw scores need care because retrieval methods use different scales. Lexical scores also change with query length and boosting. Normalizing each result list before weighted combination is one approach; it rescales values for blending without turning them into confidence percentages. RRF avoids that calibration by summing reciprocal-rank contributions:

Here Lᵢ is a ranked result list and rᵢ(d) is the document’s one-based rank in it. A document absent from a list contributes nothing from that list. The constant is configurable; the mechanism does not average positions or compare the original lexical and vector scores.

If the primary data already lives in PostgreSQL, why send it through change-data capture into a separate search system? Krenn acknowledges the convenience of pgvector when PostgreSQL meets the workload’s scale and feature requirements. The question is whether vector search alone covers the product’s needs. Exact brand names and known terms often favor lexical matching: users expect a word they know exists in the corpus to be retrievable. Short queries also provide little semantic context. Krenn has seen systems route one- or two-word queries to lexical search and longer requests to vector search, although the right policy depends on the application.

Repository history introduces a different cost problem. An attendee describes food.py remaining unchanged through 4,999 commits and changing on the 5,000th. Storing another identical embedding for every commit wastes space and inference work. The discussion distinguishes file-level and chunk-level representations, then proposes content-addressed storage: hash the content, use that hash as the document ID, and use a create-only operation rather than an update or upsert.

Create-only writes reject duplicate IDs, but a rejected duplicate still requires transmission to the server. Krenn also suggests keeping an external cache of previously seen hashes; the attendee reports using chunk-level deduplication. This is a proposed design discussion, and duplicate rejection alone does not establish whether an ingest pipeline has already performed inference. If avoiding embedding work is the objective, deduplication must happen before that work is triggered.

The audience then qualifies the PostgreSQL comparison. PostgreSQL has native full-text search, with tsvector as a built-in type, and extensions can add other ranking implementations. Its built-in ts_rank and ts_rank_cd functions are distinct from BM25 and do not use global corpus information. That is the scope of the limitation discussed here—not a claim that the PostgreSQL ecosystem cannot provide BM25. The audience mentions a Rust-based extension without establishing its name.

Retrieval and final scoring need not use the same method. Krenn’s illustrative funnel starts with a million documents, uses a cheaper method to select a thousand candidates, and runs a more expensive scorer only on that subset. The final scorer can examine candidates more deeply without paying that cost across the whole corpus. He shows a built-in reranking inference task and the version-one Elastic reranker.

The text_similarity_reranker retriever wraps candidate retrieval. Although the live discussion tentatively dates that API to 8.16 or 8.17, the archived 8.15 documentation already includes it; that does not date the Elastic model itself. A compatible cross-encoder or reranking service evaluates the query against candidate text. The described implementation targets a parent-level text field, which fits one document per chunk. Highlighted fragments or sub-document chunks can instead be sent through a separate inference call.

Composition runs from the inside out:

1. Retrieve lexical and nearest-neighbor candidates as peer branches.
2. Fuse their result lists with RRF.
3. Pass the selected documents’ text to the reranker.

Vector retrieval is optional in that procedure. A reranker can also improve the ordering of candidates obtained entirely through lexical retrieval, without embedding the whole corpus. The colleague reports roughly a 10-point evaluation gain on unspecified customer datasets, but provides no named metric or evaluation conditions. His following Gemini-versus-Amazon explanation is likewise an unverified anecdote, rather than evidence of either service’s internal implementation. The concrete capability is the independent reranking stage; in the described setup, reranking sub-document hits still happens outside the retrievers API.

Krenn next demonstrates a different combination: min-max normalization with lexical weight 2 and dense-embedding weight 1.5. The droid/robot query ranks the droids quotation highest. These weights express how the two normalized signals contribute; they do not make either signal a probability.

The exposed reranking controls include the candidate window and a minimum-score cutoff. A larger window can admit a relevant document that cheaper retrieval placed lower, but costs more to rescore. An attendee asks for more specific execution control—cheap reranking on distributed query nodes, followed by one expensive global pass. Krenn does not describe that placement control; his answer stays with the window and cutoff available to the caller.

The closing demonstration offers a shell-like query language as an alternative to long JSON requests. Its match operator can target lexical, dense or sparse fields and return the quotation, speaker and score. Krenn describes joins, observability aggregations and full-text searching, while noting that the language does not yet expose every search feature. A live attempt to simplify the displayed columns breaks the query. He also identifies weaker client-side typing, including in Java: callers submit strings and parse the returned results instead of getting the same typed structure as the other client interfaces.

The final question concerns a working hybrid pipeline: separate Elasticsearch keyword and vector queries, followed by Cohere reranking. Would placing those steps inside one retriever request improve the result? The immediate benefit is simpler orchestration—one endpoint, one round trip and one returned result set. The lexical branch, vector branch and reranker still have to do their work. Consolidating them does not by itself establish better relevance or remove their computational costs.

The colleague adds an operational reason to test consolidation: separate requests can duplicate distributed coordination and thread-pool work and may deny the optimizer opportunities to stop early. These are proposed sources of savings, not a measured latency or throughput comparison. Separate searches remain useful for profiling each branch, and the practical benefit depends on where resource pressure lies. The final architectural choice is therefore concrete: keep the retrieval stages explicit enough to inspect, while letting one composed request manage them when it reduces application and coordination work.

For attendees continuing the exercises, Krenn closes by offering to leave the shared instance running for approximately the rest of that day. That was the recording-time practice window, not a promise of continuing access.