Vector Isn't Enough: Hybrid Search & Retrieval — Jeff Vestal & James Williams, Elastic

An agent can give a convincing answer from bad evidence. That is the practical problem behind Jeff Vestal and James Williams’s hands-on Elasticsearch workshop. Agents have taken over more of the work of retrieving and analyzing information, but they still need access to changing facts, proprietary documents and prices that their models cannot reliably supply from training. A more capable model does not remove that dependency.

The workshop follows that dependency downward: first semantic retrieval, then lexical retrieval with BM25, then ways to combine them. The resulting search feeds an agent, making it possible to examine how retrieval affects the final answer. Reranking appears last, as an optional second stage rather than an automatic requirement.

Each lab starts with API commands in Dev Tools and continues in a Python notebook against a small dataset in a provisioned Elasticsearch Serverless instance. The useful exercise is to inspect why documents appear in a particular order and what changes that order. Vestal’s warning about clicking “Run All” captures the difference: finishing the notebook quickly is easy; understanding the ranking takes longer.

One architectural choice runs through the labs: embedding inference happens through the data platform. Elastic Inference Service supplies the workshop’s Jina models, covering dense embeddings and reranking. This moves some coordination that an application would otherwise perform into Elasticsearch itself.

Elasticsearch’s semantic_text field is the workshop’s entry point for turning text into searchable chunks. Williams describes sending five pages of text and having the field store chunks for retrieval, with a stated workshop configuration of 250-token chunks and one-sentence overlap. Sentence boundaries guide the default splitting, and the chunking settings can be changed. A model’s large context window does not oblige the application to fill it.

The discussion distinguishes that ordinary chunking path from late chunking. The Jina models support late chunking over large contexts, but Williams says it is not yet exposed through Elastic Inference Service in the version being demonstrated. His recommendation for these labs is conventional chunking plus hybrid retrieval. That is a practical choice for this workflow, rather than a demonstration that every chunking method produces equivalent results.

The integration is optional. Applications can bring their own chunks and vectors through the supported SDKs, taking responsibility for more of the preparation. The managed path instead handles chunking and inference behind the search API. Elastic’s stated thesis is to combine lexical and semantic retrieval in the same system, with reranking available when the first-stage results need further refinement.

Semantic retrieval depends on a shared representation: documents are embedded during ingestion, and queries must use the same embedding model. Otherwise the query vector and stored document vectors do not have the intended relationship. The semantic_text field carries the inference configuration so that every developer ingesting or querying that field does not have to remember and repeat the model choice.

What connects a text query to the returned documents? In the demonstrated path, the configured inference service produces the embedding, Elasticsearch uses approximate nearest-neighbor search, and the search returns the requested top K documents or chunks. Vestal identifies HNSW—hierarchical navigable small worlds—as the default graph-based search mechanism in this discussion. The field configuration ties the stages together; it does not make the nearest neighbors automatically relevant.

The useful case is a roomful of people asking how to configure TLS security on Elasticsearch. Their wording differs, but their intent overlaps, so semantic retrieval can lead them toward the same documentation. An opaque product ID creates the opposite problem: the precise character sequence matters, while its apparent meaning contributes little. A semantically similar document can therefore be the wrong result. BM25 and keyword filtering provide another way to recover the identifier that the vector search misses.

Two questions clarify how to read the lab output. For paging through a hundred results ten at a time, the presenters point to Elasticsearch’s point-in-time facility and continuation information, although they do not specify the full request sequence. Separately, the displayed summary is an LLM-generated convenience field created to keep long document bodies from overwhelming the notebook. In this early example, the vector search operates on the title. The readable summary should not be mistaken for the text being embedded and searched.

Quantization changes the cost of keeping vectors available for search. Williams describes converting float32 vectors to int8 representations for memory while retaining the full vectors separately. BBQ compresses the searchable representation further, toward a bit-level representation; Disk BBQ provides a disk-oriented alternative that does not use HNSW as its underlying search mechanism. These are presented as ways to reduce memory pressure while preserving useful recall. The discussion supplies no measured memory-versus-recall comparison, so it supports the architectural distinction rather than a quantified guarantee.

The next audience question concerns graph retrieval. HNSW’s graph is a mechanism for navigating vectors; the presenters explicitly distinguish Elasticsearch from a graph database used to represent domain relationships. Their concern is the maintenance and scaling cost of adding that second kind of graph, especially for their target of multi-billion-vector datasets. Their skepticism about competing graph systems is a judgment from their experience, not a comparative benchmark established here.

Several alternatives can help an agent follow relationships without requiring a maintained domain graph:

• Search, inspect, filter. Retrieve an initial sample, notice a shared tag, then use that tag in another search.
• Classification. Run a suitable model on Elastic ML nodes or through an external inference service to supply categories that help guide retrieval.
• Named entity recognition. Extract people, locations, businesses or products from the incoming question, then let the agent use those entities to construct a more specific query.

These methods can solve some relationship-finding tasks; Williams explicitly says classification does not cover every graph use case.

An audience question identifies a configurable control in reciprocal rank fusion, or RRF: the rank constant used in its denominator. The presenters confirm that it can be changed, but do not establish its default or supported range. The more consequential question follows: should a small corpus use a reranker at all? Williams’s rule is to try BM25 plus semantic search and stop if it meets the required precision or recall. Add another model only after testing whether its improvement earns the latency.

There is room for a simpler starting point. Some teams begin with BM25 alone and add semantic search when relevance falls short. Williams prefers including dense vectors early because they can avoid elaborate synonym lists and other lexical tuning needed to handle varied language. BM25 remains useful for specific words and phrases; it struggles when the user’s phrasing moves away from the terms in the document.

Latency must include query embedding. Lexical search usually avoids that inference call, whereas semantic search can spend more time at the inference endpoint than in the vector database. A fast nearest-neighbor lookup cannot rescue a slow embedding service. Bringing inference into the platform is Elastic’s answer to coordinating that complete path, but the design decision remains broader: inference and orchestration must live somewhere, either in the data platform or in application services around it.

Choosing weights automatically is a separate, unfinished problem. Williams describes a desired system that learns from searches, clicks, thumbs-up or thumbs-down feedback and business metrics, then changes weights, fusion strategies or filters. He explicitly says this automatic relevance-tuning capability has not been released. A narrower technique is already possible: an LLM examines a query and chooses filters before executing searches. Query-dependent decisions should not be confused with a system that learns its relevance settings from usage.

The immediate obstacle to hybrid search is numerical. In Vestal’s example, BM25 scores occupy an open-ended range while vector scores typically fall between zero and one. Adding those raw values can allow the lexical scale to dominate regardless of which result is more useful. The workshop introduces two ways to make the signals work together.

RRF uses positions rather than the original scores. The two searches run in parallel, then fusion combines their result orders using a rank constant. A document that appears near the top of both lists receives support from both retrieval methods. What changes between the input lists and the output list? The workshop’s ranking table is intended to make that movement inspectable: agreement across methods helps determine the fused order, without pretending their raw scores share a scale.

Filtering changes the candidate pool rather than merely rearranging it. A keyword, location or date restriction can exclude documents that should never compete for the answer. Vestal uses dinner as the example: searching broadly for food and then restricting to pizza gives a more relevant pool when pizza is what you actually want. The restriction does useful work only when it captures the intended scope of the question.

Linear combination preserves score information after normalization. The lab discusses min-max normalization to bring scores into a shared zero-to-one range, then assigns weights to the retrieval methods. Those weights offer more control than a default fusion strategy, but they create an evaluation task: which balance works for the queries and documents that matter?

A judgment set makes that task concrete. For each query, specify the documents that should be returned, using human judgments and possibly LLM assistance. Run experiments with different weights and compare the results against those desired documents. Without that target, changing weights can produce a different ranking without establishing that it is a better one.

A mixed query workload may need several strategies. Search templates can hold an RRF search, a weighted linear search and a search with reranking. A routing layer—an LLM or another classifier—examines the incoming query and selects a template. This is Williams’s “poor man’s” auto-tuning: choosing among prepared strategies, with less repeated application code. For an unfamiliar query mix, RRF is the suggested starting point. Neither presenter makes hybrid retrieval mandatory when semantic search or BM25 already suffices.

How does fusion differ from routing? Fusion combines evidence from parallel searches; routing chooses which prepared search strategy to execute. The diagram separates those relationships so that BM25 and semantic retrieval do not appear to be successive stages of the same search.

Lab four puts retrieval into an agent. Vestal returns to the failure from the opening: give a model misleading external information confidently enough and it may build its answer around that information. His deliberately blunt example is telling it that the sky is orange. The purpose is to expose the dependency between supplied context and generated answers, rather than assume the model will repair poor retrieval.

Agent Builder connects models to tools and skills that operate on Elastic data. Workflows extend this to ordered tasks and decision branches that can call APIs, MCP tools and other systems. Vestal gives an example outside the lab: observability data indicates resource exhaustion, an agent decides scaling is needed, and a workflow scales Kubernetes. That is an illustrative action path, distinct from the documentation-retrieval demonstration.

The interface is also a development aid. Agent Builder’s Kibana operations are available through APIs, allowing a prototype’s endpoints to be reused in a custom application interface. Teams can also replace components with another framework. The intended benefit is to keep the work invested in tools and retrieval while changing how the application presents or orchestrates them.

Retrieval supplies a second benefit beyond fresh knowledge: inspectable sources. A model generating a plausible continuation is not looking up a database row containing the provenance of its training. Supplying documents gives it material it can summarize and cite. Vestal uses fabricated legal cases as the warning: convincing language can conceal a missing source. Citations make checking possible; they still require the reader to verify that the cited material supports the answer.

The demonstration separates several mechanisms that can otherwise blur together:

• Skills explain how to work. They describe a task, when tools should be used and the desired outcome. The workshop agent has a short diagnostic skill covering the cause, the fix and error codes.
• Tools perform actions. A search tool retrieves information; other tools can list or invoke workflows or reach external services.
• Workflows organize execution. They run ordered tasks and can include decisions and calls to other systems.

The lab builds its skills, tools and agent programmatically, allowing those pieces to be created, changed and removed through the same platform APIs.

ES|QL supplies SQL-like syntax with piped commands: a stage runs, then passes its output to the next stage. An ES|QL tool can execute a specific query and return selected fields. Index search serves a different purpose by giving the agent context about what an index contains, its mappings and how its fields should be used.

Consider a product index alongside pricing and inventory information. An uninformed agent may first enumerate indices, inspect mappings, guess which index represents products, and then explore how to combine them. Each round trip spends tokens and time. Supplying the index purpose, field descriptions and combination guidance points it toward the useful query sooner. This is a concrete way to reduce discovery work without expecting the model to infer the data organization repeatedly.

MCP connects these tools beyond the built-in interface. Agent Builder can call an external MCP server, and Kibana also exposes built-in and custom Agent Builder tools through its own MCP server. An external agent can therefore use the same tool definitions rather than requiring every search integration to be rebuilt separately.

The example question has two jobs: explain why a cluster keeps dying with exit code 137, and identify settings that could prevent it. The agent first loads the diagnostic skill, then makes repeated retrieval calls. In this demonstration, it attributes the failure to an out-of-memory kill and retrieves advice about JVM options. The transcript does not provide the specific settings or a tested remediation, so the result is a sourced diagnosis and prevention answer for the example—not a universal diagnosis of every exit code 137.

The tool trace also helps developers discover useful retrieval logic. Williams recommends treating the agent as a prototyping and debugging interface: inspect the searches that retrieved the required data, then use that knowledge when building the application. He highlights parameterization across ES|QL, Agent Builder and MCP as a convenient path for agent search tools.

The demonstrated ES|QL query forks into two search threads and uses fusion to combine them. Pipes then carry each stage’s output forward. This makes both the branching retrieval and the subsequent processing visible in a compact query. The important distinction is that two retrieval branches can contribute to the answer; one need not be treated as a fallback that runs only after the other fails.

Williams also describes a future direction: federated retrieval, including queries over external Parquet data joined with Elastic data. He identifies that work as in development. The demonstrated fork-and-fuse query should therefore be kept separate from the broader federation capabilities he expects ES|QL to expose later.

The answer is only one view of the retrieval. Agent Builder also exposes the query activity and citations. In this example, the search is limited to five results, and the interface reports that it found five. That count connects the answer to a specific retrieved set rather than an unspecified body of documentation.

Where does a citation actually lead? Opening the results in Kibana Discover reveals the query and the five returned documents, whose bodies can be expanded for inspection. The document itself includes its source URL. That stored URL supplies the link for further reading, avoiding the need for the model to guess a destination. The useful relationship is answer → retrieved document → stored source link.

The bonus lab introduces a second retrieval phase. First-stage lexical, semantic or hybrid search finds a candidate set. A reranker then evaluates the query together with the candidate documents. Ordinary vector retrieval compares a query embedding with document embeddings prepared earlier; this reranking step performs fresh model inference on the query-document inputs. Its added cost is why the first stage narrows the corpus before reranking begins.

The decision depends on how much the final order matters. An application evaluating only the first, second and third chunks has little room for a relevant document to land too low. Passing a larger set directly to the LLM creates another risk: several passages may look like plausible answers, leaving the generator to choose among them. Reranking can help place the most relevant evidence where it will be used. If the first stage already supplies sufficiently good evidence, the extra inference may not earn its latency.

Two scoring arrangements address different candidate sets:

• Pointwise reranking scores each query-document pair independently. No other candidate is part of that pair’s evaluation. Vestal suggests this often works well when the candidates are distinct enough for relevant and irrelevant documents to separate clearly.
• Listwise reranking considers candidates together. When passages are all about nearly the same thing, their differences become part of the comparison, potentially helping the model resolve subtle relevance distinctions. Holding and evaluating the set together also costs additional memory or time.

The listwise example is described as considering up to 64 candidates, but Vestal qualifies that limit from memory; it should not be treated as a verified service constraint.

Two phases need not mean two application API calls. The lab configures a retriever with the reranker model and query so Elasticsearch performs retrieval and reranking in one request. The application avoids manually coordinating separate inference and search calls, while the underlying second-stage computation still takes place. This preserves the workshop’s recurring tradeoff: reduce integration work, but measure the relevance and latency of the complete operation.

The closing materials point participants back to the notebooks and additional labs covering streams, lexical and vector retrieval, observability, OpenTelemetry, root-cause analysis and security. The hosted workshop was offered for one week after the session; the notebooks were intended to remain usable against a suitable Elasticsearch cluster with inference configured. The final invitation is to give feedback that improves the next workshop—a fitting end to a session whose practical work is inspecting what the system actually returns.