The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

An agent writing code or doing deep research eventually needs information it does not have. It must find that material before it can finish the task. Jo Kristian Bergum, CEO of Hornet.dev, calls this agentic search: search inside an agent loop. After more than twenty years working on retrieval, he sees a new role for BM25, a roughly thirty-year-old lexical scoring function.

The system needs three parts, each responsible for a different piece of the work:

• Model: Formulates queries and uses tools to obtain missing information.
• Harness: Exposes retrieval through an interface the model can operate, such as tool calls or code mode.
• Retrieval engine: Executes searches efficiently, potentially across billions of documents.

This separation matters because model capability alone does not supply a search system. The model needs an interface for expressing its information need, and that interface needs an engine capable of searching the underlying collection. Agentic retrieval depends on how those parts work together.

BM25 stands for Best Match 25. Bergum recounts a series of scoring experiments in which the twenty-fifth worked best, leaving its number in the name. The function scores a document against a query through interactions between their terms. That score is a proxy for relevance, allowing the system to select the top K documents.

Scoring and execution solve different problems

A straightforward implementation would score every document, then keep the highest-scoring results. At large scale, the work lies in finding those results efficiently. Bergum points to decades of algorithms for accelerating top-K retrieval. BM25 names the scoring function; it does not identify the algorithm or engine used to execute it. Engines using the same scoring rule can therefore have different serving costs and performance.

Bergum’s explanation for BM25’s comeback begins with the user. A language model already knows entities, companies and dates. It can draw on that knowledge to supply specific search terms. The scoring function has stayed the same, while the client has acquired a much stronger ability to describe what it needs.

BM25 traditionally served as the baseline against which researchers compared more elaborate neural methods. Search evaluation likewise often assumed a person issuing one query and scanning ten blue links. An agent can issue many queries. That makes the single search result a less complete description of the system’s usefulness: the agent can read a response and search again.

Bergum turns to BrowseComp+, a deep research benchmark with 830 long, riddle-like questions—closer to a pub quiz than a short keyword lookup. The corpus contains roughly one hundred thousand web documents. A model receives a simple Search tool: it supplies a query string and receives snippets. Each question has a reference answer, allowing the benchmark to check whether the complete loop produces the expected answer.

The context window is a small working space

Bergum compares context windows to floppy disks. A disk held about 1.4 megabytes; he estimates that models can use around 350,000 tokens before quality starts to degrade. The token figure is his working judgment, rather than a universal threshold. The analogy explains why retrieval remains necessary: the system must choose useful information for a bounded working space.

BrowseComp+ makes that selection problem visible through two conditions:

• Evidence supplied directly: Put the documents needed to answer the question into the model’s context. Accuracy is high, including with older models.
• Evidence found through search: Give the model a retrieval tool and require it to obtain the evidence. Accuracy falls, because success now depends on query formulation, the harness and the retriever.

Bergum concludes that reasoning is not the main bottleneck in this experiment. The model can use the evidence once it receives it; finding that evidence introduces much of the difficulty. The comparison does not isolate the retriever as the sole cause of failure, since the tool interface and the model’s search decisions also change what reaches context.

A riddle consequently becomes a search trajectory. The model executes a query, reads the response, reformulates and repeats until it finds the answer or fills the context window. Bergum expects evidence selection to remain important as models improve: a more capable reasoner still needs the relevant documents in its working space.

Hornet examined these search trajectories and compared model-written queries with human queries in the AOL search logs. Bergum describes human queries as short, a pattern he also recognizes in more recent logs. The model writes longer queries and reaches for phrases and operators such as site:. Hornet’s analysis of the agent query workload develops the comparison. These are different inputs from the few terms a person typically types into a search box.

Which BM25 configuration is being compared?

Bergum notes that BM25 has two hyperparameters, then challenges the baseline used in the original BrowseComp+ research. In his account, later research found that those parameters were poorly suited to the benchmark’s long documents. Embedding-based methods looked substantially better against that baseline, while changing the BM25 parameters had a dramatic effect on overall answer accuracy.

The qualification is specific: a poorly configured baseline can distort the comparison between retrieval methods. Bergum does not give the parameter settings or numerical accuracy changes in this discussion. His argument supports examining the configuration and document characteristics before concluding that lexical retrieval is weak; it does not establish that every tuned BM25 implementation outperforms embeddings.

Why literal matches help the agent

Bergum gives three reasons for BM25’s usefulness in this workload:

• Specific identifiers: Names, entities, ZIP codes and SKUs can become precise query terms. He argues that embedding-based retrieval has more difficulty preserving these distinctions when the literal identifier matters.
• Cost and availability: BM25 avoids the embedding-inference step and has mature tooling. Bergum contrasts that with encoding text using embedding models that can have about eight billion parameters, along with the infrastructure needed to serve them.
• Inspectable feedback: The model can see the literal terms and phrases in returned text and understand why a result appeared. That gives it information for reformulating the next query.

Inspectability matters because results become inputs to later searches. If the agent recognizes which words led to a document, it has a concrete basis for changing its query. Literal matching therefore contributes both to finding a candidate and to deciding what to try next.

Bergum next introduces Scaling Direct Corpus Interaction via Dynamic Workspace Expansion, work from Jimmy Lin’s group at Waterloo. The motivating problem is web search for agents over potentially billions of documents. Those documents cannot all enter context. A retriever selects material from the large corpus and places the retrieved documents in a workspace, which Bergum likens to a search results page for agents.

Titles and snippets lead to deeper reads

Organizing the workspace as a file system enables progressive disclosure. The model first sees a document’s title and a small snippet. It can then decide whether to read more. Full documents remain available for inspection without all being inserted into context at once. The workspace separates having retrieved a document from having read its contents.

The model can inspect that material with familiar coding-agent tools: grep, ripgrep, sed and awk. The retrieval engine searches the large collection; file tools help the agent examine the selected material and manage what it reads. The resulting architecture combines retrieval infrastructure with a sandbox or virtual file system and a Bash interface.

The useful data flow is corpus → retrieved files → titles and snippets → selected deeper reads. BM25 supplies corpus-scale retrieval, while file tools give the model a way to inspect the results. In Bergum’s closing argument, BM25 and grep work well together because both expose literal text matches the agent can understand.

Build around capabilities models already practice

Bergum describes this as a pragmatic adaptation to model training. Developers of frontier models invest in coding, Bash and tool use, so a retrieval interface built around those capabilities may benefit from that investment. He expects this direction to improve with models, while leaving open whether more capable systems might eventually work directly through a browser. The design is a bet on a useful interface, not a guarantee about every future model.

Traditional retrieval evaluation often begins with one query, one ranked list and a metric such as nDCG. Bergum argues that this gives an incomplete picture of an agent that can reformulate queries, expand its search and use later results. His proposed endpoint is the task itself: for question answering, does the agent get the answer right?

His dismissal of classical evaluation is forceful, but the mechanism behind it is concrete. One ranked list measures one retrieval step. The agent’s answer emerges from a sequence in which each response can change the next query. Task success captures the outcome of that interaction between model, harness and engine.

The engine still has to serve the workload economically

Hornet is betting on BM25 as a fundamental retrieval primitive and on executing it efficiently. Bergum presents a comparison with anonymized engines using the same type of hardware and one hundred million web documents on a single node. He claims Hornet achieves more throughput for the same spending, with potential savings for companies building search infrastructure.

The performance claim remains qualitative here. Bergum corrects his explanation of the vertical axis from queries per second to latency; his explanation does not establish the horizontal axis, numerical values or competing engine identities. Those omissions prevent a quantified speedup or reproducible cost comparison. Serving efficiency also answers a different question from answer accuracy: returning results faster does not establish that the agent found the evidence it needed.

Bergum ends with the combination he wants builders to remember. A model can read and write quickly, draw on general knowledge and reformulate queries, making simple tools more powerful. BM25’s parameters, implementation and performance still need attention. Corpus-scale lexical retrieval followed by grep-style inspection gives that capable user understandable results to work with—and another search to make when the first one is incomplete.