AI Engineer World's Fair 2026
How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI
Read the talk
Teaching agents to retrieve the evidence they need
Better reasoning cannot compensate for missing evidence. Mixedbread’s search harness combines semantic queries, specialized tools and training rewards to improve what agents retrieve.
From a talk by Hanna Lichtenberg and Aamir Shakir
Before you start: Familiarity with semantic search, retrieval chunks and LLM tool calls will help; no reinforcement-learning background is required.
What if the model can reason, but cannot find the evidence?
A model may be capable of answering a difficult legal or financial question when given the relevant documents, yet struggle to find those documents itself. As reasoning improves, that access problem becomes more consequential: retrieval is how the model reaches knowledge beyond its immediate context. Mixedbread calls the widening distance between these capabilities the knowledge gap.
Hanna Lichtenberg leads agentic search at Mixedbread; co-founder Aamir Shakir opens with the contrast between rapidly improving language models and slowly improving search. He characterizes the progression from GPT-3.5 to GPT-5.5 as exponential, while describing search’s progress over two decades as much slower. The practical concern is knowledge work beyond code: a stronger reasoning model still needs the right evidence to do useful legal or financial work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the difference between finding and using evidence
Two benchmarks make that distinction concrete. BrowseComp-Plus asks complex browsing questions over a fixed corpus of approximately 100,000 documents, unlike OpenAI’s original BrowseComp, which uses the open web. Databricks’ OfficeQA Pro asks difficult questions over historical U.S. Treasury material. Shakir describes its coverage as roughly a century; the published corpus spans 1939–2025.
The comparison starts with an oracle reference: give the model the correct documents alongside the question and measure its performance. This establishes supplied-evidence performance for that setup, rather than a universal ceiling on what the model could ever achieve. Then require an agent to locate the evidence in the full corpus. The difference exposes the cost of searching through noise.
Shakir reports the following rounded figures. The browsing discussion begins with BrowseComp-Plus, although he subsequently shortens its name to BrowseComp.
| Task | Reported oracle performance | Reported drop with Codex’s default tools |
|---|---|---|
| Browsing benchmark | 93% | Nine points |
| OfficeQA Pro | 64% | Eight points |
The point figures are described as drops; they do not establish separate absolute Codex scores. The model’s ability to use supplied evidence is stronger than its ability to obtain that evidence through the default tools.
Replacing the search tool recovers much of that difference. Mixedbread uses late interaction for retrieval. Shakir reports that GPT-5 with Mixedbread comes within three points of the browsing oracle, while the OfficeQA Pro gap almost completely closes. Mixedbread’s published companion experiments use different precise figures, so these rounded talk results should be kept separate from those published runs. Better retrieval helps immediately—but the queries agents send to it reveal another limitation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A semantic search tool receives a pile of keywords
One query found during benchmarking reads: “Senator woman questions billionaires, not a company. Then okay, thank you staff. Will check hearing.” It contains fragments that might occur in a hearing transcript, but it does not clearly describe the evidence being sought. A retrieval system designed for natural questions receives guesses about word overlap instead.
Shakir attributes this behavior to three influences:
- Codebase exploration. Coding agents learn to use tools such as
grep, where matching expressions in a file is useful. They carry that habit into document search by guessing what strings the document might contain. - Web search habits. Human-oriented web tools encourage agents to imitate familiar keyword queries.
- Benchmark incentives. Shakir argues that entity-heavy queries in BEIR and NanoBEIR structurally favor BM25, reinforcing lexical matching rather than a description of semantic intent.
These are the speakers’ explanations for the observed behavior. Their design target is clear: teach the agent to describe the evidence it needs, while retaining exact matching for tasks that require it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four tools inside a bounded search loop
Lichtenberg’s agent is designed for knowledge work beyond code search, with precision, speed and cost as explicit goals. Its harness runs on Mixedbread and separates four retrieval needs instead of asking one tool to serve every purpose.
| Tool | What the agent receives or does |
|---|---|
| Overview search | Summaries of up to 50 retrieved chunks |
| Semantic search | Full payloads of the top 10 chunks |
| Filter chunks | Finds and sorts chunks using metadata facets |
grep | Matches keywords or exact patterns |
Overview search exposes breadth without loading every full chunk into context. Semantic search supplies detail once the agent has a more focused intent. Metadata filtering and grep preserve structured and lexical access where those are the appropriate operations.
The harness permits at most four search rounds, with parallel searches inside each round. Its sequence is:
- Start with the user’s query, results from an initial semantic search on that query, and hints about available metadata facets.
- Use that corpus preview to split the search intent into at most four queries covering separate aspects.
- Choose the appropriate tool for each intent and run the searches in parallel.
- Deduplicate returned chunks before adding them to the agent’s context.
- Continue searching if necessary; once the evidence is sufficient, submit the relevant chunks in a plausible ranking.
The output is a ranked evidence list. The search agent’s responsibility is to find and order useful material, rather than necessarily compose the final answer itself.
Deduplication makes parallel exploration cheaper in context space: two searches can discover the same chunk without making the agent read it twice. A compact TypeScript implementation of that operation keeps a set of chunk IDs across rounds:
typescript
type Chunk = { id: string; text: string };
function collectUnseen(
batches: readonly (readonly Chunk[])[],
seen: Set<string>,
): Chunk[] {
const fresh: Chunk[] = [];
for (const batch of batches) {
for (const chunk of batch) {
if (seen.has(chunk.id)) continue;
seen.add(chunk.id);
fresh.push(chunk);
}
}
return fresh;
}
const seen = new Set<string>(["hearing-17"]);
const fresh = collectUnseen([
[
{ id: "hearing-17", text: "The senator questions the witness." },
{ id: "hearing-18", text: "The chair thanks the staff." },
],
[{ id: "hearing-18", text: "The chair thanks the staff." }],
], seen);
// fresh contains hearing-18 once.
In this hearing-themed example, hearing-17 is already in context and hearing-18 arrives through two searches. Only one new chunk is retained; this collection step does not replace the agent’s final relevance ranking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask for evidence before asking for a query
The tool menu alone does not ensure good queries. Lichtenberg describes five harness choices that shape how the agent uses it:
- Goal. Require the agent to articulate the evidence it needs before writing a query.
- Tool. Distinguish a semantic aspect search from an exact keyword match. Use semantic retrieval for the former and
grepfor the latter. - Prompt. Ask for “one concise sentence describing what it wants to find,” rather than telling the model to write a search query. The wording is intended to avoid triggering its familiar BM25-style habits.
- Examples. Demonstrate good queries and how to divide an input into separate aspects worth investigating.
- Seed. Supply initial semantic results so the agent can learn the corpus’s language and decide where to dig deeper.
Together, these choices move query writing after an explicit decision about what evidence would help. The agent is no longer asked to guess a bag of terms before it has seen the corpus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reward both the retrieved list and the search that produced it
The next layer is training. Mixedbread chooses a small language model for speed, then trains its search strategy: tool selection, semantic-query quality, exploration, ranking and efficiency. Training begins with supervised fine-tuning using a larger teacher model, followed by on-policy reinforcement learning with a custom search reward.
That reward combines two views of success:
- Retrieval reward: evaluate the final ranked list using recall and nDCG, alongside an LLM judge. Its rubrics assess relevance of the returned chunks and whether their ranking is plausible.
- Trajectory reward: use an LLM judge to assess the search process, including tool choice, efficiency, natural-sentence query formulation and the amount of exploration. Both insufficient and excessive exploration matter.
The distinction lets training address the result and the behavior separately. A useful final list does not, by itself, show that the agent chose efficient tools or wrote good semantic queries. Conversely, a well-formed search trajectory still needs to retrieve relevant evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From parallel exploration to reported benchmark results
The example trajectory starts with the initial search and metadata hints, launches four parallel searches in its first round, then performs a single grep search in its second. Its long, rambling input comes from the Congress Hearings task in OBLIQ-Bench, called Oblique Congress in the presentation. The semantic queries are sentences describing what the agent wants to find; the grep queries remain keyword patterns. That difference is intentional: the agent changes query style with the tool rather than enforcing natural language everywhere.
At the time of the presentation, the trained agent is unreleased. Lichtenberg reports an intermediate Oblique Congress nDCG@10 of 0.4, compared with 0.18 for the benchmark paper’s GPT multi-hop agent. The paper identifies that baseline as GPT-5.2 Multi-Hop Agent and reports 0.183 in its first version. The Mixedbread figure remains a preliminary reported result; the presentation does not provide enough experiment detail to establish identical evaluation conditions.
A separate beta, Mixedbread agentic search, is already running in production. Lichtenberg reports first place on Snowflake’s MADQA benchmark with 93.4% accuracy when Gemini 3.5 Flash, as named in the presentation, uses Mixedbread agentic search as its search tool. She also reports lower effort than comparable systems using other search agents, without giving a numerical effort measure, and points to the benchmark leaderboard for comparison. These are the team’s reported results, rather than an independently established historical leaderboard snapshot.
The ending returns to the original constraint: capable models still have room to improve through their access to knowledge. Better retrieval tools recover missing performance; a dedicated search agent adds tool selection, query formulation and controlled exploration around those tools. The production beta and the unreleased trained agent represent different stages of that work, with evidence quality and the effort needed to obtain it both part of the objective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Code and evaluation materials for deep research over a fixed corpus of approximately 100,000 documents.
Databricks' document-reasoning benchmark, with scoring code and instructions for accessing Treasury Bulletin data.
Five retrieval tasks involving implicit relevance, including recovering congressional-hearing passages from imperfect recollections.
An introduction to multimodal document question answering and evaluation of answer accuracy alongside search effort.
Further reading
Mixedbread's published retrieval comparisons across BrowseComp-Plus, MADQA and OfficeQA-Pro.
Updates since the talk
Current instructions for enabling hosted agentic search, configuring ranked results and streaming search traces.
Read the complete timestamped transcript
- 0:00
Hey. Today we're gonna talk about how we taught agents to use code retrieval or how we call it internally, closing the oracle gap with knowledge agents.
- 0:10
I'm Hanna. I'm an AI engineer at Mixedbread, and I'm leading the agentic search at Mixedbread.
- 0:17
Yeah, I'm Aamir. I'm the co-founder of Mixedbread, and let's jump into it. So we have seen over the past years with the LLMs just getting better and better, that the reasoning capabilities of the models is growing exponential.
- 0:31
So just think back how good GPT-3.5 was and how good GPT-5.5 is, right? It's like a clearly exponential curve. But if you look at search over the last 20 years, basically, we see that it's improving, but it's improving very, very slowly.
- 0:49
So there's a huge gap between how basically LLMs and the reasoning is evolving and how retrieval is evolving. But these retrieval tools are the main access pattern for this reasoning layer to get the right knowledge and to be truly useful beyond code, like in legal work, in finance work, and so on and so forth.
- 1:13
Internally, we call this gap happening right now between reasoning and search the knowledge gap. And it's not just one obscure theory of ours. We see it actually with real benchmarks and real tasks that this gap exists in real world as well.
- 1:28
Let's take two benchmarks here, BrowseComp+ and OfficeQA Pro. BrowseComp+ is like a browsing benchmark where we have pretty complex queries and a corpus of one hundred thousand documents, and we just try to answer these questions.
- 1:44
It's like real world deep search task, basically. And OfficeQA Pro, which has the treasuries of the U.S. Treasury of the past, um, hundred years, basically, and we ask really complex question over this.
- 2:01
OfficeQA Pro was created by Databricks, and BrowseComp was created by OpenAI, and BrowseComp+ is a version of it with a fixed size corpus, while BrowseComp was always an open web.
- 2:11
And we see here's a oracle performance in dotted lines. And oracle means what is the maximum theoretical performance of the models if you would put in the right documents with the question, with the question.
- 2:26
We see for Office, uh, for BrowseComp, it's ninety-three percent, and for OfficeQA Pro it's sixty-four percent. And now we take something like Codex with its default tools and want to see, hey, what is the performance these tools get right now?
- 2:42
And you see there's a sharp drop in the quality of the answers Codex produces. For BrowseComp it's nine points, and for OfficeQA Pro it's eight points. So we see that the models are extremely capable if they would get the right documents.
- 2:58
But if we put them into the noisy corpus, the performance drops sharply. Meaning that actually the bottleneck here is not the reasoning, it's actually the access to the right knowledge it needs to answer this question.
- 3:16
So we see this knowledge gap in real world.
- 3:21
And if you would just drop in way better search using Mixed Bread, which is a search tool using late interaction, we can recover most of the performance. So for BrowseComp, the difference between the Oracle and the, uh, GPT-5 with Mixed Bread is just three point.
- 3:38
And for OfficeQA Pro, we even almost completely close the gap. With giving the model better search tool, we can recover most of the knowledge gap. But looking at the queries the model asks or writes right now to the search system, it gets super, super interesting.
- 3:55
So here's an example query we found, uh, during some benchmarking, which is, "Senator woman questions billionaires, not a company. Then okay, thank you staff. Will check hearing." It's basically gibberish, right?
- 4:08
If you have a search system which, uh, wants to have neural questions or semantic question and just gets these keywords over keywords, it gets confused. And the reason why the models write this type of queries is they're mostly trained for coding tasks, like for coding agents, which are then optimized for code-based exploration using tools like grep.
- 4:31
And grep just tries to find regular expressions, right? So the model tries to just write as many expression it thinks i- in the document to find the right thing.
- 4:42
The second thing is the models are trained to use the web pretty efficiently. And to use the web tools properly, which are highly optimized for humans, they try to mimic human-like query patterns.
- 4:55
So just keyword on keyword on keyword. And number three, obviously, is the benchmark bias. Most benchmarks we have right now like BEIR, NanoBEIR use caveman-style queries, which are entity-based queries that structurally favor heavily BM25.
- 5:11
So right now, the agent guesses the keywords to actually increase the overlap between the query and documents and can't really use powerful search tools properly.
- 5:23
This motivated us to make our own search agent to teach it to use powerful search properly, especially to use the right search tools for different use cases, um, and to work beyond code search for knowledge work.
- 5:40
And most importantly, the agent should be precise, fast, and cheap. The first step for building this agent was to define a very powerful harness. Um, our harness is fully built on the Mixed Bread platform.
- 5:54
Our agent has four main search tools, which are the overview search. This is used as a very wide semantic search where the agent receives up to 50 retrieved chunks, um, and it sees only summaries of the chunk contents to really have just like an overview of the corpus, so we, uh, what exists and to not fill up
- 6:18
the context too much. Then there is the main semantic search tool where the agent gets the full payload of the top 10 retrieved chunks. Um, it has a filter chunks tool where it can sort by-- sort and find chunks based on metadata facets.
- 6:36
And of course, we also have a GREP tool for the keyword match searches. Our, um, agent loop, the harness itself is very simple and short because the agent should be fast, but we still wanted it to, uh, have a lot of exploration possibilities.
- 6:54
That's why we decided to, um, define that it has at maximum four search rounds, but within each search rounds, it can have parallel searches. So it can start several search at...
- 7:09
searches at once. Um, initially, our agent sees the user query, but also already, um, results from an initial semantic search on the user query, plus, um, hints on which metadata facets are available.
- 7:25
Um, so that based on this preview of the corpus, the agent can start its search planning. It splits its search intent into a maximum four queries,
- 7:37
um, which cover separate aspects. And then for each query and each search intent, it will se- uh, pick the best tool.
- 7:49
Um, as you can see like an example here on the right.
- 7:53
Then when, um, the results of the tools are returned to the agent, we are de-duplicating chunks so that it's never seeing several chunks, um, several times to not fill up the context.
- 8:07
Um, and yeah, the next search run can start, and as soon as the agent has enough evidence to answer the user query, it submits its ranking. Um, yeah, where it just outputs all the chunks that are relevant to the query in a plausible ranking.
- 8:25
So why does this harness encourages better semantic s- queries and like better use of search tools? Um, there are five more main points for this, which is first the goal framing.
- 8:38
The agent has to articulate, articulate what evidence it needs before writing the query. Um, then the different tools are very important, so it can really use semantic search only if it needs aspects, and also only uses GREP when it needs exact keyword matching.
- 8:58
Um, then the way in which we frame the task of query, uh, writing is an important point. We kind of trick the, um, model into not thinking it has to write the typical BM25 based query by just instructing it to write one concise sentence describing what it wants to find instead of directly instructing write a search
- 9:23
query. So it cannot fall into this old pattern. Um, then we pro- also provide examples in the prompt just to show the... a few good queries and also how to divide an input query into different aspects to explore, um, the corpus.
- 9:43
Yeah. What also helps is that we provide the origination-- original query semantic search results so it can see what the corpus is about and really define, um, see the language, define where to dig deeper in which aspect.
- 10:01
The second important step into building our s- own search agent was, of course, the training itself.
- 10:08
So, you know, for training, we just decided to have a very small LLM to make the agent even faster. So we are training the small agent, uh, to optimize the search strategy itself, to have better tool choice, um, higher quality of semantic queries, more exploration ranking, and to be more efficient.
- 10:31
So the first step in training is, um, supervised fine-tuning with a larger teacher LLM. And then we, uh, did On-policy reinforcement learning with an own search reward. Our search reward is, um, a combination of both, of a retrieval reward and a trajectory reward.
- 10:54
The retrieval reward, um, is the based on like the metric result, typical retrieval metric results like we call it nDCG that the agent's final ranked list achieves. Plus also, um, an LLM retrieval judging, where it got like, uh, several rubrics where it decides, um, if the rubrics is hit or
- 11:19
not. The rubric is hit or not, which, uh, if the agent was that relevant for this query, are all chunks relevant, and is the ranking itself plausible. For the trajectory reward, we are also using an LLM judge, and here, um, is the point to like really improve the tool choice, make it more efficient, and to, um, improve
- 11:43
the quality of the queries. Of course, like we have rubrics that are whether judge is deciding if the query is really a natural sentence or so if the amount of exploration is sufficient, is it too much or too less.
- 11:59
Here you can see an example trajectory. You see that in the beginning we have the initial search and the initial metadata hints. Um, then the agent did four parallel searches in the first round and then a second round, just a simple GREP tool.
- 12:17
And here's an extract of, um, a trajectory. Where we see the agent queries. Um, the u-- input query is on the left. This is a very long query, typical rambling style query from the Oblique Congress benchmark.
- 12:36
And yeah, we see the agent queries here for the first semantic search tool. It's really a sentence describing what it wants to find and not a weird keyword-y edit behind each other formulation.
- 12:52
The web tools, of course, are like typical keyword patterns, which is exactly as intended. Our trained agent is not released yet, unfortunately. However, we have some intermediate results. Here on the left side for the Oblique Congress benchmark, we see that we achieved an nDCG of 10-- at 10 of 0.4, which is a huge jump, um, towards
- 13:17
the model that performed best on the paper of this benchmark, which is the GPT multi-hop agent, and which achieves 0.18.
- 13:28
On the right side, you can see the result of our beta version of the search agent, which we have, uh, right now in production. It's the Mixedbread agentic search.
- 13:39
And this agent is top one on the Snowflake's MetQA benchmark, achieving an accuracy of ninety-three point four when we, um, give the Gemini three point five Flash model our agentic search as search tool.
- 13:55
And this, uh, this performance was also achieved while having way less effort than comparable LLMs with other search agents. You can check out the MetQA leaderboard. These results show that there's still a lot of room for improvement when it comes to huge language models and their search tools.
- 14:18
And if you want to be part of pushing the boundaries of retrieval even further, um, we are happy awaiting your application.