Where RL Will Take Search — Maximilian-David Rumpf, SID.ai
Read the talk
Where Reinforcement Learning Can Take Search
Maximilian-David Rumpf explains how a trained search sub-agent can replace a fixed retrieval pipeline with an adaptive loop—and reduce the cost of finding context for knowledge work.
From a talk by Maximilian-David Rumpf
At a glance
Ideas worth remembering
A fixed retrieval pipeline can detect inadequate results without being able to search again. An adaptive search model can use those results to choose its next query or filter.
Rumpf’s case for reinforcement learning rests on checkable document retrieval outcomes, thousands of training attempts per second, and rewards that can account for latency.
The reported specialist averages around five seconds instead of around two minutes and costs roughly one hundredth as much as frontier-model searching on the task; it still trails vector-and-reranker pipelines in latency.
A separate search agent can keep exploratory results out of the main agent’s context and reduce search costs, but its selection quality still determines the value of the context it returns.
The closing opportunity is retrieval over internal organizational knowledge. Claims of arbitrarily good search remain a forecast, with the method’s ultimate ceiling unknown.
Better retrieval comes with an expensive search phase
Agents improve search by spending time finding the context a task needs. Maximilian-David Rumpf, founder and CEO of SID.ai, opens with the tradeoff: agents are about twice as likely to find the right documents, but their searches cost roughly 100–1,000 times more than a classical search query and take minutes rather than milliseconds. These are the comparisons he presents, rather than a universal guarantee for every search workload.
Rumpf estimates that searching consumes 30–50 percent of an agent’s tokens, usually near the beginning of a task. Before the agent can do the requested work, it must find material worth working from. His proposal is to hand that phase to a sub-agent and train a model specifically for searching. The main agent still performs the task; the specialist supplies its context.
The target is a combination of retrieval quality, speed and cost. Rumpf describes a benchmark spanning legal, finance, knowledge bases, science and email, with both academic and internal tasks. In his comparison, vector-only retrieval and reranking sit below frontier models in search performance, while frontier models spend minutes per question. The question is whether a specialist can retain high recall—finding the relevant documents—without paying that general-purpose inference bill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reranker can recognize failure without being able to fix it
The classical pipeline begins with a question. An LLM may rewrite it, a search backend retrieves candidates, and a reranker orders those candidates before returning results. Each component can be good at its own job while the complete system still fails. Rumpf’s diagnosis concerns the sequence itself: its decisions are set at design time, and each question receives a fixed amount of computation.
The reranker illustrates the limitation. It may recognize that none of the retrieved documents adequately answers the question, yet its available action is to return an ordering of those documents. It cannot launch another search to obtain better candidates. Recognizing insufficient evidence therefore does not improve retrieval unless the surrounding system gives that recognition a way to change what happens next.
Unexpected questions expose this rigidity. Developers add special cases and tweaks to handle failures, but the possible questions keep extending beyond what they anticipated. Rumpf argues that this creates a long tail of failure: improving individual components and adding exceptions cannot exhaustively specify how to search for every future question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the model decide what to search next
Rumpf frames the proposed change through earlier shifts in computer vision and chess. His vision example moves from handwritten edge detection to narrow object detectors and then more general vision-language models. His chess analogy moves from human-authored rules toward systems such as AlphaZero and MuZero that learn strategies. These comparisons motivate his search thesis: move beyond algorithms such as BM25 and PageRank, and beyond separately optimized vector retrieval and rerankers, toward a model that learns how to conduct the search.
The mechanism is a loop between one model and a database. The model searches, reads the results, and decides whether to search again. It can apply metadata filters during the process and constrain subsequent searches before eventually producing a ranked list. Reading results becomes information for choosing the next action, rather than merely the final stage before returning an answer.
This loop also makes computation depend on the question. A difficult question can receive more searching and iteration instead of the same fixed budget as an easy one. Rumpf’s aim is to let the model discover useful search strategies rather than prescribe the sequence in advance. That still leaves important design choices in the surrounding system, including the available search actions and training rewards; the learned part is how the model uses those opportunities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search offers a checkable reward and a repeatable environment
Reinforcement learning needs a way to evaluate attempts and enough opportunities to make them. Rumpf calls search rewards verifiable and grindable. For the training questions he describes, verification asks whether the model found the correct document. The environment then lets it attempt the task repeatedly, at thousands of attempts per second during a training run. A checkable retrieval outcome supplies feedback without requiring a person to judge every search attempt.
Specialization is the other part of the argument. A general-purpose language model has capabilities that a search specialist may not need. Rumpf compares this with CPUs, GPUs and ASICs: a general device may be capable of performing a task, while a more specialized device performs it much more efficiently. The analogy explains why narrowing the model’s job could reduce cost, but the talk does not specify which model components are unnecessary or how the architecture changes.
Rumpf reports that search quality increases predictably with compute in the training example he presents. Rewards can also account for latency, giving training a reason to favor efficient searching alongside successful retrieval. The goal is for the model to discover its own strategies. He says the team has not yet seen a ceiling, while explicitly acknowledging that they do not know whether the approach can improve without limit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A faster specialist also keeps the main context cleaner
Rumpf presents results for the specialized search model, including a configuration with parallel execution. He reports approximately 20 times faster searching: around five seconds on average instead of around two minutes, at roughly one hundredth of the cost of a frontier model on the task. The times and speedup are approximate. The supplied transcript does not report benchmark sizes, absolute quality scores or parallel execution details needed to assess how broadly those gains transfer; those details may appear in the referenced charts, whose contents are not supplied.
The specialist still has higher latency than a vector-and-reranker pipeline. Rumpf expects that gap to narrow, but reaching classical retrieval latency remains a future goal. The reported result is therefore a large improvement over frontier-model searching, with a remaining speed tradeoff against the simpler pipeline.
Delegation changes what enters the main agent’s context window. When the main agent searches directly, it encounters useful results alongside bad ones, and both become part of the context it carries forward. A search sub-agent can perform the reading, thinking and iteration separately, then pass back selected results. The main agent receives the output of the search rather than its entire exploratory history.
Rumpf expects this separation to improve correctness by giving the main agent more useful material and less irrelevant context. It also moves the search work associated with that 30–50 percent token share to the cheaper specialist. The benefit depends on the specialist finding and selecting good results: separating contexts does not itself guarantee retrieval quality. Nor does making the search phase about 100 times cheaper imply that the complete task becomes 100 times cheaper, because the main agent still has work to do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The destination is knowledge inside organizations
Rumpf closes with an ambitious forecast: scaling reinforcement learning could produce arbitrarily good search across domains, while further speed and cost improvements could make these models useful for voice and e-commerce. That forecast goes beyond the demonstrated results. His earlier acknowledgment that the team does not know the method’s ceiling remains the qualification on the claim of unlimited improvement.
The larger opportunity, in his view, is enabling knowledge work with information that public web search cannot supply. He uses J.P. Morgan as an example: the knowledge needed to run the organization resides deep in its databases rather than on the web. Better retrieval could make more of that internal knowledge usable by agents. The ending brings the argument back to the initial dependency: an agent needs the right context before it can do valuable work, and much of that context sits inside organizations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:12
Okay. Today I'm gonna talk about where reinforcement learning will take search. And some, some background on me. Uh, I'm the founder and CEO of SID.ai. We're a stealth-ish AI lab for search. We're backed by some pretty amazing people, uh, and we're hiring.
- 0:35
Okay. Agents are a new paradigm for search. You can now get vastly higher quality results, um, twice as likely to find the right documents. But it's incredibly expensive, um, about a hundred to a thousand times more expensive than what you'd get out of a classical search query. And it is extremely slow. You're looking at minutes and not milliseconds like TurboPuffer.
- 1:05
And what this means is that agents spend thirty to fifty percent of their tokens on searching, and this is usually at the beginning of some task. It finds the right context to then do whatever you ask it to do. And the idea here is quite simple. First, instead of having the main agent do the searching, you pass the searching to a sub-agent, and you train a model to be a great sub-agent. And the question here that we'll answer today
- 1:36
is how much cheaper and faster can we make this with reinforcement learning? And this is really our target. So this is a benchmark across legal, finance, knowledge bases, science, email, a bunch of different tasks, some academic benchmarks, some internal benchmarks. And this is where you currently are. You see reranker and vector-only performance at the bottom, and you can see frontier models essentially kind of like go through here
- 2:06
at the cost of spending many, many, you know, minutes per question. And can we get a model to kind of like be extremely accurate, um, have extremely high recall, but also be incredibly fast and cheap. And let's quickly look at classical search. Uh, this is the pipeline that many of you guys will be familiar with. A question comes in. You might have an LLM that rewrites the question. You then execute that on a search backend. You might have a reranker, um, and
- 2:36
you get your results at the end of the day. Um, it is essentially a pipeline of chained, locally optimized models. And all of the decisions are baked in at design time. And you expend a fixed amount of compute per question. And this one is really important. The reranker might know that the results are insufficient at answering the question, but the reranker can't take action. It can only essentially return the results even when they're bad. And what this means is that in practice,
- 3:06
a pipeline like this accrues a long tail of failure where unexpected questions come that, you know, the designer didn't have something for. And in practice, this usually means people add lots of edge cases, um, to essentially fix these. But of course, you can't design infinite edge cases, and you can't add infinite tweaks. And so the strategy here is one that we've seen before. Machine design outperforms human design.
- 3:36
Um, and we saw this in computer vision where you had your, you know, primitive edge detection algorithms. You then had the box around a dog generation of models, um, that were very good at this like very narrow task and locally optimized for it. And then you had VLMs that were extremely good at all parts of the search pipeline. You saw this again with chess, with IBM Deep Blue being largely a collection of human-written rules, um, Stockfish bridging the two, and then AlphaZero and MuZero essentially
- 4:05
completely just putting it all inside of the model. And we're gonna see something similar happen to search, where we have our existing algorithms like BM25 and PageRank. Um, then we had an evolution from that with small models that did some tasks very well, like vectors and rerankers. And now essentially this new paradigm of pure RL where we actually don't bake any design decisions into the model. And what this looks like in practice is, um,
- 4:36
something like this. Um, so you have one model. Uh, it goes back and forth with the database. It can search, it can read results, it can iterate, uh, it can search again until it is happy. It can set metadata filters on the fly. It can constrain its search. It can try as much as it wants. Uh, and in the end, it produces a ranked list of results. Uh, and what you get is a model that makes all of the decisions and can adapt to any question on the fly
- 5:06
and, for example, use much more compute if a user asks a very difficult question. And what helps us here is that search is verifiable, um, and reinforcement learning needs rewards that are verifiable and grindable. Verifiable here means for a given question, did you find the correct document? And we can design this and tell this quite easily. Um, and is there an environment where the model can attempt this question loads and loads of times? And in practice for us this means, uh,
- 5:35
thousands of times per second during a training run. Uh, and the second part is kind of like, can we turn the models that are currently very general and very general purpose into something that is much more specialized? And it turns out we don't actually need most of the parts of a language model to be extremely performant at search. Uh, and similarly with like CPUs and GPUs and ASICs, um, a CPU in theory can do anything that a GPU can do. But you would never want to use a CPU to do LLM
- 6:05
inference, for example, um, because the much more specialized version is much more effective. And this really makes search an ideal target for RL. Um- And this is what happens when you train a model on this task. And so again, here we added the vector and reranker-only baselines. This is of an earlier task. And what we see is that search quality increases very predictably with compute. And we can mix in other rewards like latency, um, and different kind of like retrieval strategies to make it even more performant.
- 6:35
Importantly, we don't really tell the model what to do. Much like in AlphaZero in chess, we want it to discover its own strategies and its own tricks to essentially search well. Uh, and we don't know whether this method has no ceiling, but we're definitely not yet seeing a ceiling to this approach. And these are the results. Uh, so this is the same chart as before, and this is SID-1 and then SID-1, um, with some parallel execution on the left-hand side. And so what this ends
- 7:05
up meaning is you get-- you're about twenty times faster, so instead of taking around two minutes, you take around five seconds on average. And it's about a hundred times cheaper than using a frontier model for this task. Uh, there's some more detail here, but like yeah, the cost and kind of like speed are just completely incomparable. Uh, you can-- Yeah, um, it's not quite at the latency of a vector and reranker pipeline, but in practice, we think we can get there quite quickly.
- 7:36
And how does this look like in production? So this is a usual kind of like agent execution trace. The agent does some searching here. It finds some good stuff. It finds some bad stuff. Um, but all of the bad stuff that it finds is essentially polluting its own context window. And what we can instead do is use, um, a sub-agent here that does all of the searching and thinking and iterating for the main agent, and the main agent only ever sees great results. And this means that the main agent sees more good stuff, which means it's more likely to be correct. And it
- 8:06
is also, um, extremely cost-effective, uh, where those thirty to fifty percent tokens that were earlier used by the main agent to do searching can now be passed off to this just, you know, hundred X cheaper search sub-agent.
- 8:23
Uh, and
- 8:28
where will this take us? Scaling RL will give us arbitrarily good search in any domain.
- 8:35
And RL models will become even faster, which will allow them to be used in things like voice and e-commerce. They'll become even cheaper than we are currently. Um, so the charts that you saw there, but like I think we can move even further. Uh, and better search will unlock more knowledge work tasks. Uh, the web is actually quite small, uh, in comparison to the entirety of data that is there-- uh, that is out there. Uh, and the most valuable information is
- 9:05
not on the internet. For example, how to run J.P. Morgan is nowhere on the web, but it is deep inside of the databases at, at J.P. Morgan. That's it for me. Thank you.