# Where RL Will Take Search — Maximilian-David Rumpf, SID.ai

Maximilian-David Rumpf, SID.ai

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/iJVxxxHM_Oc-where-rl-will-take-search) · [Video](https://www.youtube.com/watch?v=iJVxxxHM_Oc)

## Where Reinforcement Learning Can Take Search

Maximilian-David Rumpf explains why a search model should be able to inspect results, change its approach and try again—and how a trained specialist can make that process cheaper while keeping the main agent’s context cleaner.

### Better retrieval comes with an expensive search phase

[0:12](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=12s)

Finding the right documents can consume much of an agent’s effort before it starts the work the user requested. [Maximilian-David Rumpf](https://maxrumpf.com/), founder and CEO of [SID.ai](https://www.sid.ai/), opens with the comparison driving his research: agents are about twice as likely to find the right documents as classical search, but cost roughly 100–1,000 times more per search and take minutes rather than milliseconds. Better context comes with a substantial bill.

Rumpf estimates that searching uses 30–50 percent of an agent’s tokens, usually at the beginning of a task. The agent needs to gather context before it can act on it. His proposed division of work follows that dependency: the main agent delegates searching to a sub-agent trained specifically for retrieval, then uses the returned material to perform the task. Reinforcement learning is the proposed way to make that specialist faster and cheaper.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 76 seconds](https://ai.engineer/api/talk-essay-media/208e3aa320dd8309aeca2f555cd380c66fe98f8569922972b53f6c4ea3deb83d)

Better retrieval comes with an expensive search phase

The target spans legal, finance, knowledge bases, science and email, using a mixture of academic and internal benchmarks. Rumpf presents vector-only retrieval and reranking as lower-performing baselines, while frontier models improve retrieval by spending minutes on each question. The goal is to retain high recall—the ability to find the relevant documents—while reducing the time and cost of the search phase. These comparisons concern the workloads he presents; they do not establish the same improvement for every corpus or question.

### A reranker can recognize failure without being able to fix it

[2:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=126s)

The classical pipeline Rumpf describes follows a fixed route: a question enters, an optional LLM rewrites it, a search backend retrieves candidates, and an optional reranker orders them before the system returns results. These are chained, locally optimized models. Improving a component can make its particular operation better, but the sequence still determines which operations happen and how much computation each question receives.

The reranker exposes the problem most clearly. It may recognize that the retrieved documents do not adequately answer the question. Yet it can only reorder the candidates it already has; it cannot obtain better ones by launching another search. The system has detected insufficient evidence without giving that detection an action that could repair the failure. It returns the results anyway.

Unexpected questions then produce a long tail of failures. Developers add special cases and tweaks for questions the original design did not anticipate, but no finite collection of exceptions covers every future request. Rumpf’s objection is to making retrieval decisions once, during system design, and applying them uniformly even when the results reveal that a different approach is needed.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 204 seconds](https://ai.engineer/api/talk-essay-media/416b7554d1f4aadf98378279aa37629d7bb99607f6c093e35ed5ebef4ae3f00e)

A reranker can recognize failure without being able to fix it

### Let the model decide what to search next

[3:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=216s)

Rumpf motivates learned search strategies through two earlier changes in machine learning:

- **Computer vision:** his progression moves from handwritten edge detection to narrow object detectors—the generation of models that could put a box around a dog—and then to more general vision-language models.
- **Chess:** his analogy moves from human-authored rules, through systems combining approaches, toward learned strategies in **AlphaZero** and **MuZero**.

He expects search to follow a similar progression: established algorithms such as BM25 and PageRank, then specialized vector models and rerankers, then a model trained to make retrieval decisions throughout the search.

### Results become input to the next decision

The proposed mechanism is a loop between one model and a database. The model searches, reads what came back, and decides what to do next. It can search again, set metadata filters during the process or constrain the search before eventually returning a ranked list of documents. The results of an unsuccessful query can therefore change the next retrieval action. That is the capability missing from the fixed pipeline.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 300 seconds](https://ai.engineer/api/talk-essay-media/ce998f575996424459399a5bb6451f7243780a46ee5640bfaf5d8046be245065)

Let the model decide what to search next

The loop also lets effort vary with difficulty. A difficult question can receive more computation and more iterations than an easy one. Rumpf wants the model to discover useful strategies rather than follow a prescribed sequence. His description of removing baked-in decisions applies to the model’s search behavior: the system still supplies actions such as searching and filtering, and training still defines rewards. The learned policy decides how to use those opportunities and when to return results.

Search results can change the next search
The model reads retrieved material before choosing another query or filter. When it decides the search is sufficient, it returns ranked results.
- Search model — Chooses searches and metadata filters; decides whether to continue.
- Database — Responds to the model’s searches.
- Read results — Retrieved material informs another iteration.
- Ranked documents — The final output of the search process.
- Search model → Database (Search with chosen constraints)
- Database → Read results (Return results)
- Read results → Search model (Choose the next action)
- Search model → Ranked documents (Finish searching)

### Search offers a checkable reward and a repeatable environment

[5:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=306s)

Reinforcement learning needs feedback that distinguishes successful attempts from unsuccessful ones, plus enough opportunities to attempt the task. Rumpf identifies two useful properties of search:

- **Verifiable outcomes:** for a training question, the system can check whether the model found the correct document. The retrieval outcome supplies a concrete reward target.
- **Repeatable attempts:** the model can try the task many times in a search environment. Rumpf describes thousands of attempts per second during a training run.

This makes it possible to train search behavior through repeated outcomes rather than write a rule for every retrieval situation.

Specialization supplies another route to efficiency. Rumpf argues that strong search does not require all the capabilities of a general-purpose language model. He compares CPUs, GPUs and ASICs: a general device may be able to do the work, while a specialized one performs it much more effectively. The analogy motivates narrowing the model’s job. It does not specify which language-model components can be removed or what architecture produces the savings.

### Reward successful retrieval and efficient searching

In an earlier training task, Rumpf reports that search quality increases predictably with compute, alongside vector-only and reranker-only baselines. Training can also include latency rewards, so the model has a reason to find documents efficiently as well as correctly. The desired behavior is discovered through training rather than imposed as a fixed sequence of queries.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 372 seconds](https://ai.engineer/api/talk-essay-media/cfd31ab6d778db657bb8ca60461d80c05cc9369ddfc1a73b03e8ee50880d025c)

Search offers a checkable reward and a repeatable environment

Rumpf says the team has not yet seen a ceiling to this approach, but he explicitly leaves open whether one exists. Continued improvement in the observed training regime supports further scaling experiments; it does not establish unlimited improvement across all domains.

### A faster specialist also keeps the main context cleaner

[6:35](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=395s)

Rumpf returns to the initial comparison with **[SID-1](https://www.sid.ai/research/sid-1)**, including a configuration with parallel execution. He reports approximately 20 times faster searching: around five seconds on average instead of around two minutes, at roughly one hundredth of the cost of a frontier model on the task. Both the times and the speedup are rounded comparisons. Benchmark composition, retrieval quality and execution configuration matter when deciding whether those gains will transfer to another workload.

The specialist still takes longer than a vector-and-reranker pipeline. Rumpf expects that gap to narrow, but classical retrieval latency remains a future target. The decision therefore involves three measures together: retrieval quality, latency and cost. A large improvement over frontier-model searching does not erase the simpler pipeline’s speed advantage.

### Keep exploration in the search agent’s context

The production example adds a second reason to delegate. When the main agent searches directly, it encounters both useful documents and bad results. All of that exploratory material enters the context it carries into the rest of the task. A search sub-agent can do the reading, thinking and iteration separately, then return selected results. The main agent gets the useful output without carrying the full search history.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 475 seconds](https://ai.engineer/api/talk-essay-media/901662ad5e54bae73efadcba54008b63b4910461ed859a96cccd32a490253659)

A faster specialist also keeps the main context cleaner

Rumpf expects cleaner context to make the main agent more likely to be correct. That benefit depends on the search specialist finding and selecting the right documents; a separate context window cannot guarantee that its output is good. The mechanism reduces exposure to unsuccessful exploration, while retrieval quality determines whether the material that replaces it actually helps.

Delegation also moves the search work associated with that initial 30–50 percent token share onto the cheaper model. This is a saving within the task: making the search phase about 100 times cheaper does not make the complete task 100 times cheaper, because the main agent still performs the work after retrieval. The economic benefit grows with the amount of search the task would otherwise require.

What reaches the main agent’s context
Direct searching exposes the main agent to exploratory results. Delegated searching keeps that exploration in the specialist and returns selected documents.
- Main agent searches — Reads useful and unsuccessful results while exploring.
- Main context accumulates exploration — Good and bad retrieved material enter the context window.
- Main agent delegates — Hands the search phase to a specialist.
- Search sub-agent explores — Searches, reads, thinks and iterates in its own context.
- Main context receives selected results — Returned documents replace the full exploratory history.
- Main agent searches → Main context accumulates exploration (Retain retrieved material)
- Main agent delegates → Search sub-agent explores (Request search)
- Search sub-agent explores → Main context receives selected results (Return selected documents)

### The destination is knowledge inside organizations

[8:28](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=508s)

Rumpf closes by forecasting arbitrarily good search across domains as reinforcement learning scales. He also expects faster and cheaper models to make iterative search practical in voice and e-commerce. These are proposed destinations beyond the reported results. His earlier acknowledgment that the method’s ceiling is unknown remains an important qualification on the prediction of unlimited improvement.

The larger opportunity, in his view, is knowledge work over information that public web search cannot supply. He argues that the web contains only a small part of the data available and uses J.P. Morgan as his example: the knowledge needed to run the organization sits deep inside its databases. His claim concerns the operational knowledge held within an organization, rather than information a public search engine can readily retrieve.

![Selected presentation frame from Where RL Will Take Search — Maximilian-David Rumpf, SID.ai at 545 seconds](https://ai.engineer/api/talk-essay-media/669eb2c0a5aec97f3d689016c66170a99924807da4fd9d68b63dbbcb179f5b6b)

The destination is knowledge inside organizations

That ending gives the search specialist a concrete purpose. The main agent needs the right context before it can do useful work, and much of the relevant context may be inside an organization. A model that can inspect results, refine its search and return selected documents could make more of that knowledge usable without forcing the main agent to spend its own expensive context on every attempt.

## Key takeaways

- A reranker can recognize inadequate candidates without being able to obtain better ones. An iterative search model can use retrieved material to choose another query or metadata filter.
- Search supports reinforcement learning through checkable document outcomes and repeated training attempts. Latency rewards can encourage efficient behavior alongside successful retrieval.
- Rumpf reports around five seconds per search instead of around two minutes, at roughly one hundredth of frontier-model search cost on the task. The specialist still has higher latency than a vector-and-reranker pipeline.
- A search sub-agent keeps exploratory material out of the main agent’s context and moves retrieval work onto a cheaper model. Its document selection still determines whether the returned context is useful.
- Internal organizational knowledge is a central opportunity for better retrieval. Unlimited improvement and future voice or e-commerce applications remain forecasts; the method’s ultimate ceiling is unknown.

## Resources

- [Introducing SID-1](https://www.sid.ai/research/sid-1) — SID’s overview of the retrieval model discussed in the recording, with benchmark recall, latency and cost comparisons.
- [Training SID-1 to beat GPT-5 at search with 1k+ QPS RL](https://turbopuffer.com/blog/reinforcement-learning-sid-ai) — A companion explanation by Max Rumpf and Sam Dauncey covering training rewards, learned tool choices, parallel searches and the backend demands of large-scale rollouts.
- [SID.ai](https://www.sid.ai/) — The search research lab Rumpf introduces, with links to its research and model waitlist.

## Slides

### SID.ai

Founder and CEO Maximilian-David Rumpf introduces SID.ai as a stealth-ish AI lab focused on search.

[0:17](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=17s) [SID.ai describes itself as a stealth-ish AI lab for search.](https://assets.ai.engineer/sites/aiecode2025/sha256/2bcafeb35959e55bb28537e921e8bbeab031331bb6685aea303d835238cbec52.webp)

SID.ai

• Stealth-ish AI lab for search

### SID.ai

Rumpf introduces SID.ai as a stealth-ish AI search lab backed by researchers from several AI companies and Jeff Dean. The lab is hiring.

[0:26](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=26s) [SID.ai describes its search lab, its backers, and that it is hiring.](https://assets.ai.engineer/sites/aiecode2025/sha256/2aebbda2fb60716691e907a85dac34fad22392bf8b1a48ad490d8aa9ab6539cd.webp)

SID.ai

• Stealth-ish AI lab for search
• Backed by researchers from OpenAI, Anthropic, Deepmind, xAI, Applied Compute, Cursor, Cognition, Prime Intellect, and Jeff Dean
• Hiring!

### Agents are a new paradigm for search

Rumpf says agents are twice as likely to find the right document, but search costs $.1–$1 per question and takes minutes versus milliseconds. Agents spend 30–50% of their tokens finding context for tasks.

[1:03](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=63s) [Agents can find documents more reliably but are expensive and slow; the slide says they spend 30–50% of tokens searching.](https://assets.ai.engineer/sites/aiecode2025/sha256/94d560065bcb09f99838b97c98c79a38101c5cc2c3cb048628aa1a0c2cb00dff.webp)

Agents are a new paradigm for search

• Much higher quality results → twice as likely to find the right document
• Incredibly expensive → $.1-$1 / question
• Slow → minutes vs. milliseconds

Agents spend 30-50% of tokens on searching (docs, code, web)

### Idea

Delegate search to a trained subagent, then ask how much cheaper and faster reinforcement learning can make it.

[1:31](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=91s) [Use and train a search subagent, asking how reinforcement learning can make it cheaper and faster.](https://assets.ai.engineer/sites/aiecode2025/sha256/4601de78e0652527bff41eb57c41ae20419a4d0c18ab222e74cf8092dad9f933.webp)

Idea

1. Use a subagent for search
2. Train a model to be a great subagent

How much cheaper and faster can we make this with RL?

### Benchmarks

The target combines high recall with low latency. Several model points trade longer times per question for higher recall, while vector-only and reranker sit near 0.45 recall.

[1:39](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=99s) [Chart comparing recall and seconds per question for search approaches and models, with a high-recall, low-latency target.](https://assets.ai.engineer/sites/aiecode2025/sha256/410bc6e5e893d9751ea5dd4943b2550ac9e306d93396f0691bd35daa97c91479.webp)

Benchmarks

Target

Y-axis: Recall
Y-axis ticks: 0.45, 0.55, 0.65, 0.75, 0.85
X-axis: Seconds per question
X-axis ticks: 0, 20, 40, 60, 80, 100, 120, 140, 160

Vector only
Reranker
GPT-5.1 (none)
Sonnet 4.5
GPT-5 (auto)
Gemini 2.5 Pro
GPT-5.1 (low)
GPT-5.1 (high)
Gemini 3 Pro

### Benchmarks

Reranker and vector-only methods sit at low recall; several models reach higher recall with more seconds per question. The target aims for high recall and low latency across five question domains.

[1:43](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=103s) [Recall versus seconds per question chart, with questions spanning legal, finance, knowledge bases, science, and email.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f2a9eb1a29527e7d2eecfca75b4d4bb5888e799260d6b1d1e2e8c6ced21182f.webp)

Benchmarks

Target

Y-axis: Recall
Y-axis ticks: 0.45, 0.55, 0.65, 0.75, 0.85
X-axis: Seconds per question
X-axis ticks: 0, 20, 40, 60, 80, 100, 120, 140, 160

Reranker
Vector only
GPT-5.1 (none)
Sonnet 4.5
GPT-5 (auto)
Gemini 2.5 Pro
GPT-5.1 (low)
GPT-5.1 (high)
Gemini 3 Pro

Questions across legal, finance, knowledge bases, science, email

### Classical search

A fixed pipeline sends each question through optional rewriting and reranking. A reranker may recognize poor results but cannot take further action, leaving a long tail of failures on unexpected questions.

[2:41](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=161s) [A search pipeline with optional query rewriting and reranking; bullets describe fixed design decisions and compute per question.](https://assets.ai.engineer/sites/aiecode2025/sha256/60b41e4a2f05810b7f7316e511d472694daa003a1d89431b57e9888a6b843826.webp)

Classical search

question → LLM rewrite → search backend → reranker → results
LLM rewrite: optional
reranker: optional

• Chained, locally-optimized models
• All decisions are baked in at design time
• Fixed compute per question: Can’t react to unexpected questions

[3:02](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=182s) [A search pipeline from question through optional rewriting, search, and reranking to results, followed by limitations of fixed pipelines.](https://assets.ai.engineer/sites/aiecode2025/sha256/36a8aa003932ec7b11e34c445fdb6ec4cedf74b36fdbc8611732cfb78267f833.webp)

Classical search

question → LLM rewrite → search backend → reranker → results
LLM rewrite: optional
reranker: optional

• Chained, locally-optimized models
• All decisions are baked in at design time
• Fixed compute per question: Can’t react to unexpected questions

Static pipelines accrue a long tail of failures

### Machine design >> human design

Rumpf places computer vision on a spectrum from human-designed Canny edges through narrow dog-detection models to VLMs, arguing that machine design outperforms human design.

[3:42](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=222s) [A human-to-machine design axis comparing Canny edge images, dog-detection boxes, and a VLM label.](https://assets.ai.engineer/sites/aiecode2025/sha256/d9733eb62b127ed49483a5158b02242f9ae4e70aba7b159206db7887e29d5055.webp)

Machine design >> human design

Canny edge
“Box around dog”
VLM

Dog
Dog

human design → machine design

### RL for search

SID-1 exchanges tool calls and content with a search backend, iterating until it produces ranked results. The model makes the search decisions and adapts them to each question.

[4:58](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=298s) [A model exchanges tool calls and content with a search backend, then returns ranked results. It can iterate and adapt its decisions on the fly.](https://assets.ai.engineer/sites/aiecode2025/sha256/24f5929e18bd8a76c85a3d2d9e79ca75aed607bd9e31865049f64c129d97f226.webp)

RL for search

question → SID-1 → ranked results
SID-1 → search backend: tool calls, BM25, ANN, etc.
search backend → SID-1: content + metadata

• One model
• Can search, read results, iterate, search again

Model makes all decisions, adapts on-the-fly

### Search quality increases predictably with compute

NDCG generally rises as training compute increases, with a small dip between runs. Rumpf says the approach has not yet shown a ceiling, though he does not know whether one exists.

[6:18](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=378s) [A chart shows NDCG generally increasing with training compute, alongside model and retrieval baselines. The slide states “No ceiling!”](https://assets.ai.engineer/sites/aiecode2025/sha256/f0da071d1f87488986edbf4482496dcb978c3515227a098f70f88d12b501e041.webp)

Search quality increases predictably with compute

Y-axis: NDCG
Y-axis ticks: 0.40, 0.45, 0.50, 0.55, 0.60, 0.65
X-axis: Training Compute (log)

Gemini 3 Pro
Sonnet 4.5
SID-1
Qwen3-14B base model
Reranker
Vector only

No ceiling!

### Search subagents in production

SID-1 searches and iterates before reporting selected results to the frontier model, keeping poor hits out of its context. The slide claims 30–50% token savings alongside higher performance and speed.

[8:08](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=488s) [A diagram contrasts context pollution from ordinary search with SID-1 subagents that search and report results. The slide claims 30–50% token savings with increased performance and speed.](https://assets.ai.engineer/sites/aiecode2025/sha256/c42f1e780b166cd5f220e79836e5af65a85c37da405e63601eb9f449a7385cc0.webp)

Search subagents in production

user → frontier model → search → × ✓ × → search → ✓ × → answer
context pollution
context pollution

user → frontier model → SID-1 → ✓ ✓ ✓ → SID-1 → ✓ ✓ → answer

SID-1:
think
search: × ✓ ×
search: × × ×
search: × ✓ ✓
⋮
think
search: ✓ ✓ ✓
search: ✓ ✓ ×
report

SID-1:
think
search: × ✓ ×
⋮
search: ✓ ✓ ×
report

30-50% tokens saved, while increasing performance and speed

### Where will RL take search

Rumpf expects RL search to improve in quality, speed and cost, opening more knowledge work. He points to information inside J.P. Morgan’s databases as an example of valuable knowledge beyond the web.

[9:02](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=542s) [Predictions for RL search include better quality, speed and cost, and more knowledge work applications; the slide notes that valuable information also exists outside the web.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9511fd1c3bf7e9e7f38514f1227ea5312fb53ae306d0db30e6e09be33188294.webp)

Where will RL take search

• Scaling RL will give us arbitrarily good search in any domain
• RL models will become even faster (voice, ecommerce)
• RL models will become even cheaper
• Better search will unlock more knowledge work tasks
  • The web is quite small!
  • The most valuable information is not on the web! “How to run JP Morgan”

### AI Engineer World's Fair

[9:26](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=566s) [AI Engineer World's Fair logo in white on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca952cae4ddbde62c6b66796805cec6a6c4fa397973b6903c5aaa0607bb6b540.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=12s) Okay. Today I'm gonna talk about where reinforcement learning will take search. And some, some background on me. Uh, I'm the founder and CEO of SID.ai. We're a stealth-ish AI lab for search. We're backed by some pretty amazing people, uh, and we're hiring.

[0:35](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=35s) Okay. Agents are a new paradigm for search. You can now get vastly higher quality results, um, twice as likely to find the right documents. But it's incredibly expensive, um, about a hundred to a thousand times more expensive than what you'd get out of a classical search query. And it is extremely slow. You're looking at minutes and not milliseconds like TurboPuffer.

[1:05](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=65s) And what this means is that agents spend thirty to fifty percent of their tokens on searching, and this is usually at the beginning of some task. It finds the right context to then do whatever you ask it to do. And the idea here is quite simple. First, instead of having the main agent do the searching, you pass the searching to a sub-agent, and you train a model to be a great sub-agent. And the question here that we'll answer today

[1:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=96s) is how much cheaper and faster can we make this with reinforcement learning? And this is really our target. So this is a benchmark across legal, finance, knowledge bases, science, email, a bunch of different tasks, some academic benchmarks, some internal benchmarks. And this is where you currently are. You see reranker and vector-only performance at the bottom, and you can see frontier models essentially kind of like go through here

[2:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=126s) at the cost of spending many, many, you know, minutes per question. And can we get a model to kind of like be extremely accurate, um, have extremely high recall, but also be incredibly fast and cheap. And let's quickly look at classical search. Uh, this is the pipeline that many of you guys will be familiar with. A question comes in. You might have an LLM that rewrites the question. You then execute that on a search backend. You might have a reranker, um, and

[2:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=156s) you get your results at the end of the day. Um, it is essentially a pipeline of chained, locally optimized models. And all of the decisions are baked in at design time. And you expend a fixed amount of compute per question. And this one is really important. The reranker might know that the results are insufficient at answering the question, but the reranker can't take action. It can only essentially return the results even when they're bad. And what this means is that in practice,

[3:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=186s) a pipeline like this accrues a long tail of failure where unexpected questions come that, you know, the designer didn't have something for. And in practice, this usually means people add lots of edge cases, um, to essentially fix these. But of course, you can't design infinite edge cases, and you can't add infinite tweaks. And so the strategy here is one that we've seen before. Machine design outperforms human design.

[3:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=216s) Um, and we saw this in computer vision where you had your, you know, primitive edge detection algorithms. You then had the box around a dog generation of models, um, that were very good at this like very narrow task and locally optimized for it. And then you had VLMs that were extremely good at all parts of the search pipeline. You saw this again with chess, with IBM Deep Blue being largely a collection of human-written rules, um, Stockfish bridging the two, and then AlphaZero and MuZero essentially

[4:05](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=245s) completely just putting it all inside of the model. And we're gonna see something similar happen to search, where we have our existing algorithms like BM25 and PageRank. Um, then we had an evolution from that with small models that did some tasks very well, like vectors and rerankers. And now essentially this new paradigm of pure RL where we actually don't bake any design decisions into the model. And what this looks like in practice is, um,

[4:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=276s) something like this. Um, so you have one model. Uh, it goes back and forth with the database. It can search, it can read results, it can iterate, uh, it can search again until it is happy. It can set metadata filters on the fly. It can constrain its search. It can try as much as it wants. Uh, and in the end, it produces a ranked list of results. Uh, and what you get is a model that makes all of the decisions and can adapt to any question on the fly

[5:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=306s) and, for example, use much more compute if a user asks a very difficult question. And what helps us here is that search is verifiable, um, and reinforcement learning needs rewards that are verifiable and grindable. Verifiable here means for a given question, did you find the correct document? And we can design this and tell this quite easily. Um, and is there an environment where the model can attempt this question loads and loads of times? And in practice for us this means, uh,

[5:35](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=335s) thousands of times per second during a training run. Uh, and the second part is kind of like, can we turn the models that are currently very general and very general purpose into something that is much more specialized? And it turns out we don't actually need most of the parts of a language model to be extremely performant at search. Uh, and similarly with like CPUs and GPUs and ASICs, um, a CPU in theory can do anything that a GPU can do. But you would never want to use a CPU to do LLM

[6:05](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=365s) inference, for example, um, because the much more specialized version is much more effective. And this really makes search an ideal target for RL. Um- And this is what happens when you train a model on this task. And so again, here we added the vector and reranker-only baselines. This is of an earlier task. And what we see is that search quality increases very predictably with compute. And we can mix in other rewards like latency, um, and different kind of like retrieval strategies to make it even more performant.

[6:35](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=395s) Importantly, we don't really tell the model what to do. Much like in AlphaZero in chess, we want it to discover its own strategies and its own tricks to essentially search well. Uh, and we don't know whether this method has no ceiling, but we're definitely not yet seeing a ceiling to this approach. And these are the results. Uh, so this is the same chart as before, and this is SID-1 and then SID-1, um, with some parallel execution on the left-hand side. And so what this ends

[7:05](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=425s) up meaning is you get-- you're about twenty times faster, so instead of taking around two minutes, you take around five seconds on average. And it's about a hundred times cheaper than using a frontier model for this task. Uh, there's some more detail here, but like yeah, the cost and kind of like speed are just completely incomparable. Uh, you can-- Yeah, um, it's not quite at the latency of a vector and reranker pipeline, but in practice, we think we can get there quite quickly.

[7:36](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=456s) And how does this look like in production? So this is a usual kind of like agent execution trace. The agent does some searching here. It finds some good stuff. It finds some bad stuff. Um, but all of the bad stuff that it finds is essentially polluting its own context window. And what we can instead do is use, um, a sub-agent here that does all of the searching and thinking and iterating for the main agent, and the main agent only ever sees great results. And this means that the main agent sees more good stuff, which means it's more likely to be correct. And it

[8:06](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=486s) is also, um, extremely cost-effective, uh, where those thirty to fifty percent tokens that were earlier used by the main agent to do searching can now be passed off to this just, you know, hundred X cheaper search sub-agent.

[8:23](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=503s) Uh, and

[8:28](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=508s) where will this take us? Scaling RL will give us arbitrarily good search in any domain.

[8:35](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=515s) And RL models will become even faster, which will allow them to be used in things like voice and e-commerce. They'll become even cheaper than we are currently. Um, so the charts that you saw there, but like I think we can move even further. Uh, and better search will unlock more knowledge work tasks. Uh, the web is actually quite small, uh, in comparison to the entirety of data that is there-- uh, that is out there. Uh, and the most valuable information is

[9:05](https://www.youtube.com/watch?v=iJVxxxHM_Oc&t=545s) not on the internet. For example, how to run J.P. Morgan is nowhere on the web, but it is deep inside of the databases at, at J.P. Morgan. That's it for me. Thank you.
