[
  {
    "startMs": 12940,
    "text": "Okay. Today I'm gonna talk about where reinforcement learning will take search. And some, some background on me. Uh, I'm the founder and CEO of SID.ai. We're a stealth-ish AI lab for search. We're backed by some pretty amazing people, uh, and we're hiring."
  },
  {
    "startMs": 35940,
    "text": "Okay. Agents are a new paradigm for search. You can now get vastly higher quality results, um, twice as likely to find the right documents. But it's incredibly expensive, um, about a hundred to a thousand times more expensive than what you'd get out of a classical search query. And it is extremely slow. You're looking at minutes and not milliseconds like TurboPuffer."
  },
  {
    "startMs": 65660,
    "text": "And what this means is that agents spend thirty to fifty percent of their tokens on searching, and this is usually at the beginning of some task. It finds the right context to then do whatever you ask it to do. And the idea here is quite simple. First, instead of having the main agent do the searching, you pass the searching to a sub-agent, and you train a model to be a great sub-agent. And the question here that we'll answer today"
  },
  {
    "startMs": 96000,
    "text": "is how much cheaper and faster can we make this with reinforcement learning? And this is really our target. So this is a benchmark across legal, finance, knowledge bases, science, email, a bunch of different tasks, some academic benchmarks, some internal benchmarks. And this is where you currently are. You see reranker and vector-only performance at the bottom, and you can see frontier models essentially kind of like go through here"
  },
  {
    "startMs": 126000,
    "text": "at the cost of spending many, many, you know, minutes per question. And can we get a model to kind of like be extremely accurate, um, have extremely high recall, but also be incredibly fast and cheap. And let's quickly look at classical search. Uh, this is the pipeline that many of you guys will be familiar with. A question comes in. You might have an LLM that rewrites the question. You then execute that on a search backend. You might have a reranker, um, and"
  },
  {
    "startMs": 156040,
    "text": "you get your results at the end of the day. Um, it is essentially a pipeline of chained, locally optimized models. And all of the decisions are baked in at design time. And you expend a fixed amount of compute per question. And this one is really important. The reranker might know that the results are insufficient at answering the question, but the reranker can't take action. It can only essentially return the results even when they're bad. And what this means is that in practice,"
  },
  {
    "startMs": 186180,
    "text": "a pipeline like this accrues a long tail of failure where unexpected questions come that, you know, the designer didn't have something for. And in practice, this usually means people add lots of edge cases, um, to essentially fix these. But of course, you can't design infinite edge cases, and you can't add infinite tweaks. And so the strategy here is one that we've seen before. Machine design outperforms human design."
  },
  {
    "startMs": 216040,
    "text": "Um, and we saw this in computer vision where you had your, you know, primitive edge detection algorithms. You then had the box around a dog generation of models, um, that were very good at this like very narrow task and locally optimized for it. And then you had VLMs that were extremely good at all parts of the search pipeline. You saw this again with chess, with IBM Deep Blue being largely a collection of human-written rules, um, Stockfish bridging the two, and then AlphaZero and MuZero essentially"
  },
  {
    "startMs": 245940,
    "text": "completely just putting it all inside of the model. And we're gonna see something similar happen to search, where we have our existing algorithms like BM25 and PageRank. Um, then we had an evolution from that with small models that did some tasks very well, like vectors and rerankers. And now essentially this new paradigm of pure RL where we actually don't bake any design decisions into the model. And what this looks like in practice is, um,"
  },
  {
    "startMs": 276080,
    "text": "something like this. Um, so you have one model. Uh, it goes back and forth with the database. It can search, it can read results, it can iterate, uh, it can search again until it is happy. It can set metadata filters on the fly. It can constrain its search. It can try as much as it wants. Uh, and in the end, it produces a ranked list of results. Uh, and what you get is a model that makes all of the decisions and can adapt to any question on the fly"
  },
  {
    "startMs": 306120,
    "text": "and, for example, use much more compute if a user asks a very difficult question. And what helps us here is that search is verifiable, um, and reinforcement learning needs rewards that are verifiable and grindable. Verifiable here means for a given question, did you find the correct document? And we can design this and tell this quite easily. Um, and is there an environment where the model can attempt this question loads and loads of times? And in practice for us this means, uh,"
  },
  {
    "startMs": 335840,
    "text": "thousands of times per second during a training run. Uh, and the second part is kind of like, can we turn the models that are currently very general and very general purpose into something that is much more specialized? And it turns out we don't actually need most of the parts of a language model to be extremely performant at search. Uh, and similarly with like CPUs and GPUs and ASICs, um, a CPU in theory can do anything that a GPU can do. But you would never want to use a CPU to do LLM"
  },
  {
    "startMs": 365560,
    "text": "inference, for example, um, because the much more specialized version is much more effective. And this really makes search an ideal target for RL. Um- And this is what happens when you train a model on this task. And so again, here we added the vector and reranker-only baselines. This is of an earlier task. And what we see is that search quality increases very predictably with compute. And we can mix in other rewards like latency, um, and different kind of like retrieval strategies to make it even more performant."
  },
  {
    "startMs": 395800,
    "text": "Importantly, we don't really tell the model what to do. Much like in AlphaZero in chess, we want it to discover its own strategies and its own tricks to essentially search well. Uh, and we don't know whether this method has no ceiling, but we're definitely not yet seeing a ceiling to this approach. And these are the results. Uh, so this is the same chart as before, and this is SID-1 and then SID-1, um, with some parallel execution on the left-hand side. And so what this ends"
  },
  {
    "startMs": 425780,
    "text": "up meaning is you get-- you're about twenty times faster, so instead of taking around two minutes, you take around five seconds on average. And it's about a hundred times cheaper than using a frontier model for this task. Uh, there's some more detail here, but like yeah, the cost and kind of like speed are just completely incomparable. Uh, you can-- Yeah, um, it's not quite at the latency of a vector and reranker pipeline, but in practice, we think we can get there quite quickly."
  },
  {
    "startMs": 456440,
    "text": "And how does this look like in production? So this is a usual kind of like agent execution trace. The agent does some searching here. It finds some good stuff. It finds some bad stuff. Um, but all of the bad stuff that it finds is essentially polluting its own context window. And what we can instead do is use, um, a sub-agent here that does all of the searching and thinking and iterating for the main agent, and the main agent only ever sees great results. And this means that the main agent sees more good stuff, which means it's more likely to be correct. And it"
  },
  {
    "startMs": 486460,
    "text": "is also, um, extremely cost-effective, uh, where those thirty to fifty percent tokens that were earlier used by the main agent to do searching can now be passed off to this just, you know, hundred X cheaper search sub-agent."
  },
  {
    "startMs": 503660,
    "text": "Uh, and"
  },
  {
    "startMs": 508060,
    "text": "where will this take us? Scaling RL will give us arbitrarily good search in any domain."
  },
  {
    "startMs": 515860,
    "text": "And RL models will become even faster, which will allow them to be used in things like voice and e-commerce. They'll become even cheaper than we are currently. Um, so the charts that you saw there, but like I think we can move even further. Uh, and better search will unlock more knowledge work tasks. Uh, the web is actually quite small, uh, in comparison to the entirety of data that is there-- uh, that is out there. Uh, and the most valuable information is"
  },
  {
    "startMs": 545800,
    "text": "not on the internet. For example, how to run J.P. Morgan is nowhere on the web, but it is deep inside of the databases at, at J.P. Morgan. That's it for me. Thank you."
  }
]