Version-Controlled Search: The Web as It Was — Ryan Ahern, Exa

Welcome everyone. Looking forward to, um, giving you all a, a introduction into one of our newest features in Exa, which is version-controlled search. Uh, so I'm Ryan. I'm a forward deployed engineer here at Exa. Um, and today I'm gonna give you an overview of two things. We only have a very short amount of time. So the first one, I just wanna motivate what is Exa, what is agentic web search, just because you kind of need to believe in the dream of that to understand why you'd want to use it to do the second thing I'm gonna talk about, which is how to backtest financial models using

that. So first, just to understand where I'm at in the room. Who here has heard of Exa before or used it? Okay, like a good show of hands. Who here has ever used like Google back in the day to like do some sort of work or personal use, just like typed into Google to do some sort of search? Hopefully like everyone in the room, most people. Had pretty good market share, you might have heard of it. Last one would be, how many of you now no longer really use Google? You use, uh, ChatGPT or Claude or maybe even Instinct or Muse if you're like at the cutting edge to actually do your searches. Yeah,

most people, right? Um, so we're gonna connect all of these things together. Um, this last point where a lot of people now are like, "Hey, I don't even really use, um, desktop w- search anymore. I kind of ask an agent to do this," is something that we see more broadly in the market, right? And the idea is that like, um, agents tend to do a pretty good job at understanding how to search because they're allowed to try multiple, multiple times. They kind of don't get tired. They tend to be quite agreeable and work very, very hard. So when you say like, "This is the thing that I want," it's gonna go out and it's gonna do

many, many different searches in order to satisfy like your objective. And the other thing here is that like they tend to search different to humans. So as a human, you might put in like a few keywords for the thing and then hope that you get the best results like in the first couple, and otherwise, like maybe you try like a different query and then you kind of give up. Whereas agents will be very verbose. Um, they, they tend to describe what they want to search in great detail and will execute many searches in order to understand the space more fully before they respond. Um, and that means that they do way more searches,

um, on, on a per task basis. So generally the idea is that like, hey, nowadays most humans, you're gonna be using agents to kind of broker your search tasks. Um, and that's where Exa kind of plugs in. So, uh, when you're working with um, like many, many agentic systems, so whenever Cursor wants to go and search the latest documentation to help its coding agent complete its trajectory, whenever you're like talking to Gamma and you're like, "Hey, it'd be great if I had just this like SEC data from like Apple in like twenty eighteen to complete

my slide, like that would make it brilliant," it's gonna use Exa to go out, search the full web, so we have our own craw- uh, crawling infrastructure, our own database, our own embedding model, our own retrieval architecture, to get exactly what it needs to answer its query. Um, this is like a couple of the logos of like the bigger companies but, um, because this is AI engineer, like here are a few like kind of archetypes that we work across the financial space. I work with a lot of hedge funds, uh, asset managers, sovereign wealth funds, VCs across like a bunch of different tasks because they also need like the

latest information. So if you're a hedge fund or you're gonna be trading on information, you're looking in the market for alpha, right? And for information that allows you to understand h-how you could, uh, position your, um, your organization better. If you're a, um, uh, an insurer, you're probably understanding like what counterparties you're working with or who you're insuring to make sure that ultimately you're making the determination. So you wanna make sure you're making it b- on the best information and that your risk models are the most accurate. Or even if you're in VC and you're like, "I need the hottest startup. I need to know

the, uh, next Instinct before they blow up," and they're not necessarily gonna be in like some structured data set. Like, "I need to go and find them wherever they exist in the web," um, that's what they come to Exa to kind of support them for. Um, and then the, kind of to wrap up the last part on like why Exa, uh, and the reason why I think now you'd be like, "Okay, I'm, I'm building agents or I'm interested in agents and I definitely need agentic web search. Why Exa?" Um, and that's because like we're a lab, um, in the sense that we're very, very interested in the search market, um, and in the

ways in which agents search, and we're hyper-focused on that problem. There's three like core tenets, and then we'll talk about how that goes into versioning to segue into the next part of the talk. But first and qui- obviously is like quality, right? So you wanna make sure that when the agent does that, um, that tool call and the retrieval, it gets what it needs. 'Cause the fewer tool calls that it needs to make, the more cost-effective on your inference, and also just like the more likely it is to succeed on its overall trajectory. You're not gonna bring in something that like misaligns it. Um, uh, we're really, really interested

in the quality space. We, uh, recently released like a new benchmark, um, which we think more accurately assesses the, um- The, the performance of, like, the search tools separately to the inference because these two things can be confounded, right? It's like what agent or what model is calling the search and what search is connecting to what model. So that you can kind of have a look at that to see our latest thinking there. Secondly, on cost. So it's not just the line item of the tool call, right? Like, how much does one search cost? Because agents can broker many different searches. Um, so the

idea is both like, hey, what's the value to this to complete your trajectory, but also, like, what's the impact of this on the total cost of that agent completing its task? 'Cause that's actually the thing that you care about, right? You're generally less opinionated about one tool call. It's like, how much did this overall agent take? Uh, and that's why, like, we're inter-- we, we worked on things like, say, Dynamic Highlights, which means that for every single result, we're not r- we're not just returning full pages. Uh, we're not even just taking one page, which is our previous highlight features, and figuring out what are the exact

right snippets on that page that we need to send back to the context window to complete the trajectory. We now do that across all of the search results, which means that you can actually get higher quality results at ninety-five percent fewer tokens than you would at full page text. We have a blog about this. It's very interesting. But basically, it means you'll save a lot of money downstream on all of the input tokens you'll be spending on inference for the agent to complete its trajectory. The last one on latency. So because we own our own infrastructure, we can do very, very low latency search.

Um, I'm just gonna touch on the other side of the spectrum, which is, like, very long-running deep research. So we have Exa Agent Ultra, which we released, um, uh, like, not too long ago, maybe a few weeks ago, uh, which is, like, state-of-the-art on, um, a lot of benchmarks around, like, very, very long-running, um, search. And that's because we can do a lot of things under the hood. We have-- Obviously, we own the search harness, but we can also be opinionated about that, how that interacts with the agentic harness to do these, like, very, very long-running research tasks. So I highly recommend you check those things out.

Um, this is just a segue to be like, okay, Exa really cares about this space. I think that their search is going to be good to motivate the next part, which is really interesting and the topic of this talk, which is versioning. So just recently, relea- we released Exa Snapshot. Uh, it's four hundred billion-plus snapshots of the web, um, and you can search them by date. Um, and that's really, really interesting in the, um, in the financial services industry. We'll get into that. But first, I'm just gonna talk a little bit about, like, what this basically means. So, uh, we're, we're a search company,

so we're constantly s- uh, searching across the web for the latest information that we can add to our index. Um, and every single time we do it, we check it for f- we're, we're checking for freshness because ultimately you wanna have the latest live up-to-date information at the point at which you're searching. But be- because we're doing all of those checks, it means that we can capture a version of those URLs at the point where we checked it and have, like, a versioned history. Um, and you can access that in two ways. One is like, I know the URL I wanna find as it was at or before a given date. And so you can give us a URL,

and you say, "This is the date that I want it. I want it at that date or before, and serve it to me as you would have if I had requested it at that point." The second one, which is even more exciting, is this concept of versioned search. So we're a search, um, API, uh, and the idea is like, hey, what would happen if I had executed that search but at a prior point, right? So I only get results that would have been served at that point or before. Um, that has very, like, interesting implications. I'm just gonna do one, like, very small

example here. So, uh, OpenAI, um, uh, recently, or say as of like October first. So on the left-hand side here, you'll see a snapshot search as it was in October first, and on the right-hand side was a live search as of like a couple of days ago when I made this slide. Um, but, uh, at the time in, um, early October, like, there were some, uh, discussions that, like, OpenAI, uh, had a, like, um, annualized, like, run rate around, like, the seventy billion, right? Um, and that was leading

to certain ways in which the market was projecting how and where they would be valued. And then more recently, there was some... And, and that reporting was originally by A- Axios. And then, uh, more recently, there were some publications that were reporting like, actually, it's closer to fifty billion for fun accounting reasons, right? Like, it turns out that they were counting gross revenue when actually, like, some of their revenue was coming from cloud providers. And technically, the way that you should account for that would be that there would be a vendor pass-through, so net annualized revenue is better. And that's how-- what some, what some other players were reporting, et cetera, et

cetera, right? But basically, it meant that the, the, the net annualized run rate was, uh, like, kind of down forecast from seventy billion down to fifty billion, right? Now, this is just like a financial thing that some of you may or may not find interesting. But the so what for versioned search is like, let's say you're in the interest of creating a product or a solution or an agent that is made to-- that, that will-- that, that's job is to, to find these things, right? You can now go back and be like, okay, before everyone figured it out, right,

before the market and before some financial analyst published the think piece that was like slamming down on this particular thing and everyone knew about it, what information existed on the internet, right? Like, was there an SEC filing that would have revealed this thing in an accounting footnote? Like, was there some other reporting from some other vendor that maybe is like you would consider as like less reputable but actually was able to find that kind of thing and, and maybe you should consider them differently? Um, you can poten- you can, uh, potentially like trace these things before like the rest of the market found it.

But most importantly, like not only can you test that, but you could backtest that, right? You could then adjust the ways in which you're setting up your model. You can adjust the ways that you're setting up your agent to be able to be like, could it have found this discrepancy? Um, and more importantly, could it have found that discrepancy in a way that is generalizable, right? Let's like create a set of, um, like, you know, like fifty sample sets, see if it can perform against that, check it against the holdout and be like, "Okay, I think that my model is now gonna be pretty good at

finding this kind of feature." And that's something that's kind of new in this, in the, in the web search space, um, that you can backtest that. Um, this is like a, like a little, um, diagram just to kind of motivate the idea. But at any, like, live search that you do with, like, any existing search product, the idea is it's live, so it has all of the information. So in this particular little example, um, where we're trying to predict whether a company got acquired or not. Um, in this case, if you do a live search, you're going to find, like, that the deal was announced, right? Because that

happened, it got released everywhere. You could, um, be like, "Hey, I'm trying to train a model that's able to predict whether this company got acquired or not." So I'm gonna go, like, one day before the announcement, seven days before the announcement, a month before the announcement, and see whether I could or could not predict it, right? Uh, and you can build a set of those companies, um, and basically train your model, um, or whatever system it is that's trying to do this prediction and see, like, was it able to reliably, um, like, predict this particular feature. Um, so

I'm gonna spend a little bit of time here just talking about how different systems are currently being backtested with, uh, Snapshot. So there's like three kind of archetypes here. The first one is I'm operating like a very traditional model. Um, maybe it's ML, maybe it's even more simple than that, like a decision tree or something. Um, and all I wanna do is just gather unstructured data from the versioned web into some sort of structured set of information. So it could be something like software prices, it could be some sentiment

analysis, it could be like the terms and conditions of a particular site. Whatever it is that, like, powers your model, um, not all information is continually available. Like, if you're trying to understand software prices, um, like, you know, the, the page of that software price is not necessarily stored in a versioned, um, uh, way for that, uh, website in a way that you could reliably, um, uh, access. Or potentially, like, you have, um, some like, you know, sentiment analysis that you wanna do, but, like, that's

gonna be mired in, like, what point was that sentiment released in time, right? And you wanna be able to get, like, a reliable set of those signals. And then you can just transform it into, like, some array of signals that goes into your model, and you can predict it. That's like the th- sort of basic space. Uh, and some people are just, like, enriching data that they wouldn't have had access to, um, uh, for their existing models. So that's just like sort of like the status quo plus one. The interesting space is in like the next two areas. Um, so the first one is this idea of, like, adding an agentic feature.

Um, and the i-- a-and, and here it's like, okay, well, I have this existing model. It performs pretty well, but I have an idea that, like, if I add one or two, like, agentic features to this model, it would be able to perform better. So let's say something like an adverse media scan. Necessarily, like adverse media is going out into the web, searching across it for all of the information that exists, potentially doing it in an agentic way, right? Where you're doing multiple searches, you're exhausting multiple different trajectories, and then you're gonna take all of that and

then condense it into some sort of structure. Um, uh, another example here would be, let's say, like equity research in much the same way. Like, hey, I wanna go out, I wanna look at the full universe of companies. I wanna find like, potentially all of their like, like, estimated, uh, like rack revenue or headcount or all of the features that matter. Um, and the idea is that you'd be using something like Exa Search at the point at which you're doing a prediction live to go out and gather all of that information to then send into your, um, traditional model.

And you can now use like Exa Snapshot to backtest those particular features. So you can go back and be like, "Hey, if I was to do this, like adverse media scan like one month before, or maybe I've already like onboarded a bunch of people into my portfolio like four months ago, what would I have said at the point where they were going to enter my portfolio? And would-- how is that tracked against today?" Um, and that kind of motivates, like, especially with a lot of financial services customers, like you don't now just have to believe that, like search will be better. You can actually go back and backtest

that against your existing processes and say, "Hey, it's better." Um, so that's like the maybe the next evolution. It's like one step plus, plus. Um, on the far right-hand side here, um, is kind of like maybe the future. So there's some like startups that are working in this prediction space, and the idea here is that the full sort of like simulation or prediction is being managed by an agent. So in the middle, there's like an ML model that's maybe doing a prediction or some algorithm or decision tree. On the right-hand side, like it is like an

agent that's just going off and doing a bunch of things. It could be searching internal knowledge, it could be accessing data feeds, it could be doing whatever it is. But one of the things that it may do that is valuable is to tool call, uh, and access the web. And the idea is like, now you don't have to, um, try and create some internal corpus that tries to represent what it would be like to access the full web. You can just be like, "Hey, I want to make sure that I can control that feature, so I know that future information is not just gonna leak directly into my model."

Like, I'm trying to predict company acquisitions through some very, very insane, like agentic process that's accessing everything. You can now be like, "All right, let's just make sure that's not just gonna have one web search that finds that it was acquired, comes back and predicts it with like one hundred percent accuracy." Um, so that's kind of the, the sort of right-hand side there. Um, I'm just gonna give like a little example of like the middle, um, just to round off, um, this, this, this kind of discussion. Um, so, uh, yesterday we ran like, um, a kind of extended workshop on Snapshot.

Uh, and we had a bunch of-- uh, around about a hundred people in the room that were kind of, uh, running through this, the same example we had before about trying to like predict company acquisitions. Um, one thing that was kind of interesting is that, um, there-- this was like some-- like one of the example features from the workshop from one of the people, um, that, uh, that, that got a relatively high score. But the thing that was interesting here is like, um, yeah, the, uh-- just to, just to motivate. So the first one was this idea that like, hey, within the past year, did an investor or a significant shareholder publicly call

for a supported sale of the company as a whole? And the second one, which is the interesting one, is like, hey, did this company's share price, um, was it at least sixty percent lower its high share price during the fa- past five years? Which the idea is like it's kind of dipped down. Um, the last bit that's interesting is like, hey, is the company named Dominion? Now, this is a little bit of overfitting for people that know like backtesting. Um, like, a- and the idea here is like this could be something that like an agent could do, right? If it's trained on like the live set, um, it can potentially just overfit to the training data that you

have. But the idea with Snapshot is like not only can you go back and test for these things, um, you can go back and like set up your, your, your evals and your agentic harness in such a way that you can detect these things, but you can also try and check that it generalizes. And this is something that would be very, very hard to do if you are gated by like testing live, like in production. And you can only change the ways in which these features are described over time and check against real-time, right? You might be waiting for months to try and see how and why your agentic setup,

um, actually is predictive or is overfitting in different ways. Um, that is like you can describe these like quite complex features like, um, these ones are like maybe you could imagine trying to recreate in, um, uh, in like some sort of like, uh, like non-semantic search. But you can go as crazy as you want within e- within existing like, uh, like Exa semantic search world. So you can say things like, "Hey, I want to see if there was any adverse media against like reputable VPs that are operating like, um,

uh, that are operating within VC funds that have like been on the AI Engineer podcast and they've said something bad about it," right? Like, you can go as, as hard as you want on these features that you think are going to be predictive because that's what core Exa Search allows. But then you can backtest it, right, and see if it's performative or not. Um, then I'm just going to kind of like recap. So the first thing that we talked about here is like, um, using Exa because it's quality, cost-effective agentic web search across every latency. Um, we have lots of blogs around this, so encourage you if you've not tried

Exa, um, to have a go at this. Um, because the first thing you want to know is like, "Hey, do I actually think this is going to add some benefit to my model?" The second things we talked about is like, hey, using Exa Snapshot to be able to like backtest these financial models or whichever model of risk, risk model, prediction model with these web search features in order to see like, hey, does adding it, uh, add value to, to you and your agentic processes? Thank you so much.