Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

Hello, everyone. First of all, I'm Ryan. I'm a forward deployed engineer here at Exa. Very excited to walk you all through this very cool capability. Um, we basically have perfect search for AI agents coming soon, but at least now you can version control it. Um, what I'm gonna take you through today is a few things. So first off, just a little bit like what is Exa? What is agentic web search? Why do I even care about this thing? At the end of point one, you're gonna be like, "I need agentic web search. I need it in my thing," regardless if you're doing anything financial or not. Number two, "Okay, I know I

want this thing, but why do I care about version control?" At the end of number two, you'll know why you care, and it'll be very important 'cause on number three, there'll be a little hands-on exercise where you may or may not win a prize. Hopefully, that motivates you to pay attention through points one and two. So, um, I'm gonna start off a little bit with like a show of hands just to situate. So who here has heard of Exa before? Okay, we have a good few people, but a few people that haven't heard, which is perfect. How many of you have used like a search engine like Google to do desktop research in a pr- a private-- prior role?

Like using Google just to search things around at home, at work, whatever. Hopefully like most people. Sweet. Then the last piece is going to be, hey, how many times have you now stopped using Google and instead used ChatGPT or Claude or some other agent and say, "Hey, can you find this thing out for me?" Raise your hand if like now you do that. Sweet. Great. Okay, perfect. We're gonna tie all of those three things together because, um, the exact same thing that so many people just put their hands up, that's what we're seeing right now. So AI is actually searching

a lot more than humans do nowadays. Why? 'Cause it can handle so much like back and forth, right? It can go out, it can do multiple searches, it can find what you're really looking for, can search through multiple pages and work really hard to get you the answer you need. And we're sort of seeing way more people are using AI to search, and AI searches more, and it searches more efficiently than humans. Humans are kind of lazy. You'd be like, "All right. I'm looking for this thing. I'll put in a few keywords. Hopefully, it's on page one. Otherwise, I don't care." Whereas AI systems, they really wanna satisfy these, are the-- tend to be quite agreeable.

I don't know if you found that out. Uh, and so they'll continue to searching until they find the thing. And so all of these things mean there's a lot of search. Um, and all of those tools, when you ask it a question, it's like, "Oh, I need to look outside my knowledge window. I need to consult the Internet," it will use a tool like Exa, um, and that's what we provide. So we actually provide search for some of the largest, um, like coding agents and agentic applications in the world. So whenever you're using Cursor, and Cursor's like, "Gosh, if only I knew what the docs were, I could really help," it will use Exa to go out

and find that doc, um, and bring it back to its orchestration agent, so it can answer the question. Or if you're using Gamma, and you're like, "Gee, if only I knew like what the financial returns were of like, um, I don't know, Bank of America in twenty nineteen," this slide would pop. It's able to use Exa to go out and find that information. So that's kind of where we sit. I'm also just gonna like do like a quick thing on finance 'cause this is like Wall Street meets AI. So hopefully, you're either in finance or you're interested in this. But we work across a lot of these areas. There's a particular area of interest to me.

I work with some hedge funds and sovereign wealth funds and asset managers and VCs across these topics because they want information. Either they're like in a, a hedge fund and they're looking for alpha to, to make, uh, financial decisions against. They're an asset manager, they're trying to forecast risk. They're a VC, and they're like, "Oh, I want the hottest new startup in my pipeline, and I gotta find them before like The Verge or someone like, um, um, like puts up an article about it." They all leverage Exa. Um, so hopefully by the end of this, you're

like, "Okay. Yes, um, this is a really, really interesting space." But like kinda like why Exa? Like what do we do? Um, where we're a little bit different is like we, we kinda see ourselves as a lab and doing fundamental research to really push this forward. So I'm gonna touch on like the core triangle of like quality, cost, and latency really quick, just so you know that you really wanna put Exa in your application. And then we'll talk about versioning, um, which will be like, "Oh, now I can actually see how and where this would be used to backtest my application." So

first, quality. You want your results to be good. We care about that so much that we built our own regenerative benchmark. Why do you wanna do that? Because search moves very quickly. You publish a benchmark and all of the search, um, providers and eventually the labs would be like, "We've gotta train on that and climb the hill." So what you really, really care about if you're a search engine that really believes in your stuff, is you wanna create the, uh, you wanna create the benchmarks that people can make for themselves against their use case and keep it up to date so that your search engine is working for

you because you wanna return the latest information. That's what the whole, uh, that's the whole reason why you're using a search provider and not just asking, uh, ChatGPT or OpenAI and it, it answering from its weights. But you're just like, "Okay. I want quality, but I want it at the lowest costs." We care about it, not just at like the search time, but also, you know, search is a lot of tokens, and tokens goes into a context window. And what does that mean? You're paying input costs. So we develop technologies to try and return just the tokens that you need, not just across one result, but because all the results so that

your coding agent or whatever application can answer the question without having to pay a lot on input token costs. Lastly, latency. Really quickly, you can go very fast. So we have like sub three hundred millisecond search all the way up to like, hey, work for like, um... I think I did one was like, find me all the video game publishers 'cause I was trying to do some eval, and it worked for twenty minutes, and it found three thousand two hundred and twelve of them and the developers that they work with. So across the full spectrum, we got you no matter what point you're trying to implement it into your architecture. And now you're saying, "Ryan, okay. That's great.

I really like Exa, but I wanna version control it," or, "Why do I wanna version control it?" Um, and that's gonna be the, uh, kind of focus of this discussion today to talk about this new capability we've released called Exa Snapshot, which allows you to take our search as it is today, um, but actually see what it would look like if you searched at a point in the past. I'm gonna motivate like firstly what this is and then why you care about it. So firstly, what is this? So, uh, we're a search engine, so we spend a lot of time looking over the web.

Uh, and whenever we find this content, what we've been doing has been putting it into our index, and when we find, like, a new version of the page, or even when we just discover the same page at a date in the past, we store a version of that. Um, and that means that, like, every URL across our full index, which is an index of the entire web with, um, I think around a hundred billion documents tracking trillions of URLs, uh, you can now go back in time. Um, and there's two ways you can do that. The first one is like, "I know a URL, but I wanna see what was

that like in the past." Um, and there's content versioning for that. So you can go in, you can provide a date, and we will show you the content as of that date or the nearest date that we had before. So you can guarantee that you have a version of that page that we would have served at that time. More broadly, you can use version search, which is not just like finding a URL, but executing a search as you would have done now, but in the past. Let's, like, dive into just, like, two very, very

small examples around this, um, and then, like, can kind of give, like, a vibe of some use cases. So first one, let's say you're, uh, interested in financial markets. Who would be interested in fin- financial markets around here? Um, and one thing you might be trying to do is figure out, like, "Hey, how can I understand the hottest and latest startups?" Maybe you guys would have heard of OpenAI. And there was some interesting news happening recently where, um, previously Axios and a few other people were reporting around, uh, OpenAI's, uh, annualized,

um, like, revenue, and there's ideas like, "Oh, it might be up near, like, sort of seventy billion." And then more recently, there's a lot of news about is it seventy billion or fifty billion? Turns out to be some, like, you know, accounting management about whether it's gross revenue or net revenue, whether you're passing through certain cloud providers' costs to you. Do you recognize that revenue or not? But if you're a person who, uh, is in the business of understanding these things, it would be really helpful if you can be like, "Hey, I wanna make sure that I'm designing a system that is

able to, uh, look at these things," right? That is able to, like... At the point where you would have done that search, when you would have had that information, like, is it tuned in such a way that it would reveal this, like, accounting footnote that could actually be, uh, very valuable to trade against? Let's say you're trying to do something, uh, arguably even more or less important, which is to understand the adverse media for, like, very important folks, uh, or maybe less important folks. Um, so, like, Ed Sheeran, he

had some, like, adverse media against him q- relatively recently. Um, but if you look back a little bit before that event, you can see, like, hey, what are a few other things that existed? Or, you know, he w- he was not really good at i-insurance. Maybe you wouldn't wanna sponsor him for some insurance product. Um, but the main thing here is that, like, at the point now where you're like, "Hey, I'm-- maybe I'm an insurance provider, and I'm wanting to do some, like, counterparty due diligence or KYB or KYC," um, or you're working, um, a, like any business and you're trying to do partnerships

and part of your flow is understanding another party, it would be good to be able to go back and be like, "Hey, I've designed this system to go out and do a bunch of web search and figure out what this person is, what this business is, how they could be classified, wh-whether we would insure against them or not." And now you're able to design that system and then with Exa go back and test it. Um, so that just gives like a few examples of sort of how and where, uh, like Snapshot can be useful in order to, to kind of like backtest these systems. So,

um, here's just like an e- an example of this. Um, a- as a note, this is important to take, uh, some, some awareness of 'cause it may help with, uh, you achieving some, some, some great excellence later. Um, but let's say you're like trying to understand something around like acquiring companies. So you may be like, "Hey, I'm trying to design my system. If you do a search right now, I could see that this company's already acquired, but I might wanna go back in the past and be like, 'Hey, what about like one day before the acquisition, one

week before the acquisition, a month before the acquisition? Would I be able to find the information to predict that?'" So any live search necessarily is kept up to date. That's what you're kind of paying for, right? That if you did a search now, you'd have the latest information. But, um, that also leaks kind of the future knowledge of that into an evaluation of the past. If you're creating some predictive system, and all it does is goes like, "Okay, I look at the news and it says that this thing was acquired, so tick, it was

acquired," one hundred percent successful true positive rate as- assuming the search is good enough, which you hope it would be. Um, what's, what's helpful with, uh, with Snapshot and, and kind of version con- version-controlled search, you can constrain like what particular information is exposed at which time. So here in this little example, I might set the snapshot date to like just before the outcome was announced. So I know that the only thing that's gonna be informing this system is, uh, information that,

um, doesn't actually leak the exact confirmation of that thing, right? And then you could tune it to do different things. Like maybe I have a feature that I really care about hiring. I wanna make sure I find the right hiring signals. Maybe I have a feature that cares about like analyzing earnings calls or, or transcripts or, or podcast interviews with the CEO. Whatever it is that's important to you, um, you're able to kind of tune that system and see, "Hey, would that have actually improved my prediction or would it not have improved my prediction?" Um, I'm gonna share

just like... Uh, and I'll spend like a, a tiny bit of time here, but just talking about like a couple of patterns that we've seen, uh, in the backtesting space recently. So, um, the first one is around this idea of like a traditional model. So... And, and the idea is here is you're just gonna gather data that you need to backtest a model that you already have. You can kind of think of this as like, "Hey, maybe I have some system, and it operates in the world, and I don't need Exa. I don't need all that agentic web search. I love my system. It's amazing."

Right? But maybe one thing that you can't do is actually go back and gather the features that you had in the past today, right? Could be that they're based on web pages, and those web pages just don't store a version that you can go back to. Um, it could be that you're trying to have some sort of, um, system that like takes like newspaper pages or some other articles and, um, you can't really trust the publish date. Very small digression here, but like publish dates of articles actually update all of the time,

um, both for like, "Hey, like we've legitimately updated it," all the way down to like, "Hey, like SEO really likes up-to-date articles, so maybe we just like update it one more time." Um, so that means that like you can't really trust that like publish date featu-feature, and it's really important, especially if you're making quite like substantial decisions around this, that you're able to trust what that feature is. So you may just decide, "Hey, I'm gonna use Snapshot. Um, I know some URLs or some websites that I care about. I'm gonna like jump back and just

gather the data that I need, um, and then put it in whatever feature I need, some array of numbers, et cetera." Exa's not even involved in the prediction, but this is like a perfectly valid and like relatively straightforward way that you might already be using some kinds of version data to backtest your system. Okay. So that's the, that's the existing space. Um, what about the future? The future of perfect search. Okay, so there's two sort of patterns here. The first one is like I'm gonna add an agentic feature to a traditional model. It's kind

of similar to what we were talking about before, but the idea is in the live model, it's actually gonna be using Exa or a tool like Exa to go out, search the web, turn all of that unstructured information into a structured signal, and then trade against it. So you could think about something like the adverse media scan like against our friend, uh, Ed Sheeran before. Um, you, you might be designing your system to be able to do some like counterparty due diligence. Um, and in the live production

system, that might just be like one arm, right? It's, uh, the system itself may not have any agency. It's just gonna collect a bunch of features, and then maybe you've like trained a model, um, um, just like a traditional ML model to say like, "Yes, we're gonna invest," or, "No, we're not going to invest." You can, uh, go back and backtest those agentic features one by one with Exa and be like, "Hey," uh, against like a true positive match where something, um, like, uh, it's like, uh,

uh, the, the, the thing that you care about happened, either like that business succeeded or it didn't succeed. You can test and be like, "Were we able to correctly predict that, um, or did we not?" And most importantly, you can then tune that system because the way that, uh, with all of these non-deterministic systems, the way that you tune them is like really, really important. So if you tune it to search a certain way, maybe it succeeds more often. If you choose to search another way, it succeeds less often. And that is you can control that experiment and then add it into your system like one feature at a

time. Um, then like, uh, on the far, would it be right-hand side? The idea is like there's this like future where there's some like predictions, startups, and other folks operating, where potentially like the full system is agentic. Um, so the system itself may or may not choose to search the web through some tool call. Um, it could be, uh, combining internal knowledge. It could be di- combining like structured external knowledge, and it can kind of do that at runtime. The idea is be like, "Hey, this is kind of what

humans do when they go out and do some prediction. They have some agency. They're gonna search for this or search for that." And essentially, you can ground that, uh, like interface for your model into the outside web, uh, uh, into the outside world, and then you can, uh, ensure that when that system was operating, it was able to search, um, only the information that it had at that time. So one like practical example of this is like, let's say you're trying to improve your coding model, and hey, one part of that

coding model would be, uh, whether it can solve a PR or not. You could be like, "Hey, I'm gonna freeze right before that PR was solved, uh, in an, in, in an external repo, and then let the system run." And then hopefully, if it's able to resolve the task, you can have more confidence that that's a change in the model weights or whatever component in the system that you're actually paying to improve through your training process than it is just like future information sort of like leaking into the current. Okay. So that's just like a little bit of like an overview of

like Snapshot and a few of the ways that it's being used to backtest. I'm just gonna like, uh, recap what we've seen so far. So first, we really like Exa. Why do we like Exa? Because it's very good search, and search is very important for models because they need to know about, uh, the current, uh, and it needs to be high quality, it needs to be like, uh, cost-effective, and it needs to be at the latency that makes sense for my system. Okay, we've all learned that. And then we're like, "Hey, I really want that," and a big part about putting that into my model is being able to version

control it so I can prove that it adds value, not just like vibes and hope that it adds value. Um, what we're going to do now is like the fun part of this workshop, which is we're gonna give you all time to kind of go hands-on with this tool. So there's a little QR code on the screen. You can scan it with a mobile. You can also just like enter this little URL into your computer, and the, uh, uh, interface on your laptop will probably be, uh- More amenable. Um, and then I'm just gonna give you

like a little bit of an overview about what we're trying to do here. So the idea is, if you think back to the example that we had before, um, you're going to be trying to, uh, predict like what are company acquisitions right before they happen. Um, to tell like a little bit of the story here, so let's assume that you're working in some financial institution. Uh, your boss's boss's boss's boss like went to some AI engineer workshop, and they're like, "Hey, I heard this like really, really cool web search feature. I think we should

present it to our portfolio manager." And portfolio manager is like, "Okay, I wanna see it. You've got twenty minutes. Um, try and see if you can make it good." Um, and the most confounding variable here is you don't have like a coding agent, so you have to do it by hand because your usage just ran out or something. Um, so that's the kind of context of where we're sitting ourselves at. Your task is going to be to find the best two features that like accurately predict a company acquisition. Uh, and then Exa Snapshot is gonna use that to go back in

time, uh, one day, seven days, and thirty days before the acquisition and test your feature, um, or a random date if they weren't acquired. So there's a few confounding ones in there. Um, a feature is just like a yes/no question, um, that you believe would predict an acquisition. You can test as many as you like. There's a few examples in there, and I'm gonna be coming around just to talk with you all, um, and kind of like help you along the way. Um, and you can test as many as you like. Scoring, you get plus one point if you, uh,

uh, if your feature declares that this company would've been acquired, uh, on one, seven, or thirty days before it did. You get minus one if you say that it got acquired, um, but it didn't. And then no impact for misses or passes 'cause we're very friendly. Best score automatically captured. You can try as many as you want. I'm gonna set us up for, let's say, like thirty minutes. Um, so I will kick that off now. And you have thirty minutes to try and put in the absolute best

feature that you want. I'll be coming around. Maybe we can put on like a little bit of soft music that kind of gives people the vibe of working in a portfolio manager company, uh, and listening to like lo-fi beats. Uh, and I'll come around and talk with you all.

Okay. I'm just gonna do like one like real quick example just to kind of show people how it works. If you're already going away and, and, and winning, keep-- feel free to keep grinding. Um, but I'm just gonna put one example in here. So you'll see this interface in front of you. You have a bunch of, um... It'll ask you to put in a new feature. You can pick one of the examples or type whatever you want and hit Run. Um, and what it'll do is it'll go out, it'll use Exa Snapshot and search a bunch of them. This one was fast because it was cached, so, uh, that's a performance improvement to show you that we did test this beforehand.

Um, but it'll go down, and it'll run it against this set of ten companies. So within this set of companies, some of them were acquired, some of them were not acquired. They're confounding variables. Um, and the idea is that like we're gonna take this particular query, so has this company lost its C- CEO or CFO or appointed interim leadership? And for all of this, we're gonna go back and do that version search. So Delivery Hero, we can see, hey, um, there was an executive leadership turnover, yes. And the company did end up being acquired. So, hey, maybe this is

predictive. Except Quadient, I don't know if I'm pronouncing that correctly, they also had like executive leadership turnover, but they didn't end up being acquired. Oh, no, that's a confounding variable. So your task is to try and be like, "Hey, what are the one or two features that I can add to this that'll be the most predictive?" So here I'm gonna like add one more.

So this one is like, "Hey, has it been reported to be exploring a sale?" Does adding that improve it or make it worse? And in this case, we can see, hey, we've picked up a lot of the, um, uh, we've picked up like a lot of the acquisitions, so we're doing pretty well in terms of finding stuff. But also like, hey, we're picking up like a lot of false alarms here. So we might need to be more, uh, constraining in the way that we're combining these two features. So here I've put a very simple like combined thing. You can either choose or, which is if either of your two features trigger yes,

then we will count that whole thing as being like, yeah, you're predicting that it is gonna be acquired as of that day. You can go and, and if I see if I go and here, the score improves a little bit, which means both of these two features need to be true. And if both of those two features are true for that company on that day, then you'll trigger yes. Um, that's just like a PSA. I'll go back to the kind of like leaderboard there, but feel free to chuck up your hands with any questions as we come through. Okay, ten seconds left. If you haven't already hit Run, it may be too late. Feel free to knobble

over the combine and/or to try and get your last few results. Okay, congratulations. So Zane, John, and Ryi, could you come and join me on stage if you won a prize? If not, we'll go further down in the list. Can we have a big round of applause?

Come up. Don't be shy, don't be shy. Brilliant goats and mystery boxes await you.

How's it going?

Hey, congratulations. Congratulations, John. Very value aligned.

Cheers.

Thank you.

Hey, congratulations. Yeah. Um, so yeah, let me hand these out.

Thank you.

You're all lined up brilliantly, so we can do a photo as well.

Oh, yeah.

There's nothing. Goat.

I just get a deep diving goat.

You do get a

deep diving goat.

Okay, sweet. Can we do a photo? Is there a photographer or... Okay, sweet. Let us know. I don't know how to get this video. Just video. Okay, sweet.

How about a round of applause?

Okay, we'll take a still out of that video later. So let's just like poke around and be nosy and see like what did our number one person did. So Zane, is this company a publicly listed company? Very interesting feature. Number two, is this company a company other than QIAGEN, Vitrus, Bank of Cryptos, and American Electric Power? Congratulations. You are the reason why I wanted to have a holdout set in this eval. I removed it because I wanted it to be simple. But this is a good example of why you need to do backtesting and why you need to do

backtesting with a holdout set. Um, so I'm gonna take this as a feature, not a bug of, of not including a holdout set and why overfitting is very dangerous because I do not think that this particular algorithm would be successful in the future, but who knows?

Number two, Rhi. So okay, he stole Zane. Maybe we've been seated nearby. And lastly with John, so within the past twelve months, has an active investor or significant shareholder publicly called for a p- supported sale of company as a whole, or is company's current share price at least sixty percent below its highest price during the past five years, or is it named Dominion? Also, a little bit of overfitting there. But I would argue it would probably perform better. Um, so this gives like an example of just like, hey, how and why it's important to do backtesting, right? Because, you know, this is like a little constrained example. I,

I, I feel like I'm regretting removing my holdout set, although I do feel like people would've been way more confused if I did do that. Um, but this gives you an idea of like why it's valuable, right? Because if you know what the existing set is, you can, you can, uh, overfit it. But the idea is like, because you have this capability, you can check for that, right? You can go back and be like, "Hey, I'm gonna go back in my book," and be like, "Hey, I'm gonna constrain like these ten companies that I know something had happened to them, twenty companies that didn't, and train and iterate on my model," right? "And make sure that it continues to perform,

um, beyond just like the publicly visible sets." Um, so let's like recap the highlights. I'm also gonna... Oh my God, there's a lot of people. I'm gonna have to scroll very far. I'm gonna put you all on a, uh, thank you page, or you should already be on the thank you page. On the thank you page, you've got a link to our docs. We love documentation. You have a link to our blog. It's more readable than the docs a-and is more like human-facing. Uh, and lastly, there's a chance to send me an email. I really regret that now because there are 100 people in the room, so I'm expecting

a lot of emails. Um, but this is an enterprise feature as well. We have some public usage. But if you're interested in having access to this for your team, send me an email, and I'll connect you to our enterprise team. Um, let's just like recap the highlights. Number one, we love Exa. Why? Because it's great agentic web search. It's good value. It's very cost-effective. It's at whatever latency I need. I want it in my model. Number two, Snapshot. It lets me backtest my model. It lets me see just how good Exa is at improving

your model. And number three, um, consider really adding the holdout set to your competition because otherwise you may overfit. Thank you, everyone.