Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

A coding agent looking for documentation can search the web, inspect results and bring relevant information back to its orchestration process. Ryan Ahern, a forward deployed engineer at Exa, describes the same pattern for financial research: an agent can make several searches and follow leads that a person might abandon after the first page. 2:12

Ahern frames search design around three tradeoffs:

• Quality: A regenerative benchmark can keep evaluation aligned with changing information and a particular use case, instead of relying indefinitely on a fixed benchmark that providers may optimize against.
• Cost: Retrieved text consumes model input tokens. Returning relevant excerpts across results can reduce the context an application pays to process.
• Latency: Ahern describes searches taking under 300 milliseconds at one end and a roughly 20-minute search for video game publishers at the other; the latter found 3,212 publishers and their developers. The appropriate budget depends on the task.

Exa Snapshot adds a historical dimension to search. As Exa indexes content, it stores versions of pages it encounters. For a known URL, content versioning accepts a date and returns the version at that date or the nearest stored version before it. Versioned search applies the date to a search for relevant pages, where the URLs are not necessarily known in advance. The result depends on what Exa had stored; the nearest earlier copy need not capture every change that occurred before the requested date. 5:42

One proposed use is testing whether a financial research system would have found a meaningful accounting distinction when it was available. Ahern uses reports about OpenAI’s annualized revenue to illustrate why gross versus net revenue, and the treatment of cloud-provider costs, could change the interpretation of a headline number. The test is whether the system finds the relevant detail at the chosen time, rather than merely repeating the number known today.

A due-diligence system faces a similar question: would a search for adverse media have surfaced useful information about a person or business before a later event? Historical retrieval lets a team evaluate the search and classification process against an earlier information set.

Suppose a company has already been acquired. A live search may find the announcement, making a prediction of that same acquisition look trivially successful. That is look-ahead bias: information from after the decision point has entered the evaluation. 9:43

Move the search cutoff to one day, one week or one month before the acquisition and ask whether earlier signals would have supported the prediction. A team can then compare features based on hiring, earnings calls or CEO interviews. The cutoff constrains the retrieved web information; an evaluation must also consider future information that could enter through other sources, including the model itself.

The role of search changes with the system being evaluated:

• Historical data for an existing model: Retrieve older versions of known pages, turn their contents into features and feed them to a model that does not itself search. Ahern cautions that an article’s displayed publication date can change after an update, so the date alone may not establish what text was available earlier.
• A search-based feature in a traditional model: An agent searches unstructured material, such as adverse-media reports, and converts it into a structured signal. A conventional model combines that signal with other features. Historical searches let the team test and tune the search feature against known outcomes.
• A fully agentic system: The system chooses at runtime whether to search and how to combine results with internal or structured external information. A historical search interface constrains its web access while it makes those decisions.

For a coding evaluation, Ahern proposes freezing search before an external repository’s pull request was solved, then letting the coding system attempt the task. That makes a successful result more informative about the system being tested, because its search cannot simply retrieve the later solution. The historical search cutoff alone does not establish that every other source of future knowledge has been removed. 15:42

The workshop asks participants to write up to two features, each a yes-or-no question intended to predict an acquisition. Snapshot tests acquired companies at one, seven and 30 days before their acquisitions, and non-acquired companies at a random date. The exercise awards one point for a positive prediction of an acquisition, subtracts one for a false alarm and assigns no points to misses or passes. Participants can try multiple features and keep their best score. 18:11

The demonstration runs a question across ten companies: has the company lost its CEO or CFO, or appointed interim leadership? In the exercise, Delivery Hero has leadership turnover and is labeled acquired. Quadient also has turnover but is labeled not acquired. The second case shows the problem with treating turnover alone as a sufficient signal: it can produce a false alarm.

Ahern adds a second question: has the company been reported to be exploring a sale? Combining the features with OR predicts an acquisition if either question returns yes. That picks up many acquisitions in the demonstration, but also many false alarms. With AND, both questions must return yes for the same company and date. The score improves a little in this example, though the talk does not give the underlying counts.

The result answers a practical question: does requiring both signals improve this exercise’s score? It does here because the stricter combination changes which cases trigger a positive prediction. The scoring rule matters too: it penalizes false alarms but does not penalize missed acquisitions, so the winning combination reflects that particular tradeoff.

After the prizes, the top entry reveals an uncomfortable shortcut. One feature asks whether the company is publicly listed; another excludes several companies by name. Ahern had removed the holdout set to keep the exercise simple, and the entry fits the visible cases instead of offering a convincing rule for future acquisitions. 23:38

Another entry combines more general questions—whether an activist investor or significant shareholder supported a sale in the past 12 months, or whether the share price is at least 60% below its five-year high—with a company-name condition. Ahern expects this entry might generalize better, but its named exception still fits the known set. That is a judgment, not a measured result on unseen companies.

A holdout set supplies cases that were not used to choose the feature. Tune on one collection of historical acquisitions and non-acquisitions, then evaluate the chosen rule on separate cases. Snapshot helps reconstruct the information available before each outcome; the holdout asks whether the rule learned from the visible examples travels beyond them.