Introduction
Jeffrey Pinner, CTO of Exa, explains why agent search needs to retrieve the right entities and facts, not merely return pages that resemble a query.
A model can reason well and still lack the fact that makes its answer correct. Pinner opens with the growth of search traffic at Exa, then turns to the more durable reason agents need search: the world changes after training, and training never contained all of the world in the first place.
New APIs appear. Prices change. Regulations change. Specialized information may never have been prominent enough to enter a model's training data. A larger model does not make those coverage gaps disappear.
The relevant question is therefore not whether an agent can produce a plausible answer from memory. It is whether its harness can give it the missing knowledge, in a form that helps it complete the task.
Evaluate the completed research task
Pinner introduces Atlas, a benchmark built around finding entities that satisfy conditions and enriching those entities with facts. Think of a research table: each row identifies something, and its columns contain the attributes the requester needs.
A row only counts as correct when all the required attributes are correct. That is stricter than checking whether one search result contains a useful keyword. It tests the thing the user would actually consume: a completed, accurate piece of research.
In the results Pinner presents, the closed-book model gets only 9% on this task. He also shows performance worsening as questions depend on more recent information. These are results for the particular benchmark setup, not a claim that a model answers only 9% of ordinary questions correctly. Their role is to establish that reasoning without external knowledge is insufficient for this workload.
Agent queries carry constraints that keywords can lose
Agents do not always search like people. Pinner describes queries that are longer and more conditional, packing several requirements into a single request. A company search might require a particular business category while explicitly excluding companies in the United States.
A keyword engine can retrieve pages containing the right terms and still violate the exclusion. The result is superficially relevant but unusable. The problem is especially apparent when the agent is assembling a table: one ignored condition can invalidate the whole row.
Coverage creates another failure mode. Some useful information is proprietary, behind a paywall, or simply absent from the index. More careful phrasing cannot retrieve a document the search system never had access to. Pinner separates this coverage problem from ranking quality rather than treating every failure as a bad prompt.
There is also a cost after retrieval. If the agent repeatedly downloads whole pages, parses navigation and boilerplate, and searches for a few useful lines, its context fills with material unrelated to the answer. Finding a page is only the beginning of the knowledge-delivery problem.
Search the meaning, then broaden the accessible corpus
In Pinner's reported Atlas comparison, adding keyword search raises performance to 28%. That improvement establishes the value of external information, but leaves considerable room to improve the retrieval system.
He describes a semantic pipeline: crawl pages, extract their meaningful content, compute embeddings, and build an index. A query is embedded into the same space so the system can retrieve content by semantic similarity rather than relying only on matching words.
Semantic retrieval is not itself a guarantee that every logical constraint will be respected. Its purpose here is to supply candidates whose meaning better fits the request, giving the downstream agent better material to evaluate.
Pinner then broadens the data layer. The system can combine public web content, partnered proprietary sources, and a knowledge graph behind one API. Partnered access matters: this is a proposal to integrate authorized sources, not an assertion that semantic search bypasses their access restrictions.
Return the relevant evidence, not the whole page
The final retrieval improvement is to select the passage that answers the query. Pinner uses a lithium-ion battery page: a long document contains a short passage identifying the scientists who shared the 2019 Nobel Prize in Chemistry. Sending that passage to the agent is much cheaper than making it rediscover the same lines inside the full document.
At 3:09:25, the slide contrasts the full page with a sourced highlight. Its token-reduction figure describes this displayed example, not a guarantee for every query.
This changes what search delivers. The useful unit is no longer merely a URL or a page body. It is a relevant piece of evidence attached to its source. The source association remains important because a concise excerpt should not become an unsupported assertion that the reader or agent cannot inspect.
The final chart shows an Atlas score of 45% for semantic search, compared with 28% for keyword search and 9% for closed-book answering. It separately labels the improvement over keyword search as +56% and reports 28% lower cost. Those are different quantities: 56% is not the semantic system’s absolute score. The talk does not establish that all of those gains transfer unchanged to another dataset, and the configurations change more than one part of the system. The comparison is best read as a demonstration of harness leverage: retrieval quality, coverage, and context efficiency can materially change the work a fixed model can finish.
The closing point is practical. Improving the model is one route to better answers. Improving what the model can know at the moment it acts is another, and it can address failures that better reasoning over stale information cannot fix.