Evaluating AI Search: A Practical Framework for Augmented AI Systems

How do you evaluate an AI search agent when both its information sources and its users keep changing? A live agent makes decisions using evolving web content, user interactions, and chains of tools. Retrieval failures, reasoning errors, and hallucinations can occur together, so a successful request does not necessarily mean a successful answer. This is the problem explored by Julia Neagu, introducing herself as Quotient AI’s CEO and co-founder; Deanna Emery, its founding AI researcher; and Maitar Asher, Tavily’s head of engineering.

Quotient’s approach is to monitor live agents with evaluators that identify failures without waiting for reference answers, human feedback, or benchmarks. Tavily founder Rotem brought the team a particularly demanding application: web search. The web changes independently of the agent, while users ask malformed questions, omit context, and venture beyond prepared test cases. Julia reports that Tavily processes hundreds of millions of production search requests, without specifying a time interval. Evaluation has to work under those conditions, not only against a fixed test set.

Tavily supplies real-time web information to language models. Its customers illustrate how differently that information can be used:

• Legal assistance: A contract lifecycle management company provides case insights to legal and business teams.
• Sports updates: A news outlet combines retrieval-augmented generation, or RAG, with chat to deliver scores, game information, and news.
• Fraud investigation: A credit card company searches for merchant locations to help detect fraud.

These applications do not share one definition of a useful search result.

Two requirements follow. First, evaluation must keep pace with changes to the web. Second, correctness can depend on the source, timing, and user’s needs. A current sports score and an interpretation of legal material pose different evaluation problems. Where a single absolute answer is difficult to establish, the evaluation design still needs to address bias, fairness, and the context in which an answer will be used.

The first offline evaluation decision is which data to use. SimpleQA provides short, fact-seeking questions with single empirical answers. It is a factuality benchmark used here to evaluate search-augmented answers, rather than a direct measure of retrieval accuracy; its original questions deliberately target answers intended to remain stable over time. HotpotQA adds a different challenge: answering questions that require reasoning across multiple documents.

Static datasets remain useful, but they cannot by themselves show whether a search system keeps up with rapidly evolving information or avoids regressions on today’s questions. Nor do fixed, single-answer tests resolve subjective questions. Regularly refreshed datasets extend coverage to the domains and use cases an application actually serves. The evaluation data needs to change when the information need changes.

Tavily built an open-source agent to generate domain-specific question-and-answer pairs from web information. Asher credits Eyal, Tavily’s head of data, with initiating the project. The agent uses LangGraph to organize the generation workflow.

1. Generate broad search queries for the selected domains, tailoring coverage to the application.
2. Collect grounding documents from multiple search providers. Asher describes this as a way to broaden coverage and reduce the bias of using Tavily’s own retrieval to create the questions on which Tavily will be evaluated.
3. Generate evidence-based question-and-answer pairs with answer context. Each pair retains the sources and specific evidence used to create it, so someone can inspect the basis of the answer.
4. Track offline experiments in LangSmith to compare performance over time.

The answer context is a critical part of the dataset. A generated answer without its supporting evidence is harder to audit and can introduce hallucinations into the evaluation itself.

The proposed extensions were broader question types, including simple facts and HotpotQA-style multi-hop questions; further work on freshness, fairness, bias, and coverage of different perspectives; and a supervisor node to coordinate the multi-agent workflow. These were next steps, distinct from the generation process already described.

Dataset quality is only one part of the framework. Accuracy should sit alongside source diversity, source relevance, and hallucination rates. Reference-free evaluation reduces dependence on labeled answers, making evaluation easier to apply at scale and to questions where a single reference answer is difficult to define. The next experiment tests both the choice of dataset and the usefulness of those additional metrics.

Emery describes a two-part evaluation of six anonymized AI search providers. First, the team compared correctness on static and dynamic datasets. Second, it evaluated the dynamic responses using reference-free metrics and compared those results with reference-based correctness. The question was whether those metrics could provide useful signals when reference answers were unavailable.

The experiment compared SimpleQA with a Tavily-generated dynamic benchmark of about 1,000 rows, with roughly similar topic distributions. Both were evaluated using the SimpleQA correctness approach: an LLM judge compares a response with a reference answer and assigns one of three labels—correct, incorrect, or not attempted.

The SimpleQA provider scores shown in the comparison were self-reported, and Emery notes that their calculation methods were not all clearly documented. In the presented results, dynamic-benchmark correctness scores were substantially lower, and the relative rankings changed. Provider F ranked worst on SimpleQA but best on the dynamic benchmark. The practical implication is that a static leaderboard alone does not establish which search provider will perform best on refreshed questions relevant to an application.

Inspecting the responses exposes a second problem: the judge is also fallible. Emery describes examples labeled incorrect even though their text contains the reference answer. These are cases where the team found a mismatch between the assigned label and the answer present in the response.

The opposite problem appears in a response labeled correct: the expected answer is present, but the response also contains additional material that may include hallucinations. The official SimpleQA rubric does not treat answer inclusion as sufficient when a response contradicts the reference. Still, its short-answer correctness task does not establish the factuality of every claim in a long response. Matching the requested fact and grounding the entire answer are separate evaluation tasks.

Most production queries do not arrive with ground-truth answers. Reference-free evaluation instead examines the question, the response, and—when available—the retrieved evidence. Emery applies three Quotient metrics to the providers’ dynamic-benchmark responses.

The unknown completeness category covers an answer that says it does not know. Hallucination detection here is specifically a grounding check: it asks whether the retrieved documents support the response’s facts. It does not require a separate reference answer, but it does require access to the evidence used to produce the answer.

The answer-completeness rankings closely followed the dynamic-benchmark correctness rankings. Emery reports a correlation of 0.94 between average answer-completeness and dynamic-benchmark correctness scores across the six providers. This is a relationship between aggregate provider scores, not a per-answer accuracy measurement. Completeness still does not establish correctness: an answer can address every part of a question and be wrong. To examine its support, evaluation needs the grounding documents.

Only three of the six evaluated providers returned the retrieved documents used to generate their answers. Emery notes that providers typically supply citations instead, which limits evaluation at scale and makes debugging less transparent. The distinction matters operationally: a citation points toward a source, while the retrieved text shows what evidence was actually available to the answer generator.

The three providers with document access were re-anonymized for the remaining comparisons. Document relevance was examined in two ways: the average percentage of relevant documents, and the number of responses with no relevant documents at all. These views separate retrieval that includes some irrelevant material from retrieval that supplies no useful evidence.

Emery reports a strong inverse relationship between document relevance and the number of unknown answers. That relationship fits the intended behavior: when the retrieved material cannot support an answer, the model should acknowledge that it does not know rather than fill the gap. Unknown responses therefore need to be interpreted alongside retrieval quality, not automatically treated as failures of the answer generator.

The hallucination results complicated the picture. Among the three document-returning providers, Provider X had the highest hallucination rate and the highest document relevance. It also had high answer completeness, the lowest unknown-answer rate, and the highest benchmark correctness within that group. Strong retrieval and a strong correctness score did not eliminate unsupported content elsewhere in the response.

Emery offers a possible explanation: Provider X may produce more reasoning, interpretation, or detail, creating more opportunities to introduce claims absent from the retrieved evidence. This is a hypothesis about the observed pattern, not an established causal result.

The metrics measure different dimensions of quality, so their importance depends on the application. More complete answers can come with greater hallucination exposure; a cautious answer can avoid unsupported claims while leaving the user’s question unresolved. Reading completeness, relevance, and hallucination results together provides a more useful basis for diagnosing failures than collapsing them immediately into one ranking.

Consider the diagnostic example Emery gives: an incomplete response, relevant retrieved documents, and no detected hallucinations. Relevant documents can still leave part of the question unanswered. This combination suggests that the agent may lack some necessary information, so retrieving additional documents is a possible remedy. It does not establish that retrieval expansion will fix the answer.

A small Python rule can express that diagnostic combination as a proposed next action:

from dataclasses import dataclass

@dataclass(frozen=True)
class Evaluation:
    complete: bool
    has_relevant_documents: bool
    hallucination_detected: bool


def propose_next_action(result: Evaluation) -> str:
    if (
        not result.complete
        and result.has_relevant_documents
        and not result.hallucination_detected
    ):
        return "Retrieve more documents to investigate missing information."
    return "Inspect the response and evidence before choosing a repair."


result = Evaluation(
    complete=False,
    has_relevant_documents=True,
    hallucination_detected=False,
)
proposal = propose_next_action(result)

The rule proposes an investigation; it does not execute retrieval or declare the response repaired. That is the useful transition from measurement to debugging: identify the kind of issue present, then select a strategy that could address it.

Julia closes by extending this diagnostic loop into a goal for agent design. An agent could learn recurring patterns: which information becomes outdated, which sources prove unreliable, and which needs users repeatedly express. Evaluation would then influence how it searches and answers, rather than remain a report produced after the interaction.

The further ambition is to detect hallucinations during a conversation and correct course without human intervention. That is a direction for future systems, not a deployed capability demonstrated here. Dynamic datasets supply current evaluation questions; multiple quality dimensions expose different failures; reference-free checks make those signals available when no answer key exists. Together, they provide the proposed building blocks for agents that use evidence about their own failures to improve subsequent decisions.