Evaluating AI Search: A Practical Framework for Augmented AI Systems

Talk transcript

1. 0:00

   [on-hold music] Hi, everyone.

2. 0:16

   Uh, thank you so much for coming. Uh, my name is Julia. I'm CEO and co-founder of Quotient AI.

3. 0:22

   Uh, I'm Deanna Emery. I am founding AI researcher at Quotient AI.

4. 0:27

   My name is Maitar Asher. I'm head of engineering at Tavily.

5. 0:30

   And today, we are going to talk to you about, uh, evaluating AI search. So let me start with a fundamental challenge we're all facing in AI today. Traditional monitoring approaches simply aren't keeping up with the complexity of modern AI approaches.

6. 0:46

   First off, these systems are dynamic. Unlike traditional software, AI agents operate in constantly cha-changing environments. They're not just executing predetermined logic, they're making real-time decisions based on evolving web content, user interactions, and complex tool chains.

7. 1:03

   The-these systems can also have multiple failure modes that happen at the same time. They hallucinate, their retrieval fails, uh, they make reasoning errors, and all of these are interconnected.

8. 1:15

   A little bit about what we do at Quotient. We monitor live AI agents. Uh, we have expert evaluators that can detect objective system failures without waiting on ground truth data, human feedback, or benchmarks.

9. 1:27

   A year ago, we met Rotem, uh, Tavily's founder and CEO, and he posed us with a problem, uh, that really crystallized the core issues we needed to solve.

10. 1:37

    Here's the challenge: How do you build production-ready AI search agents when your system will be dealing with two fundamental sources of unpredictability you cannot proactively control? Under the hood, Tavily's agents gather their context by searching the web.

11. 1:52

    The wor- the web is not static. Traditional benchmarks assume stable ground truths, but when you're dealing with real-time information, ground truth itself is a moving target. Your users also don't stick to your test cases.

12. 2:04

    They can ask odd, malformed questions. They have implicit context they don't really share and you're not aware of. Uh, and this is not just a theoretical problem. Tavily's, uh, Tavily processes hundreds of millions of search requests for its AI a-agents in production, and they need a solution that work that scale in these real-world conditions.

13. 2:25

    And this is a story of how we built that.

14. 2:30

    Yes. So at Tavily, we're building the infrastructure layer for agentic interaction at scale, essentially providing language models with real-time data from across the web.

15. 2:43

    There are many use cases where real-time AI search deliver values, and this is just a few example of how our clients are using Tavily to empower their applications. Beginning from a CLM company that built an AI legal assistant to power their legal and business team with instant case insight, to a sports news outlet that they created a

16. 3:03

    hybrid RAG chat agent that delivers scores, games, and news updates, to a credit card company that uses real-time search to, uh, fight fraud by pinpointing merges-- merchant locations.

17. 3:19

    So as you can imagine, evaluate a system in this kind of vast, fast-moving setting is quite challenging. We have two principles that guide our evaluation. First, the web, which is our foundation of our data, is constantly changing.

18. 3:35

    This means that our evaluation method must keep up with the ongoing change.

19. 3:41

    Second, that truth is often subjective and contextual. Evaluating correctness can be tricky because what's right may depend on the source, or the timing, or the user needs. So we have a responsibility to design our evaluation methods to be as unbiased and fair as possible, even when absolute truth is hard to pin down.

20. 4:05

    So the first thing to think about in offline evaluation is which data to use to evaluate your system.

21. 4:13

    Static datasets are a great start, and there are many widely, uh, open source datasets available out in the web. SimpleQA is one example. It's a benchmark and a datasets from OpenAI that, um, serve as a standard for evaluating retrieval acc-accuracy.

22. 4:31

    We have me-many leading AI search providers that use SimpleQ-QA to evaluate their performance. SimpleQA is designed to evaluate the system ability to answer short, fact-seeking question with a single empirical answer.

23. 4:48

    Another widely, uh, adopted dataset is HotpotQA, which, uh, tests the-- which evaluate the system ability to answer multi-hop questions where reasoning across multiple documents is required to retrieve the final answer.

24. 5:04

    Dataset like SimpleQA and HotpotQA are a great start for evaluating your system. But what happens when you're, um,

25. 5:14

    when you're evaluating real-time, um, systems that, um, especially when measuring your s- that your system keeps up with rapidly evolving in-information and avoiding regression, regression, like where we operate?

26. 5:29

    Also, those kind of static datasets, uh, don't address the challenge of benchmarking questioning-- questions where, uh, they don't-- where there's no one truth answer or subjectivity is involved.

27. 5:45

    This is what led us to think beyond static datasets towards dynamic evaluation that reflects the changing, uh,

28. 5:55

    the, the pace of the, of the web, essentially. Um, dynamic dataset are essential for per- for benchmarking RAG agents in real-world production system. You can answer today's questions with yesterday data.

29. 6:09

    Dynamic datasets have real-world alignment. They have broad cover- coverage, as you can easily create eval sets for any domain or use case that is relevant to your specific needs.

30. 6:22

    And they alsure- also ensure continuous relevancy because they are regularly refreshed, which means that your system is cons- is always evaluated against the latest data.

31. 6:34

    This led us to build an open source, um, agent that basically build dynamic eval sets for web-based RAG system. It's open source, and we encourage everyone to check it out and contribute.

32. 6:50

    And I also want to acknowledge the work of Eyal, our head of data at Tavily, who initiated this project a couple months ago.

33. 6:57

    As you can see here, an example of a dataset gen- generated by the agent. It generates question and answer pairs for targeted domains using information found in the web.

34. 7:11

    So the agent leverage the LangGraph framework, and it consists of these key steps. First, it generates broad web search queries for targeted domains, which essentially lets you create eval sets for any, uh, domain of your choice and specific need of your application.

35. 7:30

    The second step is to aggregate grounding documents from multiple real-time AI search providers. We understand that we cannot just use Tavily to search the web on specific domains, find grounding documents, then generate question and answer pairs from those documents, and then evaluate our performance on those documents.

36. 7:51

    That's why we use multiple real-time AI search providers to both maximize coverage and minimize bias.

37. 7:59

    The third step, which is the key, uh, step in this process, is to generate the evidence-based question and answer pairs, and we ensure that in the generation process, the agent is obliged to, to generate answer context, which als- which also increase the reliability of our question and answer pairs and reduce hallucinations.

38. 8:21

    You can always go back and ch- and check which sources were used and which evidence from, so those sources were used to generate each question and answer pair. And lastly, we use LangSmith to track our experiments, which is a great observability tool to manage these, uh, offline, um, evaluation runs and see how are your performance at different

39. 8:42

    time steps. The next steps that we want to address is to support a range of question types, both simple fact-based questions and multi-hop questions similar to the HotpotQA. We also want to ensure furnish and fairness and coverage by proactively addressing bias and covering a wide range of perspective for each subject we

40. 9:07

    generate question and answer to. Additionally, we want to add a supervisor node for coordination, which prove itself to be valuable, especially in these multi-agents, uh, architectures, and this will increase the quality of our question and answer pairs.

41. 9:26

    The next step to think about is, uh, benchmarking, and we argue that it's important to measure accuracy, but you should not stop there. You should ensure an holistic evaluation framework which use benchmark like for our, for our case that, um, measure your s- your source diversity, your, uh, source relevancy, and hallucination rates.

42. 9:50

    It's also important to leverage unsupervised evaluation method that remove the need for labeled, for labeled data, which enable to scale your evaluations and address the subjectivity, uh, issue. With that, I'll pass it over to Deanna, who, who will explain more about this reference-free benchmarks and also share, uh, results from, uh, an experiment we ran using a static

43. 10:14

    and a dynamic dataset that was generated by the agent I described before.

44. 10:21

    So, uh, we performed a two-part evaluation of six different AI search providers. Um, the first component of this experiment was to compare the accuracy of search providers on a static and a dynamic benchmark in order to demonstrate that static benchmarking is not a comprehensive method for evaluation of AI search.

45. 10:43

    The second component was to evaluate the, uh, dynamic dataset responses using reference-free metrics, and we compare these results to the, uh, reference-based accuracies that we get from the benchmark in order to demonstrate that reference-free evaluation can be an effective substitute when ground truths are not available.

46. 11:07

    So jumping right in, um, for our static versus dynamic benchmarking comparison, we use SimpleQA benchmark as the static dataset, and we're using a dynamic benchmark of about 1,000 rows created by Tavily.

47. 11:23

    And as you can see here, uh, both datasets have roughly similar distributions of topics, and this helps to ensure a fair comparison and diversity of questions.

48. 11:33

    To evaluate the AI search provider's performance on these two benchmarks, we're using the SimpleQA correctness metric, and this is an LLM judge, which is used on the SimpleQA benchmark.

49. 11:46

    It compares the model's response against a ground truth answer in order to determine if it's correct, incorrect, or not attempted.

50. 11:58

    And so here we're showing the correctness scores from that SimpleQA benchmark compared against the dynamic benchmark. And, uh, we've anonymized the search providers for this talk. Um, but I do wanna call out that the SimpleQA accuracy scores here are all self-reported, and so they don't all necessarily have clear documentation on how they were calculated.

51. 12:18

    But, um, as you can see, the correctness scores are, for the dynamic benchmark in blue, are substantially lower. Um, and not only that, the relative rankings have also changed pretty considerably.

52. 12:31

    For example, um, provider F all the way on the end of this plot here performs the worst on SimpleQA, but it performs the best on the dynamic benchmark.

53. 12:43

    Looking a little closer to the... in the results, um, while this SimpleQA evaluator is useful, it's certainly far from perfect. Um, I have a few examples here of, um, model responses that were flagged as incorrect by this LLM judge.

54. 13:00

    Uh, but if you look at the actual text in the model outputs, they do contain the correct answer from the ground truth.

55. 13:07

    On the flip side of things, um, here is an example where, uh, the LLM judge classified it as correct, and yes, you can see that the correct answer is in this response.

56. 13:18

    But while the correct answer might be present, that doesn't necessarily mean that the full answer is right. Um, this evaluation is not accounting for any of the additional text in this response, and there might be hallucinations in there, and that would invalidate it.

57. 13:33

    So ultimately, this evaluation falls short of identifying when things go wrong in AI search.

58. 13:42

    So what are some other ways that we can identify when things go wrong? Up to this point, we have been talking about a reference-based approach to evaluation. But what if we don't have ground truths?

59. 13:53

    In most online and production settings, this is typically the case, and as we've already discussed, it's especially so in AI search.

60. 14:02

    Um, so the question is, can reference-free metrics effectively identify issues in AI search? For this talk, we're going to look at three of Quotient's reference-free metrics. Um, we'll look at answer completeness, which identifies whether all components of the question were answered.

61. 14:21

    Um, so it classifies model responses as either fully addressed, unaddressed, or unknown, uh, if the model says, "I don't know." Then we'll look at document relevance, and this is the percent of the retrieved documents that are actually relevant to addressing the question.

62. 14:37

    Um, and then finally, we'll look at hallucination detection, which identifies whether there are any facts in the model response that are not present in any of the retrieved documents.

63. 14:49

    And so we use these metrics to evaluate the search provider's responses on this dynamic benchmark.

64. 14:58

    So we've got answer completeness plotted here. Um, the stacked bar plot shows the number of responses that were either completely answered, uh, unaddressed, or marked as unknown. And if we look back at the overall rankings that we saw earlier on the dynamic benchmark, um, you can see that the rankings from answer completeness pretty closely match.

65. 15:20

    Um, the average performance scores for the two get a correlation of point nine four. So this indicates that the reference-free metric can capture relative performance pretty well. But completeness is still not the same thing as correctness, um, and when we have no ground truths available, then we have to turn to the next best thing, and that is

66. 15:42

    the grounding documents. So this is where document relevance and hallucination detection come in. Uh, both of these metrics are going to be looking at those grounding documents in order to measure the quality of the model's response.

67. 15:56

    Unfortunately, uh, of all of the search providers we looked at, only three of them actually return the retrieved documents used to generate their answers. Um, the majority of search providers typically only provide citations, and these are largely unhelpful at scale and also really limit transparency when it comes to debugging.

68. 16:18

    So these are those document relevance scores for the three search providers, um, and they've been re-anonymized here. Um, the plot to the left shows the average document relevance, the percent of retrieved documents that are relevant to the question, and the plot to the right shows the number of responses that have no relevant documents.

69. 16:40

    And if we consider these results in conjunction with answer completeness, we find that there's a strong inverse correlation between document relevance and the number of unknown answers. And this kind of matches intuition.

70. 16:55

    Uh, if you think about it, if you have no ground-- no relevant documents for the question, the model should say, "I don't know," rather than trying to answer it.

71. 17:04

    And so this brings us to hallucination detection. And here we were actually surprised to see that there was a direct relationship with the hallucination rate and document relevance. Provider X here has the highest hallucination rate, but it also had the highest overall document relevance.

72. 17:24

    And this is kinda counterintuitive, um, but if we think about it more, Provider X had high answer completeness, the lowest rate of unknown answers, and it also had the highest answer correctness from the benchmarking earlier of these three providers.

73. 17:43

    So this probably implies that maybe in Provider X's responses, um, they're more likely to provide new reasoning or interpretations in their response, or maybe even they're more detailed and thorough, and this just creates more opportunity for hallucination in their responses.

74. 18:02

    Um, but the point I wanna make here is that when considering these metrics, depending on your use case, you might index more heavily on one over another. Um, they're measuring different dimensions of response quality, and it's often a give and take.

75. 18:16

    If you perform really well in one, it might be at the expense of another. Um, and as we see here, uh, there is a trade-off between complete-- answer completeness and hallucination.

76. 18:29

    But also, um, if you take these three metrics in conjunction, you can use them to understand why things went wrong and, uh, identify potential strategies for addressing those issues.

77. 18:40

    This diagram here shows a few examples on how you can interpret your evaluation results. Um,

78. 18:48

    sorry, uh, how you can interpret your evaluation results to identify what to do to fix it. So we've got one example here where, um, maybe your response is incomplete, um, but you have relevant documents, you have no hallucinations.

79. 19:02

    Uh, so this probably means you don't have all the information you need to answer the question. Uh, and so just retrieving more documents might solve that. Um, but the big picture idea is that your evaluation should do more than just provide relative rankings.

80. 19:16

    It should help you identify the types of issues that are present, and it should also help you understand what strategies to implement to solve those issues.

81. 19:26

    Okay. So, um, uh, so in conclusion, let me just quickly paint a picture of where we're heading with all this, because this is not just about building the agents we've been building for the past couple of years and then slapping evaluation on it, and then continuing to do the same thing.

82. 19:41

    Uh, it's actually... It's not about building better benchmarking, it's not better monitoring, it's not about better evaluation. It's about creating AI systems that can, uh, continuously improve themselves. And imagine for a second that agents don't just retrieve information, but learn from the patterns of what information is outdated, what sources are unreliable, and what users need.

83. 20:03

    Um, they can also, like, maybe detect hallucinations mid-conversations and, uh, correct the course, all without human intervention. And this framework that we shared today, dynamic datasets, holistic evaluation, reference-free metrics, are the building block-blocks for getting there.

84. 20:21

    Uh, and this, this is where we want to get f- uh, with augmented AI. So thank you so much for your time. [outro music]