RAG Evaluation Is Broken! Here's Why (And How to Fix It)

Talk transcript

1. 0:00

   It seems that RAG is so 2023, and everywhere you go, you see, "My first RAG," and, "RAG pipelines," and everything is RAG related, as if it's already solved. But actually, not really.

2. 0:14

   Uh, I'm Yuval. With me is Niv. We both work at AI21 labs, and today we're gonna talk about why RAG evaluation is broken and how to fix it. And the reason that RAG evaluation is broken comes from a lot of things, but mostly because how easy it is to build benchmarks the way humans are regular to think.

3. 0:35

   And what I mean by that is that most benchmarks comprise from local questions that has local answers. There is some sort of assumption that the answer lies in a certain chunk in the data.

4. 0:47

   Makes sense, right? What is the most natural way to build such a benchmark? To go read some long document, find a question that the answer lies somewhere inside, and then this is the golden answer.

5. 1:00

   And there are a lot of benchmarks who try to overcome that. It's a lot of multiple hope, uh, questions, uh, famously frames by Google, but those questions are not really realistic, right?

6. 1:12

   Something like, "If my future wife have the same name of the 15- first lady of the United States' mother and her surname..." Like, what is this? Does it even mean something?

7. 1:21

   Does it represent something real? Not really. This is all very manufactured and not really representative of the real world. There's also not a holistic way to test the entire system.

8. 1:35

   Most of the benchmarks are either retrieval-only benchmark, meaning that we try to retrieve the best segments or the best chunks for our database, and we assume that, again, the answer is in one, or two, or several of them.

9. 1:50

   Or it's generation-only benchmarks, which is essentially just grounding benchmark, right? Just to see that we can answer a question based on the contexts that are already in the prompt.

10. 2:01

    But what about chunking? What about parsing? What about specific cases that do not actually have this effect of the question, uh, directs us to a certain place in the data?

11. 2:16

    The answer is not as simple. And really, that's the main problem here, that benchmarks do not correlate with real world data, that real world data is messier. Each data is different, and this thing is not really generalized as well.

12. 2:32

    We have some sort of a vicious cycle when we are developing a RAG system, where we're building a RAG system for flawed benchmarks because we have to optimize for something.

13. 2:41

    And then when we get high scores, we celebrate about it, we write a nice post, whether on Slack, on LinkedIn, on Twitter, and then, "Look at us, we're so good."

14. 2:51

    But then when we actually give it to users or we test it on customer data, we see that they struggle. It's not as good as we hoped. So we create new benchmarks, optimize for them, and we're showing that we are the best, but those benchmarks probably have the same problems, and this goes over, and over, and over.

15. 3:11

    RAG pipelines with local questions and local answers. That's what we have in most of the cases. So how do we fix it?

16. 3:21

    Um, so we will not, uh, say, uh, how we will fix everything, but we'll try to think some things. Uh, so for example, if you take like a fin- uh, financial data, like finance bench or others, uh, like SEC filings or any other financial data, you may, uh, imagine that someone would ask a aggregative question such as,

17. 3:44

    "Which company has reported the highest quarterly revenue the most times?" Or, "In how many fiscal years did Apple exceed the $100 billion in annual revenue?" Um, or list all of something.

18. 3:57

    Uh, so, so you may imagine how RAG systems, uh, currently do on such questions because they are so limited, and they just grab the top K chunks that they get and try to compile an answer by that.

19. 4:12

    But, uh, when you encounter questions like all Fortune 500 companies, so imagine that you bring 10 chunks, uh, then maybe you will get another Fortune 500 company in the 11th one and the 12th one.

20. 4:27

    And, uh, for every N, you can, uh, consider the N plus one. Uh, so these problems really make, uh, RAG systems struggle.

21. 4:38

    Um, so we try to, to estimate how, uh, how do a current system do with such questions. So we built an, uh, a corpus, a small corpus of, uh, like 22, uh, 22 documents, uh, from the historical pages of FIFA World Cup, uh, in Wikipedia.

22. 4:56

    Uh, so this is our corpus, and, uh, we asked, uh, many questions about this corpus, such as, "Which team has won the FIFA World Cup the most times?" "In how many FIFA World Cup did Bra- Brazil participate?"

23. 5:08

    And, uh, we used, uh, we used the knowledge base, uh, the Wikidata and the Kaggle knowledge bases to answer these questions. And, uh, as you can see, uh, some...

24. 5:20

    a common RAG pipeline, like, uh, the, the first one you see on a LangChain or LlamaIndex, uh, and OpenAI responses both fail miserably. Uh, it's 5% and 11% of the questions answered correctly, and these are questions that you can find the answer to in very local segments, uh, as Yuval mentioned.

25. 5:44

    Um, so our idea to handle such corpuses, um, is take this unstructured corpus and convert it into a data structure, and then ask the question on top of this data structure.

26. 5:57

    Because essentially, the questions we talked about are questions are SQL questions, right? It's just like how many times, or it's counting questions, or max and min questions, or, uh, um, calculation questions, and you just, it does, it just doesn't, doesn't make sense to try to answer these questions by going over the pages, uh, or, uh, or specific

27. 6:22

    chunks of the pages. Um, so how do we do it? Uh, we split it in two, uh, similarly to how a regular RAG flow is done and where you invest the compute in the ingestion, and you try to do things quickly in the inference.

28. 6:38

    So in the ingestion phase, uh, we first cluster the document into sub-corpora. For example, we may have financial corpus and FIFA World Cup corpus that we, uh, do this clustering.

29. 6:51

    Then per each sub-corpus, we identify the schema that I- that, uh, represents this corpus. Uh, we populate the schema according to each and every document, and finally, we upload the, the results into, uh, an SQL DB.

30. 7:08

    Uh, so for example, if you have the co- the FIFA corpus, and you have a schema which, uh, is compiled of a year, the winner, the top three teams, the top scorer, and many other attributes.

31. 7:20

    And then, um, you use an LLM, uh, we... It's not just an LLM, it's like a pipeline, but, uh, its goal is to populate a schema for every document.

32. 7:31

    At the inference flow, uh, whenever we have a query, we identify the schema that, uh, is relevant for this query. For example, is it a financial question or a FIFA World Cup question?

33. 7:43

    And then we just do a regular text to SQL over our data, and we return the final answer.

34. 7:51

    Um, so here for the FIFA World Cup, which team has won the most, the FIFA World Cup the most times, it's just a simple SQL query.

35. 8:00

    Um, this approach does not come to solve everything, uh, not at all. Uh, first and foremost, not every corpus or query is, uh, relational DB material, and not every corpus is, uh, homogeneous, uh, in terms of the, uh, the attributes.

36. 8:18

    Uh, it doesn't necessarily contain a schema, uh, underlying it. Um, but we do see many, uh, such corpuses, uh, with our clients. Um, normalization. Even with a toy example such as the FIFA World Cup, we see the struggle with building the correct schema, uh, because, um, for example, in the host country,

37. 8:43

    you have West Germany, but then if I want to ask how many times did Germany won, does West, West Germany count or doesn't it? A host country, is it a single attribute or is it a list because South Korea and Japan hosted together the World Cup?

38. 8:58

    Um, so the normalization is an issue both during the ingestion and inference.

39. 9:04

    Um, abstinence or ambiguity. Um, what happens if a user asks me, "Did Real Madrid win the, uh, 2006 final?" It's not a World Cup, but I do see some things that I can try to do with my schema, and as you probably know, LLMs, uh, try, uh, uh, tend to try to, uh, please the users.

40. 9:28

    Um, the, the points in the ingestion where we try to cluster and they infer the schema, uh, there's some trade-off here on the complexity and how fine-grained we are, uh, and how much compute, uh, we would like to invest during the ingestion time.

41. 9:46

    Um, and finally, the text to SQL where whenever the schemas become complex, i- is just a complex issue, uh, which is, uh, very known and studied. Um, so the main takeaways, uh, are just that, uh, RAG is not a one-size-fits-all system, and, uh, you have to account for every client, uh, separately.

42. 10:08

    Um, there, there are, of course, clusters, but we should, um, we should note that, uh, the, uh, regular pipeline of chunking, embedding, uh, retrieving, re-ranking is not good enough for many questions.

43. 10:22

    Um, existing benchmarks fail to capture these, uh, these, uh, use cases. Uh, as Yuval mentioned, they are s- very limited. Um, and the... In order to solve the problems, we may need to go beyond standard RAG for specific settings.

44. 10:41

    If you thought this is interesting and you want to learn more about structured RAG and the problems with RAG evaluation, please check out our full episode of YAAP Podcast by AI21.

45. 10:52

    Thank you.