AI Engineer World's Fair 2025
RAG Evaluation Is Broken! Here's Why (And How to Fix It)
Read the talk
RAG Evaluation Breaks When the Answer Requires Every Record
Local-answer benchmarks reward finding the right passage. Aggregate questions expose a different requirement: complete evidence, structured extraction, and explicit query semantics.
From a talk by Yuval Belfer and Niv Granot
Before you start: Familiarity with RAG retrieval and basic SQL will help; the article explains the aggregation query.
Local answers hide system failures
Read a long document, find a passage containing a fact, and write a question whose answer is that fact. This natural way to build a benchmark embeds a consequential assumption: the answer lives in a local chunk. A retrieval-augmented generation system can score well by finding that chunk and restating it, without demonstrating that it can answer questions about the corpus as a whole. This is the opening problem Yuval Belfer and Niv Granot of AI21 Labs identify beneath the apparent maturity of RAG.
Multi-hop questions try to escape that locality by requiring several connected facts. Belfer points to Google’s FRAMES, then paraphrases a dataset question that constructs a future wife’s name from facts about a US first lady’s family. His objection is about realism: a manufactured chain of lookups may exercise reasoning without resembling a customer’s work. FRAMES explicitly targets end-to-end factuality, retrieval, and reasoning; the criticism here is that those capabilities alone do not establish representative question design.
Other evaluations isolate one part of the pipeline:
| Evaluation | What it tests | What it assumes |
|---|---|---|
| Retrieval only | Finding relevant chunks | The answer is in the selected chunks |
| Generation only | Answering from supplied context | The necessary context is already in the prompt |
Neither view by itself tests the entire path from raw documents to a useful answer. Parsing and chunking can affect what evidence survives, and some questions do not point toward a particular location at all. Real customer corpora are messy and differ from one another, so success on a convenient benchmark need not generalize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The optimization loop preserves the assumption
Teams need something to optimize, so they build a RAG system around the available benchmark. Scores rise, the team celebrates, and the result looks like progress. Then users ask questions over customer data, and the system struggles. The team builds a replacement benchmark and optimizes again—but if the replacement still consists of local questions with local answers, the same failure repeats. Changing the benchmark does not help if its underlying assumption stays fixed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Aggregate questions need complete evidence
Financial documents make the missing question types concrete. Starting from corpora such as FinanceBench, SEC filings, or similar financial material, Granot proposes questions a user might reasonably ask:
- Repeated comparisons: Which company reported the highest quarterly revenue the most times?
- Threshold counts: In how many fiscal years did Apple exceed $100 billion in annual revenue?
- Exhaustive lists: List every entity satisfying a condition.
These are illustrative questions, not identified FinanceBench items. The financial slide adds a specific list example: Fortune 500 companies that ranked in the top three but never first. Each question requires considering a set of records, not simply locating a sentence.
A conventional pipeline retrieves the top K chunks and asks a model to compile an answer. That can supply relevant examples without supplying all the evidence required for a count or an exhaustive list. If ten chunks mention Fortune 500 companies, an eleventh or twelfth may introduce another qualifying company. For any chosen N, relevant evidence may remain in chunk N + 1. Increasing K can improve coverage, but the cutoff itself does not establish completeness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small World Cup test exposes the gap
To probe this failure, the speakers assembled a corpus of 22 documents from Wikipedia’s historical FIFA World Cup pages. They asked questions such as which team had won the tournament most often and how many World Cups Brazil had participated in. Reference answers came from Wikidata and Kaggle knowledge bases. The task was to answer questions across the tournament corpus, rather than extract answers from selected local passages.
The speakers reported the following results on their aggregate-question test:
| System | Questions answered correctly |
|---|---|
| Common introductory RAG pipeline | 5% |
| OpenAI Responses | 11% |
The first baseline is described as the kind of introductory pipeline found in LangChain or LlamaIndex. These are historical results from this particular experiment, not general product accuracy figures. The talk does not supply the question count, scoring procedure, model versions, retrieval settings, or exact OpenAI Responses configuration, and it does not report accuracy for the structured approach introduced next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract structure before answering
The proposed response is to convert the unstructured corpus into a data structure and ask questions over that structure. Counts, maxima, minima, and calculations are natural database operations. Once the relevant records exist, the system need not reconstruct an aggregate from whichever passages retrieval happens to return. As in ordinary RAG, the architecture separates ingestion from inference: invest computation while preparing the corpus so that answering can be fast.
Ingestion proceeds through four stages:
- Cluster the documents into sub-corpora. Financial records and World Cup pages, for example, belong to different groups.
- Identify a schema for each sub-corpus. Choose attributes that represent its recurring information.
- Populate the schema from every document. Extract records across the corpus rather than only from documents retrieved for one question.
- Store the results in a SQL database. The extracted representation becomes the basis for later queries.
For World Cup pages, the schema can include the year, winner, top three teams, top scorer, and other attributes. Granot stresses that populating it is the job of an LLM-based pipeline, not merely a single model call. The ingestion process must turn each document into a record that fits the shared representation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translate the question into an aggregation
At query time, the system first identifies the relevant schema: is this a financial question, a World Cup question, or something else? It then applies text-to-SQL to the extracted data and returns the final answer. Retrieval of a few passages is replaced, for these questions, by a formal query over the prepared records.
The question about the team with the most World Cup wins becomes a grouping and counting operation. With a world_cups table containing one row per tournament and a winner column, the core SQL is:
sql
SELECT
winner,
COUNT(*) AS wins
FROM world_cups
GROUP BY winner
ORDER BY wins DESC
LIMIT 1;
GROUP BY gathers tournaments by winner, COUNT(*) counts the rows in each group, and descending order puts the largest count first. LIMIT 1 selects one leading team. The aggregation considers the stored tournament rows instead of only the top-ranked text chunks; its correctness still depends on the records extracted during ingestion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structure creates its own correctness requirements
This approach requires a useful relational representation. Not every corpus has homogeneous attributes, not every collection implies a shared schema, and not every question belongs in a relational database. The speakers encounter suitable corpora among their clients, but they do not present structured extraction as a universal replacement for RAG.
Even the World Cup example contains normalization decisions that change the meaning of an answer:
- Historical identity: When counting Germany’s wins, should West Germany count as Germany? Grouping raw names and grouping a normalized identity can produce different totals.
- Field cardinality: A host-country field cannot always hold a single country. South Korea and Japan hosted jointly, so the representation must accommodate multiple hosts.
These decisions affect both ingestion and inference. The stored representation and the interpretation of the user’s question must agree about what is being counted.
Schema selection also needs a way to reject a poor fit. A question about whether Real Madrid won a 2006 final is not a World Cup question, even though it contains familiar football terms and a year. An LLM inclined to satisfy the user may try to force it into the available schema and produce an unsupported answer. Granot names ambiguity and abstention as challenges; the talk does not supply an implementation for resolving them.
There is also an ingestion trade-off. Finer-grained clustering and schema inference can represent more distinctions, but they increase complexity and the computation invested before a query arrives. At inference time, more complex schemas make text-to-SQL itself harder. Moving aggregation into SQL therefore changes the engineering problem: schema design, extraction, routing, and query generation all become part of answer correctness.
Evaluation and architecture must match the client’s questions. The familiar sequence of chunking, embedding, retrieval, and re-ranking is insufficient for many tasks, and benchmarks that omit those tasks conceal the limitation. Where the work demands counts, comparisons, and complete lists over recurring records, answering may require a structured representation beyond standard RAG. The benchmark must exercise those same requirements if its scores are to guide useful improvements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original paper introducing a benchmark for factuality, retrieval and multi-hop reasoning in end-to-end RAG.
Browse benchmark questions, answers and supporting Wikipedia links, including the future-wife example.
Public financial QA examples with annotated answers, evidence, source documents and an evaluation notebook.
Podcast episode page for Yuval and Niv's discussion of evaluation failures, chunking and structured RAG.
Updates since the talk
Later research coauthored by Niv Granot formalizes S-RAG and introduces HOTELS and WORLD CUP aggregation datasets.
Read the complete timestamped transcript
- 0:00
It seems that RAG is so 2023, and everywhere you go, you see, "My first RAG," and, "RAG pipelines," and everything is RAG related, as if it's already solved. But actually, not really.
- 0:14
Uh, I'm Yuval. With me is Niv. We both work at AI21 labs, and today we're gonna talk about why RAG evaluation is broken and how to fix it. And the reason that RAG evaluation is broken comes from a lot of things, but mostly because how easy it is to build benchmarks the way humans are regular to think.
- 0:35
And what I mean by that is that most benchmarks comprise from local questions that has local answers. There is some sort of assumption that the answer lies in a certain chunk in the data.
- 0:47
Makes sense, right? What is the most natural way to build such a benchmark? To go read some long document, find a question that the answer lies somewhere inside, and then this is the golden answer.
- 1:00
And there are a lot of benchmarks who try to overcome that. It's a lot of multiple hope, uh, questions, uh, famously frames by Google, but those questions are not really realistic, right?
- 1:12
Something like, "If my future wife have the same name of the 15- first lady of the United States' mother and her surname..." Like, what is this? Does it even mean something?
- 1:21
Does it represent something real? Not really. This is all very manufactured and not really representative of the real world. There's also not a holistic way to test the entire system.
- 1:35
Most of the benchmarks are either retrieval-only benchmark, meaning that we try to retrieve the best segments or the best chunks for our database, and we assume that, again, the answer is in one, or two, or several of them.
- 1:50
Or it's generation-only benchmarks, which is essentially just grounding benchmark, right? Just to see that we can answer a question based on the contexts that are already in the prompt.
- 2:01
But what about chunking? What about parsing? What about specific cases that do not actually have this effect of the question, uh, directs us to a certain place in the data?
- 2:16
The answer is not as simple. And really, that's the main problem here, that benchmarks do not correlate with real world data, that real world data is messier. Each data is different, and this thing is not really generalized as well.
- 2:32
We have some sort of a vicious cycle when we are developing a RAG system, where we're building a RAG system for flawed benchmarks because we have to optimize for something.
- 2:41
And then when we get high scores, we celebrate about it, we write a nice post, whether on Slack, on LinkedIn, on Twitter, and then, "Look at us, we're so good."
- 2:51
But then when we actually give it to users or we test it on customer data, we see that they struggle. It's not as good as we hoped. So we create new benchmarks, optimize for them, and we're showing that we are the best, but those benchmarks probably have the same problems, and this goes over, and over, and over.
- 3:11
RAG pipelines with local questions and local answers. That's what we have in most of the cases. So how do we fix it?
- 3:21
Um, so we will not, uh, say, uh, how we will fix everything, but we'll try to think some things. Uh, so for example, if you take like a fin- uh, financial data, like finance bench or others, uh, like SEC filings or any other financial data, you may, uh, imagine that someone would ask a aggregative question such as,
- 3:44
"Which company has reported the highest quarterly revenue the most times?" Or, "In how many fiscal years did Apple exceed the $100 billion in annual revenue?" Um, or list all of something.
- 3:57
Uh, so, so you may imagine how RAG systems, uh, currently do on such questions because they are so limited, and they just grab the top K chunks that they get and try to compile an answer by that.
- 4:12
But, uh, when you encounter questions like all Fortune 500 companies, so imagine that you bring 10 chunks, uh, then maybe you will get another Fortune 500 company in the 11th one and the 12th one.
- 4:27
And, uh, for every N, you can, uh, consider the N plus one. Uh, so these problems really make, uh, RAG systems struggle.
- 4:38
Um, so we try to, to estimate how, uh, how do a current system do with such questions. So we built an, uh, a corpus, a small corpus of, uh, like 22, uh, 22 documents, uh, from the historical pages of FIFA World Cup, uh, in Wikipedia.
- 4:56
Uh, so this is our corpus, and, uh, we asked, uh, many questions about this corpus, such as, "Which team has won the FIFA World Cup the most times?" "In how many FIFA World Cup did Bra- Brazil participate?"
- 5:08
And, uh, we used, uh, we used the knowledge base, uh, the Wikidata and the Kaggle knowledge bases to answer these questions. And, uh, as you can see, uh, some...
- 5:20
a common RAG pipeline, like, uh, the, the first one you see on a LangChain or LlamaIndex, uh, and OpenAI responses both fail miserably. Uh, it's 5% and 11% of the questions answered correctly, and these are questions that you can find the answer to in very local segments, uh, as Yuval mentioned.
- 5:44
Um, so our idea to handle such corpuses, um, is take this unstructured corpus and convert it into a data structure, and then ask the question on top of this data structure.
- 5:57
Because essentially, the questions we talked about are questions are SQL questions, right? It's just like how many times, or it's counting questions, or max and min questions, or, uh, um, calculation questions, and you just, it does, it just doesn't, doesn't make sense to try to answer these questions by going over the pages, uh, or, uh, or specific
- 6:22
chunks of the pages. Um, so how do we do it? Uh, we split it in two, uh, similarly to how a regular RAG flow is done and where you invest the compute in the ingestion, and you try to do things quickly in the inference.
- 6:38
So in the ingestion phase, uh, we first cluster the document into sub-corpora. For example, we may have financial corpus and FIFA World Cup corpus that we, uh, do this clustering.
- 6:51
Then per each sub-corpus, we identify the schema that I- that, uh, represents this corpus. Uh, we populate the schema according to each and every document, and finally, we upload the, the results into, uh, an SQL DB.
- 7:08
Uh, so for example, if you have the co- the FIFA corpus, and you have a schema which, uh, is compiled of a year, the winner, the top three teams, the top scorer, and many other attributes.
- 7:20
And then, um, you use an LLM, uh, we... It's not just an LLM, it's like a pipeline, but, uh, its goal is to populate a schema for every document.
- 7:31
At the inference flow, uh, whenever we have a query, we identify the schema that, uh, is relevant for this query. For example, is it a financial question or a FIFA World Cup question?
- 7:43
And then we just do a regular text to SQL over our data, and we return the final answer.
- 7:51
Um, so here for the FIFA World Cup, which team has won the most, the FIFA World Cup the most times, it's just a simple SQL query.
- 8:00
Um, this approach does not come to solve everything, uh, not at all. Uh, first and foremost, not every corpus or query is, uh, relational DB material, and not every corpus is, uh, homogeneous, uh, in terms of the, uh, the attributes.
- 8:18
Uh, it doesn't necessarily contain a schema, uh, underlying it. Um, but we do see many, uh, such corpuses, uh, with our clients. Um, normalization. Even with a toy example such as the FIFA World Cup, we see the struggle with building the correct schema, uh, because, um, for example, in the host country,
- 8:43
you have West Germany, but then if I want to ask how many times did Germany won, does West, West Germany count or doesn't it? A host country, is it a single attribute or is it a list because South Korea and Japan hosted together the World Cup?
- 8:58
Um, so the normalization is an issue both during the ingestion and inference.
- 9:04
Um, abstinence or ambiguity. Um, what happens if a user asks me, "Did Real Madrid win the, uh, 2006 final?" It's not a World Cup, but I do see some things that I can try to do with my schema, and as you probably know, LLMs, uh, try, uh, uh, tend to try to, uh, please the users.
- 9:28
Um, the, the points in the ingestion where we try to cluster and they infer the schema, uh, there's some trade-off here on the complexity and how fine-grained we are, uh, and how much compute, uh, we would like to invest during the ingestion time.
- 9:46
Um, and finally, the text to SQL where whenever the schemas become complex, i- is just a complex issue, uh, which is, uh, very known and studied. Um, so the main takeaways, uh, are just that, uh, RAG is not a one-size-fits-all system, and, uh, you have to account for every client, uh, separately.
- 10:08
Um, there, there are, of course, clusters, but we should, um, we should note that, uh, the, uh, regular pipeline of chunking, embedding, uh, retrieving, re-ranking is not good enough for many questions.
- 10:22
Um, existing benchmarks fail to capture these, uh, these, uh, use cases. Uh, as Yuval mentioned, they are s- very limited. Um, and the... In order to solve the problems, we may need to go beyond standard RAG for specific settings.
- 10:41
If you thought this is interesting and you want to learn more about structured RAG and the problems with RAG evaluation, please check out our full episode of YAAP Podcast by AI21.
- 10:52
Thank you.