# open-rag-eval: RAG Evaluation without "golden" answers.

Ofer Mendelevitch

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/1cQlnfwmIdU-open-rag-eval-rag-evaluation-without-golden) · [Video](https://www.youtube.com/watch?v=1cQlnfwmIdU)

## Open RAG Eval: Evaluating Retrieval and Answers Without Golden References

Open RAG Eval captures a pipeline’s retrieved passages and answers, then evaluates relevance, information coverage, citation support, and response consistency without hand-written golden references.

Familiarity with RAG—retrieving passages and using them to generate an answer—is helpful.

### The golden-reference bottleneck

[0:00](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=0s)

How do you evaluate a RAG system when you have useful questions but no curated set of correct answers or ideal retrieved passages? Creating those references adds work every time the evaluation set grows. [Open RAG Eval](https://github.com/vectara/open-rag-eval), introduced here by Ofer Mendelevitch of Vectara developer relations, addresses that bottleneck: **evaluate retrieval and generation without manually supplied golden chunks or golden answers**. The opening slide’s QR code provides access to the repository.

The project draws on research conducted with the University of Waterloo’s Jimmy Lin Lab. Its approach replaces the requirement for curated references with automated assessments of what the pipeline actually retrieves and generates. Understanding those assessments matters: each measures a different part of the RAG system.

### From queries to evaluation files

[0:43](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=43s)

Start with 10, 100, or 1,000 queries that matter to your RAG application. These are example evaluation-set sizes, not throughput claims. The workflow then follows the data through three stages:

1. **Capture the pipeline’s behavior.** A RAG connector collects the actual retrieved chunks and generated answers. The demonstrated integrations include Vectara, LangChain, and LlamaIndex.
2. **Evaluate the captured outputs.** The resulting RAG outputs feed metrics grouped into evaluators. This separates collecting a pipeline’s behavior from assessing it.
3. **Write evaluation files.** Evaluators emit files containing the results for inspection and comparison.

The connector is therefore the boundary between your application and the evaluation machinery: it supplies what the system did, without requiring you to author the ideal answer first.

![Architecture diagram linking queries, a RAG connector and pipeline, RAG outputs, evaluators with metrics, model boxes, and RAG evaluations.](https://ai.engineer/api/talk-essay-media/e3c2ec20dfc598e9f0d20aa4c8778066d5e9d0cf46caa3c69a25e7994c240730)

Open-RAG-Eval connects a RAG pipeline to evaluators, metrics, and models.

### Retrieval relevance with UMBRELA

[1:34](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=94s)

The package introduces four complementary assessments: [UMBRELA](https://arxiv.org/abs/2406.06519) for retrieval, [AutoNuggetizer](https://arxiv.org/abs/2411.09607) for generation, citation faithfulness, and hallucination detection using a Vectara model. Keeping these dimensions separate helps distinguish retrieving useful evidence from producing a well-supported answer.

**UMBRELA scores individual passages against the query**, without requiring golden chunks. Its relevance scale runs from 0 to 3:

| Score | Meaning described in the talk |
| --- | --- |
| 0 | Passage has nothing to do with the query |
| 3 | Passage is dedicated to the query and contains the exact answer |

The endpoints distinguish irrelevant retrieval from a passage that directly supplies the requested information.

Mendelevitch reports that UMBRELA correlates well with human judgment, citing the Waterloo research. That supports using automated relevance assessment without golden chunks, but the consequential distinction is between agreement in evaluating systems and identical labels for every passage. The associated research supports system-ranking agreement; the talk supplies no correlation coefficient or experimental conditions for its broader qualitative claim.

### Answer coverage with AutoNuggetizer

[2:52](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=172s)

AutoNuggetizer evaluates generation by breaking the information an answer should cover into **nuggets**, or atomic information units. Instead of comparing the generated response with a single golden answer, it checks coverage of selected units:

1. **Create nuggets.** Produce atomic units of information for the evaluation.
2. **Prioritize them.** Assign each nugget a `vital` or `okay` importance rating, order the nuggets, and select the top 20.
3. **Judge answer coverage.** An LLM judge examines the RAG response to determine whether each selected nugget is fully or partially supported by the answer.

The top-20 limit is a selection setting, not an accuracy result. The mechanism measures information coverage: an answer can cover one important point completely and another only partly.

![Three-step AutoNuggetizer slide: create nuggets with vital or okay ratings; the slide labels selection as “Top 20 vital nuggets”; an LLM judge checks full or partial answer support.](https://ai.engineer/api/talk-essay-media/8c16a4fa3313fe4eb7eacbd4b16b8be2f0c01b320b1dd5f8b87f21a654a90644)

AutoNuggetizer creates nuggets, orders them by importance, selects up to 20, and checks answer support.

For a small Python implementation of the importance-ordering step, retain each nugget’s text and rating together, then select the first 20 after sorting. This helper represents selection only; nugget creation and support judgments remain separate operations.

```python
from typing import Literal, TypedDict

class Nugget(TypedDict):
    text: str
    importance: Literal["vital", "okay"]


def select_nuggets(nuggets: list[Nugget]) -> list[Nugget]:
    priority = {"vital": 0, "okay": 1}
    ordered = sorted(
        nuggets,
        key=lambda nugget: priority[nugget["importance"]],
    )
    return ordered[:20]
```

The original nugget records survive the selection, so the subsequent judge can assess each retained unit against the generated answer.

### Citation support and whole-response consistency

[3:25](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=205s)

Mendelevitch points to the papers for fuller method details before moving to two support checks. These operate at different scopes:

| Assessment | Scope | Question |
| --- | --- | --- |
| Citation faithfulness | A citation and its associated claim | Does the cited passage support the claim? |
| Hallucination detection | The entire response | Does the response align with retrieved content? |

Citation faithfulness distinguishes **full support, partial support, and no support**. A citation’s presence alone does not establish that its passage substantiates what the answer says.

For the response-level check, **[HHEM](https://huggingface.co/vectara/hallucination_evaluation_model)**, Vectara’s hallucination detection model, assesses alignment between the answer and retrieved evidence. Here, support is relative to that evidence: a statement can be true elsewhere yet unsupported by the passages supplied to the model. This complements nugget coverage by asking whether the answer stays grounded in what was retrieved.

### Inspecting the results

[3:54](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=234s)

Once evaluation finishes, you can inspect the result files directly. For easier comparison, the talk shows a hosted interface and describes dragging the evaluation files onto `open-evaluation.ai`. The populated view lists queries alongside retrieval and generation scores, with groups of colored results arranged side by side. This makes the separate evaluation dimensions visible together without requiring manual inspection of every file.

![Slide showing the Open Evaluation interface with query rows, two groups of colored metric scores, and aggregate scores above them.](https://ai.engineer/api/talk-essay-media/17dcea637db52a6575632190cab514a40ab5b58bb4b6e2ad917c12b54bf65a44)

The openevaluation.ai interface displays evaluation results side by side.

### Tuning a pipeline and extending its connectors

[4:26](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=266s)

The intended use is to optimize and tune a RAG pipeline while keeping its evaluation methods inspectable. Open source provides access to the metric implementations, so a score need not remain an opaque judgment: you can examine how the assessment works.

Vectara, LangChain, and LlamaIndex provide the demonstrated integration paths. For a custom pipeline, the extension point is the connector that captures retrieved chunks and generated answers. Mendelevitch invites issues and pull requests for additional connectors, allowing other pipelines to feed the same evaluator workflow.

![Summary slide listing Apache 2.0 licensing, Vectara, LangChain and LlamaIndex connectors, an invitation to contribute connectors, and no need for golden chunks or answers.](https://ai.engineer/api/talk-essay-media/9c9bcde3c7ce85ee403f51caed4b4ad6f86e825569e731c19a02412edd9f7e47)

Open-RAG-Eval summarizes its open-source license, connectors, and evaluation without golden references.

## Resources

- [Open RAG Eval](https://github.com/vectara/open-rag-eval) — Python toolkit with RAG connectors, evaluation metrics, examples and contribution instructions. Update from the repository, checked 2026-08-28: its viewer link points to openevaluation.ai and JSON is the default viewer format. Viewer functionality was not tested.
- [Towards a Gold Standard for RAG evaluation](https://www.vectara.com/blog/towards-a-gold-standard-for-rag-evaluation) — Project introduction coauthored by Ofer Mendelevitch, explaining the workflow and four evaluation dimensions.
- [UMBRELA](https://arxiv.org/abs/2406.06519) — Original paper on automated retrieval relevance judgments using GPT-4o.
- [AutoNuggetizer: Initial TREC 2024 Results](https://arxiv.org/abs/2411.09607) — Automatic information-nugget creation and answer coverage evaluation, with initial comparisons against human assessment.
- [Large-Scale Validation of LLM Relevance Assessments](https://arxiv.org/abs/2411.08275) — TREC 2024 study comparing system rankings derived from automatic and manual relevance judgments.
- [Open RAG Eval Metrics Documentation](https://github.com/vectara/open-rag-eval/blob/dev/METRICS.md) — Current metric inputs, scoring categories, formulas and configuration details.
- [Vectara HHEM Model Card](https://huggingface.co/vectara/hallucination_evaluation_model) — Model documentation and usage examples for assessing whether generated text is supported by supplied evidence.

## Slides

### Open RAG Eval

Open RAG Eval aims to make RAG evaluation scalable without golden answers or chunks. The research-backed project was developed with the University of Waterloo and supports side-by-side comparison.

[0:15](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=15s) [Overview of Open RAG Eval's research foundation, reference-free evaluation and side-by-side comparison.](https://assets.ai.engineer/sites/aiecode2025/sha256/81592962c061e707f995b20f08ddc187d013f18af4e9d97bc0e57491a3dc42bc.webp)

Open RAG Eval

Research-Backed,
Community Driven

No Golden
Answers Needed

Instant Insight with
Side-by-Side
Comparison

vectara
UNIVERSITY OF WATERLOO

### Open RAG Eval Architecture

A RAG connector collects chunks and answers from a pipeline for a set of queries. Evaluators then run grouped metrics on the RAG output and produce evaluation files.

[0:41](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=41s) [Diagram of queries flowing through a RAG connector and pipeline into evaluators, models and evaluation outputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/0bb3ea88c6b4d86382fbc0ab640e301ee61d384fbcacde353aaa5f5f2772ba05.webp)

Open Rag Eval - Open-Source Rag Evaluation package

Queries
RAG connector
RAG pipeline
Vectara
Langchain
LlamaIndex
etc...

Open-RAG-Eval
RAG Output
Evaluator
Metric 1
Metric 2
Evaluator
Metric 3
Model A
Model B
...
Model D
RAG Evaluations

### The Metrics

UMBRELA evaluates retrieval without golden chunks, while AutoNuggetizer evaluates generation without golden answers. Two other metrics check citation correctness and whether responses are consistent with the chunks.

[1:33](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=93s) [Overview of UMBRELA, AutoNuggetizer, citation faithfulness and hallucination detection.](https://assets.ai.engineer/sites/aiecode2025/sha256/18a4d4f7ab1932bc86f36b1002f5876bf048d681209c813960c0d1b772e17493.webp)

The Metrics

• UMBRELA: retrieval evaluation without “golden chunks”
• AutoNuggetizer: generation evaluation without “golden answers”
• Citation Faithfulness: are the citations in the response really correct?
• Hallucination detection: is the response factually consistent with the chunks?

### UMBRELA (Retrieval)

UMBRELA scores passages from 0 (unrelated) to 3 (dedicated to the query and containing the exact answer). Intermediate scores cover related or partially useful passages. The speaker says research found a correlation with human judgments.

[2:06](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=126s) [UMBRELA rates passage relevance from zero to three, with descriptions of each score.](https://assets.ai.engineer/sites/aiecode2025/sha256/16c13f6adf961bece4c803bf80b9bba5c4ff2ce9f6ca16cb8d42f11c0f551423.webp)

UMBRELA (Retrieval)

• On a scale of 0-3 measures how relevant each passage is to the query.
• Research shows UMBRELA correlates well with human judgments.

0 = represent that the passage has nothing to do with the query

1 = represents that the passage seems related to the query but does not answer it,

2 = represents that the passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information and

3 = represents that the passage is dedicated to the query and contains the exact answer.

### AutoNuggetizer (Generation)

AutoNuggetizer turns retrieved passages into rated atomic facts, selects the top 20 vital nuggets, then asks an LLM judge whether the RAG answer fully or partially supports each one.

[2:51](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=171s) [Three steps for creating and rating nuggets, selecting vital nuggets and assessing their support in a RAG response.](https://assets.ai.engineer/sites/aiecode2025/sha256/f5f69b09c545241f53a6a2c9d7e77ee0f6b24644ae8da0961e22e481e00c2211.webp)

AutoNuggetizer (Generation)

• Step 1: Nugget Creation -
  a. An LLM judge is asked to create a set of ‘Nuggets’ of information based on retrieved passages. These are ‘atomic facts’ that should appear in your responses.
  b. Each nugget gets assigned a ‘vital’ or ‘okay’ rating depending on whether it must absolutely be present in the generated response to the query or whether it’s good to have but not absolutely necessary.

• Step 2: Top 20 vital nuggets are selected from Step 1.

• Step 3: LLM judge analyzes the RAG response to determine if each of the selected Nuggets is fully supported or partially supported by the answer.

### Citation Faithfulness

An LLM judge compares each cited sentence in a RAG response with its cited passage, then labels the support as full, partial, or absent.

[3:29](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=209s) [An LLM judge compares cited sentences with their cited passages and labels support as full, partial, or absent.](https://assets.ai.engineer/sites/aiecode2025/sha256/85588a5b375860896153f4eaf9135ce575f436d1ceea9ce45ecd35a785c315a0.webp)

Citation Faithfulness

• Generated RAG response is broken down into sentences that have a citation, then each sentence is compared by the LLM judge to the passage that the citation claims it came from.
• Labels: Full Support, Partial Support, or No Support
• Focused on citation-to-passage fidelity

### Open Evaluation UI for Open-RAG-Eval

Drag evaluation files into the UI to compare reports side by side, with retrieval relevance and generation scores for each query.

[3:42](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=222s) [A comparison interface displays queries and retrieval and generation scores for two evaluation reports.](https://assets.ai.engineer/sites/aiecode2025/sha256/3c24281e17e7d9c04f50bd02d3c3b4faa8a92571bea5b9c10a150720f5b8ec82.webp)

openevaluation.ai: UI for Open-RAG-Eval

vectara_query_config_option_a.csv
Retrieval: 2.73 Relevance
Generation: 0.59 Groundedness, 0.7 Factuality, 0.84 Citations

vectara_query_config_option_b.csv
Retrieval: 2.73 Relevance
Generation: 0.68 Groundedness, 0.69 Factuality, 0.54 Citations

Scores below are shown in the order Relevance, Groundedness, Factuality, Citations, for options A and B respectively.

Is there overshooting in the optimal strategy if [illegible]?
A: 3, 0.5, 0.8, 1
B: 3, 0.6, 0.81, 0.33

What is the Dareplane platform used for?
A: 3, 1, 0.98, 0.5
B: 3, 0.95, 0.95, 0.6

How does rising demand uncertainty affect the placement of safety stock in the supply chain?
A: 3, 0.75, 0.22, 1
B: 3, 0.7, 0.7, 0.14

Why might focusing on the sample distribution be preferable to assuming I.I.D. sampling?
A: 3, 0.5, 0.91, 1
B: 3, 0.8, 0.87, 0.53

What is an excluded game in the context of extensive-form games?
A: 3, 0, 0.91, 1
B: 3, 0.83, 0.81, 0.86

Does the diquark structure have a spin of 1?
A: 3, 0.33, 0.25, 1
B: 3, 1, 0.72, 0.57

What is the primary advantage of the ‘mirrored’ optical lever?
A: 3, 0.75, 0.95, 1
B: 3, 0.5, 0.94, 0.83

How does the rank-dependent peer effect model differ from traditional models?
A: 3, 0.83, 0.86, 1
B: 3, 1, 0.67, 0.8

Interface text:
Open Evaluation
Add reports
Help
Expand all

### Summary

Open-RAG-Eval is open source and includes connectors for Vectara, LangChain, and LlamaIndex. It avoids requiring golden chunks or answers and invites contributions of other connectors.

[4:24](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=264s) [Summary of Open-RAG-Eval as an open-source, community-driven tool with connectors and no need for golden chunks or answers.](https://assets.ai.engineer/sites/aiecode2025/sha256/5620690ae65a83657a2f67e2954391b140a06bbf4c39c382610073ba384dc091.webp)

Summary

• RAG Evaluation has been difficult
• Open-RAG-Eval is fully open source (Apache 2.0 licensed) and community driven.
• Includes “connectors” to Vectara, LangChain, LlamaIndex
  ◦ Please consider contributing other connectors
• No need for golden chunks or golden answers → scalable and practical

## Full transcript

[0:00](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=0s) Hi, everyone. My name is Ofer from Vectara. I run our developer relations, and I'm gonna talk to you today about Open RAG Eval, a new open-source project for quick and scalable RAG evaluation.

[0:12](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=12s) You can use the QR code here to get to the repo directly.

[0:16](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=16s) So what is Open RAG Eval? It's a open source project that is aiming to solve one of the major problems of RAG evaluation, the fact that you require golden answers or golden chunks to do anything, which is really non-scalable.

[0:31](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=31s) It is research-backed, so it's work that we've done with collaboration with University of Waterloo, the Jimmy Lin Lab specifically. And let me show you how it works.

[0:43](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=43s) So generally, this is kind of an architecture view of what, what it does. So you start with a set of queries. It could be ten or a hundred or a thousand queries that you've collected that are important for your RAG system.

[0:55](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=55s) We have a RAG connector, and that really collects all the actual information, the actual chunks, the actual answers that are generated by a RAG pipeline. We have one for Vectara, for LangChain, for LlamaIndex, and a growing number of other connectors.

[1:09](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=69s) And those connectors generally generate these outputs shown here, which is the, the RAG outputs. From there, you run the actual evaluation, which runs a bunch of metrics. Metrics are grouped into evaluators, and that's kinda how the, the internal architecture works.

[1:24](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=84s) And those evaluators generate RAG evaluation files. And that has everything you need to know to really evaluate your RAG pipeline.

[1:34](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=94s) Now, the metrics are where it gets really interesting. How do we do this thing with no golden answers? Well, there's a few metrics I wanna mention here that are part of this package.

[1:44](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=104s) One is called UMBRELA. It's really allows you to do retrieval without the golden chunks. The other one is called AutoNuggetizer. I'll talk more about this for generation. We have the citation faithfulness, which measures whether citations in the response are really correct, and hallucination detection, which is based on Vectara's hallucination detection model.

[2:05](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=125s) So let's go into each of these in detail. UMBRELA is a retrieval metric, as I mentioned, and what it does is it actually takes a chunk and gives it a score between zero and three, zero being that this chunk or passage has nothing to do with the query, and three being that it's dedicated to the query and

[2:24](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=144s) contains the exact answer. Now, the nice thing about this is not just the scale here, which is pretty self-explanatory, but the fact that the research done by the University of Waterloo Lab of Jimmy Lin shows that if you use this approach, it correlates well with human judgment, and that is really, really powerful.

[2:44](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=164s) So if you use this, even without the golden chunks, you know that the results will be good.

[2:52](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=172s) Now, AutoNuggetizer is the generation metric. And again, it doesn't require the golden answers in this case. And it works a little bit differently. There's three steps here. The first step is nugget creation, where you create these atomic units called nuggets.

[3:07](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=187s) Then for each nugget, you assign a vital or okay rating. You sort of sort these nuggets by... get the top twenty. And then there's a step where an LLM judge analyzes the response you get from your RAG to determine if each of the selected nuggets is either fully supported or partially supported by the answer.

[3:25](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=205s) Again, you can see all of this in the papers in, in much more detail. The third metric is citation faithfulness, which essentially measures whether the citation, the passage, you know, is high fidelity.

[3:37](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=217s) It's, it's, it's fully supported, partially supported, or there's no support for the citation in the response. And then the last one is really using HHEM, Vectara's hallucination detection model, to check if the entire response aligns with the retrieved content.

[3:54](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=234s) So that's, that's the metrics we have, and there's also a very cool user interface for this. So once you finish with your running your evaluation, you can, of course, look at the files yourself, but that's tends to be pretty complex.

[4:08](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=248s) So you can drag and drop those files onto open-evaluation.ai, as shown here, and you get this really, really cool UI that shows you all the queries you ran and all the things that you wanna compare between the retrieval scores, the different generation scores, et cetera.

[4:26](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=266s) And that's it. So I encourage you to take a look at this. It's a very powerful package, can help you optimize and, and tune your, your RAG pipeline. Again, it's open source, so all the source is open.

[4:37](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=277s) You can take a look and see how it works. This drives a lot of transparency, so the metrics are very clear in how they work. As I said, it includes connectors to Vectara, LangChain, and LlamaIndex.

[4:47](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=287s) But if you have your own RAG pipeline or some other RAG pipeline you wanna contribute, we're very much in favor of contributing other issues or PRs for other connectors.

[4:57](https://www.youtube.com/watch?v=1cQlnfwmIdU&t=297s) And let us know if you have any questions. This is about Open RAG Eval, and thank you for listening.
