open-rag-eval: RAG Evaluation without "golden" answers.

How do you evaluate a RAG system when you have useful questions but no curated set of correct answers or ideal retrieved passages? Creating those references adds work every time the evaluation set grows. Open RAG Eval, introduced here by Ofer Mendelevitch of Vectara developer relations, addresses that bottleneck: evaluate retrieval and generation without manually supplied golden chunks or golden answers. The opening slide’s QR code provides access to the repository.

The project draws on research conducted with the University of Waterloo’s Jimmy Lin Lab. Its approach replaces the requirement for curated references with automated assessments of what the pipeline actually retrieves and generates. Understanding those assessments matters: each measures a different part of the RAG system.

Start with 10, 100, or 1,000 queries that matter to your RAG application. These are example evaluation-set sizes, not throughput claims. The workflow then follows the data through three stages:

1. Capture the pipeline’s behavior. A RAG connector collects the actual retrieved chunks and generated answers. The demonstrated integrations include Vectara, LangChain, and LlamaIndex.
2. Evaluate the captured outputs. The resulting RAG outputs feed metrics grouped into evaluators. This separates collecting a pipeline’s behavior from assessing it.
3. Write evaluation files. Evaluators emit files containing the results for inspection and comparison.

The connector is therefore the boundary between your application and the evaluation machinery: it supplies what the system did, without requiring you to author the ideal answer first.

The package introduces four complementary assessments: UMBRELA for retrieval, AutoNuggetizer for generation, citation faithfulness, and hallucination detection using a Vectara model. Keeping these dimensions separate helps distinguish retrieving useful evidence from producing a well-supported answer.

UMBRELA scores individual passages against the query, without requiring golden chunks. Its relevance scale runs from 0 to 3:

The endpoints distinguish irrelevant retrieval from a passage that directly supplies the requested information.

Mendelevitch reports that UMBRELA correlates well with human judgment, citing the Waterloo research. That supports using automated relevance assessment without golden chunks, but the consequential distinction is between agreement in evaluating systems and identical labels for every passage. The associated research supports system-ranking agreement; the talk supplies no correlation coefficient or experimental conditions for its broader qualitative claim.

AutoNuggetizer evaluates generation by breaking the information an answer should cover into nuggets, or atomic information units. Instead of comparing the generated response with a single golden answer, it checks coverage of selected units:

1. Create nuggets. Produce atomic units of information for the evaluation.
2. Prioritize them. Assign each nugget a vital or okay importance rating, order the nuggets, and select the top 20.
3. Judge answer coverage. An LLM judge examines the RAG response to determine whether each selected nugget is fully or partially supported by the answer.

The top-20 limit is a selection setting, not an accuracy result. The mechanism measures information coverage: an answer can cover one important point completely and another only partly.

For a small Python implementation of the importance-ordering step, retain each nugget’s text and rating together, then select the first 20 after sorting. This helper represents selection only; nugget creation and support judgments remain separate operations.

from typing import Literal, TypedDict

class Nugget(TypedDict):
    text: str
    importance: Literal["vital", "okay"]


def select_nuggets(nuggets: list[Nugget]) -> list[Nugget]:
    priority = {"vital": 0, "okay": 1}
    ordered = sorted(
        nuggets,
        key=lambda nugget: priority[nugget["importance"]],
    )
    return ordered[:20]

The original nugget records survive the selection, so the subsequent judge can assess each retained unit against the generated answer.

Mendelevitch points to the papers for fuller method details before moving to two support checks. These operate at different scopes:

Citation faithfulness distinguishes full support, partial support, and no support. A citation’s presence alone does not establish that its passage substantiates what the answer says.

For the response-level check, HHEM, Vectara’s hallucination detection model, assesses alignment between the answer and retrieved evidence. Here, support is relative to that evidence: a statement can be true elsewhere yet unsupported by the passages supplied to the model. This complements nugget coverage by asking whether the answer stays grounded in what was retrieved.

Once evaluation finishes, you can inspect the result files directly. For easier comparison, the talk shows a hosted interface and describes dragging the evaluation files onto open-evaluation.ai. The populated view lists queries alongside retrieval and generation scores, with groups of colored results arranged side by side. This makes the separate evaluation dimensions visible together without requiring manual inspection of every file.

The intended use is to optimize and tune a RAG pipeline while keeping its evaluation methods inspectable. Open source provides access to the metric implementations, so a score need not remain an opaque judgment: you can examine how the assessment works.

Vectara, LangChain, and LlamaIndex provide the demonstrated integration paths. For a custom pipeline, the extension point is the connector that captures retrieved chunks and generated answers. Mendelevitch invites issues and pull requests for additional connectors, allowing other pipelines to feed the same evaluator workflow.