open-rag-eval: RAG Evaluation without "golden" answers.

Talk transcript

1. 0:00

   Hi, everyone. My name is Ofer from Vectara. I run our developer relations, and I'm gonna talk to you today about Open RAG Eval, a new open-source project for quick and scalable RAG evaluation.

2. 0:12

   You can use the QR code here to get to the repo directly.

3. 0:16

   So what is Open RAG Eval? It's a open source project that is aiming to solve one of the major problems of RAG evaluation, the fact that you require golden answers or golden chunks to do anything, which is really non-scalable.

4. 0:31

   It is research-backed, so it's work that we've done with collaboration with University of Waterloo, the Jimmy Lin Lab specifically. And let me show you how it works.

5. 0:43

   So generally, this is kind of an architecture view of what, what it does. So you start with a set of queries. It could be ten or a hundred or a thousand queries that you've collected that are important for your RAG system.

6. 0:55

   We have a RAG connector, and that really collects all the actual information, the actual chunks, the actual answers that are generated by a RAG pipeline. We have one for Vectara, for LangChain, for LlamaIndex, and a growing number of other connectors.

7. 1:09

   And those connectors generally generate these outputs shown here, which is the, the RAG outputs. From there, you run the actual evaluation, which runs a bunch of metrics. Metrics are grouped into evaluators, and that's kinda how the, the internal architecture works.

8. 1:24

   And those evaluators generate RAG evaluation files. And that has everything you need to know to really evaluate your RAG pipeline.

9. 1:34

   Now, the metrics are where it gets really interesting. How do we do this thing with no golden answers? Well, there's a few metrics I wanna mention here that are part of this package.

10. 1:44

    One is called UMBRELLA. It's really allows you to do retrieval without the golden chunks. The other one is called AutoNuggetizer. I'll talk more about this for generation. We have the citation faithfulness, which measures whether citations in the response are really correct, and hallucination detection, which is based on Vectara's hallucination detection model.

11. 2:05

    So let's go into each of these in detail. UMBRELLA is a retrieval metric, as I mentioned, and what it does is it actually takes a chunk and gives it a score between zero and three, zero being that this chunk or passage has nothing to do with the query, and three being that it's dedicated to the query and

12. 2:24

    contains the exact answer. Now, the nice thing about this is not just the scale here, which is pretty self-explanatory, but the fact that the research done by the University of Waterloo Lab of Jimmy Lin shows that if you use this approach, it correlates well with human judgment, and that is really, really powerful.

13. 2:44

    So if you use this, even without the golden chunks, you know that the results will be good.

14. 2:52

    Now, AutoNuggetizer is the generation metric. And again, it doesn't require the golden answers in this case. And it works a little bit differently. There's three steps here. The first step is nugget creation, where you create these atomic units called nuggets.

15. 3:07

    Then for each nugget, you assign a vital or okay rating. You sort of sort these nuggets by... get the top twenty. And then there's a step where an LLM judge analyzes the response you get from your RAG to determine if each of the selected nuggets is either fully supported or partially supported by the answer.

16. 3:25

    Again, you can see all of this in the papers in, in much more detail. The third metric is citation faithfulness, which essentially measures whether the citation, the passage, you know, is high fidelity.

17. 3:37

    It's, it's, it's fully supported, partially supported, or there's no support for the citation in the response. And then the last one is really using HHEM, Vectara's hallucination detection model, to check if the entire response aligns with the retrieved content.

18. 3:54

    So that's, that's the metrics we have, and there's also a very cool user interface for this. So once you finish with your running your evaluation, you can, of course, look at the files yourself, but that's tends to be pretty complex.

19. 4:08

    So you can drag and drop those files onto open-evaluation.ai, as shown here, and you get this really, really cool UI that shows you all the queries you ran and all the things that you wanna compare between the retrieval scores, the different generation scores, et cetera.

20. 4:26

    And that's it. So I encourage you to take a look at this. It's a very powerful package, can help you optimize and, and tune your, your RAG pipeline. Again, it's open source, so all the source is open.

21. 4:37

    You can take a look and see how it works. This drives a lot of transparency, so the metrics are very clear in how they work. As I said, it includes connectors to Vectara, LangChain, and LlamaIndex.

22. 4:47

    But if you have your own RAG pipeline or some other RAG pipeline you wanna contribute, we're very much in favor of contributing other issues or PRs for other connectors.

23. 4:57

    And let us know if you have any questions. This is about Open RAG Eval, and thank you for listening.